App-review → structured JSON: base vs LoRA-tuned
Fine-tuning a small open model (Qwen2.5-0.5B-Instruct) with LoRA to
extract a strict, closed-vocabulary JSON object from an unstructured app-store
review — and measuring the gain against the untuned base with fully
deterministic, rule-based scoring (no LLM judge).
62.3% → 76.6%Aggregate score (base 3-shot → tuned 0-shot)
+14.2points on 200 held-out examples
26% → 74%topics field — the biggest lift
1.75%of params trained (LoRA, 8.8M / 503M)
What is this page? A live Gradio demo needs a paid HF tier, so this is a
free static gallery of real, precomputed model outputs on 20 actual
held-out reviews — each shown against its gold label. To run inference on your own
text, load the adapter or run
space/app.py from the repo (both linked above).
Results
| Model | Parse OK | Aggregate | sentiment | topics | mentions_price | rating_implied |
|---|---|---|---|---|---|---|
| Base (3-shot, fp32) | 100% | 62.3% | 68.0% | 26.1% | 95.0% | 60.2% |
| Tuned (LoRA, 0-shot) | 100% | 76.6% | 72.5% | 73.5% | 97.5% | 62.7% |
Scoring: exact match (sentiment, mentions_price), set-F1 (topics), exact / off-by-one with half credit (rating_implied). Deterministic; no model judges.
Live examples — base vs tuned, scored against gold
matches gold
off by one (rating)
wrong
dashed = gold topic the model missed
Training & evaluation at a glance
Honest caveat. 3 of the 4 label fields are generated by deterministic
rules, so part of the tuned gain is the model learning that labeling function rather
than the underlying concept.
rating_implied uses the dataset's native star
rating and is the most trustworthy field; mentions_price is ~96% one class.
Full detail in the repo's publication/limitations.md.