Case study: the 10 most common coding tasks
10 slimmed-down, single-file versions of the most common day-to-day coding requests - fix a bug, add a feature, write a function, write tests, refactor, validate input, debug an error, integrate a real API, write docs, simplify code - each sent to the top 3 paid models cheaper than GLM 5.2 (by success rate) and the top 3 free models (by success rate). 60 runs, Prompt Guidance on.
100%
Overall success (60/60)
63%
First-try, no fixer needed (38/60)
6
Models: 3 cheap-paid + 3 free
Key finding - a real detector gap, found and fixed mid-batch
The first run of this batch landed 55/60 (92%) - tencent/hy3:free failed 5 of 10 tasks, every time by writing a real, substantive plan or worked-through calculation in its reply and then never calling the edit tool. The auto-retry never fired for any of them, because the "announced intent, did nothing" detector only matched specific phrasing like "I'll build/create X" - a reply that just launches straight into working through the problem in prose didn't match, so it was silently treated as a normal (if useless) reply instead of triggering a retry. We fixed the root cause - outside Plan Mode, any zero-edit reply is now flagged regardless of wording - and re-ran the full batch. Every one of those 5 recovered via the fixer chain, taking the batch to 60/60 (100%). The numbers below are from that fixed re-run.
Fix a bug
Add a small feature
Write a function from scratch
Write unit tests
Refactor messy code
Add input validation
Debug from an error message
Simple API integration
Write docs
Simplify/cleanup code
Every screenshot and HTML file is the actual model output, rendered as-is - click any thumbnail to open the live page.
Full testing history and methodology: /stats.