Why AI Coding Benchmarks Fail in Real-World Tasks
AI agents ace coding benchmarks but still need human help with complex, real-world tasks.
Illustration made on Lumeta with GPT Image 2.5Make one like it
On Nov. 8, 2026, researchers from Alibaba and several Asian universities reported that AI coding agents still fall short on most real-world app-editing tasks. In their new Co-Design Bench test, the best model fully completed only 26.4 percent of tasks, so human cleanup was still needed on nearly three out of four jobs. For teams using coding agents today, the practical takeaway is simple: treat them as strong assistants, not hands-off finishers, and test the model together with its working environment before you rely on a leaderboard.
What the new benchmark found
Co-Design Bench is designed to look more like actual product work than older coding tests. Instead of clean, isolated tasks, the benchmark asks agents to modify existing web apps based on real user change requests, sometimes using images, audio, video, or documents as input. According to the paper, GPT-6 Astra was the top performer of the ten models tested. It met about 70 percent of a task's requirements on average, but its work was fully correct in only 26.4 percent of tasks.
Illustration made on Lumeta with GPT Image 2.5Make one like it
Why the scores are much lower than older tests
The researchers argue that many older coding benchmarks have become too easy for agentic systems. When a model can run, inspect, and revise its own code, scores can climb above 99 percent, which makes it hard to tell strong systems apart. Co-Design Bench tries to avoid that by using real requests from sessions where users often had to restate instructions or undo bad changes. Grading is strict too: one AI judge tests the finished app in a browser, then a second re-checks positive results and looks for mock-ups or hardcoded shortcuts. Features count only if they actually work.
The hardest tasks were the ones that mixed several kinds of change at once, such as layout, design, interaction, and content updates. Every model got worse when a task required three or four edit types. The study also found that the harness matters almost as much as the model. In one comparison, Qwen3.8-27B scored 75 or 56 percent on larger overhauls depending on the environment it was run in. That means companies should evaluate the model and the agent setup as a pair. The benchmark also has limits: each task uses one fixed request, follow-up clarification is not part of the test, and AI judges can still make mistakes.
Illustration made on Lumeta with GPT Image 2.5Make one like it
Does this mean coding agents are not useful?
No. The results suggest they can do a large share of the work, but they still often miss at least one requirement. That makes them helpful for speeding up implementation, but risky to use without review.
What makes Co-Design Bench different from older coding benchmarks?
It uses real user change requests on existing web apps instead of cleaner benchmark tasks. Some tasks also require agents to use images, audio, video, or documents, and grading checks whether the finished app actually works in a browser.
Why do real-world tasks seem harder for these agents?
Real requests often combine several types of edits at once, such as design, layout, interaction, and content changes. The study found that this kind of mixed work consistently hurt performance across all tested models.
What should teams do before choosing a coding agent?
Do not rely on a model leaderboard alone. Test the model together with the harness or agent environment you plan to use, and run it on tasks that look like your real product changes.
