Taste-Bench shows AI agents still struggle with long-term decisions.
A new benchmark, Taste-Bench, finds AI agents are weak at long-term choices: the best scored 59.7 percent, and results fell to near guessing when payoffs came late.
Illustration made on Lumeta with GPT Image 2.5Make one like it
On Oct. 18, 2026, researchers from Microsoft and City University of Hong Kong introduced Taste-Bench, a benchmark for long-running agent decisions, and the early result is blunt: current AI agents are not very good at choosing the better long-term path. The best model in the study scored 59.7 percent, and performance fell near random when the benefit of a choice only became visible much later, which matters for anyone using agents on coding or research tasks.
What Taste-Bench actually tests
Instead of asking whether an agent eventually finishes a task, Taste-Bench asks whether it can recognize the better direction at a key fork in the work. The benchmark contains 502 questions built from real agent trajectories in coding and AI research tasks. At each decision fork, the later steps are hidden and the model must choose between two paths before the outcome is obvious. Because each question is asked twice, once with the options flipped, pure guessing only scores 25 percent under the paper's rules.
Illustration made on Lumeta with GPT Image 2.5Make one like it
How current models performed
The headline result was modest. GPT-5.6 Sol led the benchmark at 59.7 percent, followed closely by GPT-5.5 at 59.5 percent, and the rest of the tested models were well below that. The harder cases were the ones where the deciding clue only appeared after much later work: average accuracy across models was 62.3 percent when the clue was already visible in the earlier context, but just 21.0 percent when it only showed up after extensive later steps. The researchers also tested different reasoning budgets and found that thinking longer barely changed accuracy.
Illustration made on Lumeta with GPT Image 2.5Make one like it
The more encouraging part of the paper is that this judgment seems trainable. Using Qwen3.6-27B as a base, the team distilled a teacher model's decisions into a student that only saw the benchmark question itself. On unseen tasks, that student's judgment accuracy rose from 30.0 to 47.9 percent. When the student gave advice at each fork to a separate agent, success on 41 held-out SWE-bench Pro tasks increased from 14.6 to 33.7 percent. The authors say this shows long-term planning taste can be measured and improved, not just observed. They also made the benchmark and code public.
What does 'taste' mean for an AI agent here?
In this paper, 'taste' means an agent's ability to choose the better path during a long task before the payoff is obvious. It is about judgment at intermediate decisions, not just whether the final answer passes a test.
Was performance really close to random?
Sometimes, yes. Under the benchmark's scoring setup, pure guessing gets 25 percent because each question has to be answered correctly in both option orders. When the useful evidence only appeared much later in the trajectory, average accuracy across models dropped to 21.0 percent, which is near that random baseline.
Did giving models more reasoning time help?
Not much. The researchers tested different reasoning settings for two models and reported that moving from lower to higher reasoning budgets barely changed accuracy.
Why does this matter if a model can still finish some tasks?
Because an agent can make an early choice that looks fine at first but creates problems later. For coding and research work, those hidden bad decisions can waste budget, block future changes or send the agent down an unproductive path.
