Why local agents fail, and none of it is the benchmark score

A model that scores well on a coding benchmark can be useless in an agent loop, and the gap is not small. In one published evaluation on a 32 core machine, deepseek-coder at 33B scored 90 percent on code generation and 10 percent on agent tasks. The same weights. The difference is that a benchmark asks for one answer and an agent loop asks for forty decisions in sequence, and the second thing is where local deployments actually die.

These are the four failures we look for first, in the order they usually bite.

1. Silent context truncation

Some runtimes default to a context window far smaller than the model supports, and truncate to fit without raising an error. The agent then behaves as though it has amnesia partway through a task, which reads to the operator as the model being stupid rather than the configuration being wrong. This is the single most common misdiagnosis we see, and it costs people weeks before anyone checks the setting.

2. The tool count cliff

Small and mid sized models emit valid structured tool calls reliably up to a handful of available tools, and then degrade sharply. Reported behaviour for Qwen3-Coder through one common runtime puts the reliable ceiling at around five tools. Above that, malformed calls rise and the loop starts failing in ways that look random.

The practical consequence is architectural, not a tuning problem. If a local agent needs twenty tools, it needs several agents with five each and something routing between them. Designing that in from the start is cheap. Discovering it after integration is not.

3. Error compounding across a trajectory

In a long horizon loop, a minor wrong decision in an early step does not stay minor. It enters the context as established fact and every later step reasons from it. Frontier models are better at this than open weight models, but neither is good at it, and no benchmark that scores single turns will show you the difference.

4. Nobody can tell which step was hard

This one is worth stating carefully because it is the most counterintuitive. Published work on routing inside agent trajectories found that rule based routing failed 38 of 40 trajectories, and that a frontier model used as a judge to assess step difficulty identified only 7 of 147 genuinely hard steps and failed all 40 trajectories it was tested on.

What this means for a deployment

• Set the context window explicitly. Never trust a runtime default.
• Keep the tool surface per agent small. Five is a working ceiling, not a target.
• Checkpoint the trajectory so a wrong turn can be replayed from before it, not just retried.
• Put a human review gate at the step that touches something expensive or irreversible.
• Evaluate on your own tasks. A leaderboard position has close to zero predictive value here.

A local agent is genuinely useful for small, well scoped, repeatable work. It is not a frontier cloud model and anyone selling it as one is either not measuring or not telling you. We would rather scope the first deployment narrowly and have it work.