Do we really need so many harnesses?

A harness is the program around an AI model. It builds each prompt, runs the tools that the model asks for, keeps a record and decides when to stop.

  • The model has no memory. It reads text and writes text.
  • The harness keeps the past conversation. It chooses what to send to the model on the next turn.
  • The harness connects the user, the tools and the computer. It runs the tools and sets the permissions.
  • A harness is a complex program. It has its own failure modes, in addition to those of the model.

Inside, harnesses are quite similar. They share one loop and the same few parts. But they make different choices in a few places: how much fixed text to send, when to summarize, when to stop, and what to allow. These choices change the quality and the cost, and no one choice is best for everyone. The right choice depends on your model, your task and your team. Today, to change one choice, you must pick another harness or write your own. A new harness also changes all its other choices, so you cannot tell which choice caused a change in the result.

I believe that we do not need more harnesses. We need one harness where each choice is a setting. You change one setting, keep the others, and see what that setting does.

In our Harness Engineering workshop, we deep dive into these settings, and how changing them affects cost and performance. Sign up here.

A mixing desk: one set of sliders for the settings of a harness


1. Everyone is building one

How features spread from harness to harness

A harness is easy to start and hard to finish. Yet everyone is building one.

Claude Code, Codex, Hermes, Pi: if you code with an AI agent, you probably use one of these, and you may have noticed that they behave differently even when they run a similar model. Meanwhile, someone on your team is probably writing their own. The list is long: the model makers ship Claude Code, Codex CLI, Gemini CLI and Mistral Vibe. The open-source world adds Pi, Hermes, OpenCode, OpenClaw, OpenHands and Aider, and researchers run small ones such as Mini-SWE-Agent, SWE-agent and Terminus-2 (Harness Engineering study; SWE-bench experiments; Long-Horizon-Terminal-Bench).

A second wave of tools does not replace the harness but wraps around it. Caveman sits between the harness and the model provider to trim the bill, TokenEconomizer learns from past sessions, actor-gatherer picks what the model gets to see, and Omnigent drives several harnesses as interchangeable parts (same study).

Why so many? Starting is easy. Mini-SWE-Agent does the whole job in about a hundred lines. One study of eleven harnesses ends with a working one in ninety lines (Harness Engineering study). The hard part comes later.

Why it looks wasteful. When one harness finds a good idea, the others have it within weeks. Three features each spread to several harnesses in one quarter: loading tool descriptions only when they are needed, a separate planning mode, and an AI that reviews risky commands before they run (same study). The copying is visible in the code. Codex uses the hook names of Claude Code, and OpenCode reads the skills folder of Claude Code.

So, why does each team still build its own?

2. Why they all build one anyway

Share of user messages that correct the agent, after a summary and at other times

Teams have good reasons to build their own harness. I found several reasons, and the first is the most basic.

Mitchell Hashimoto (co-founder, Hashicorp) describes harness engineering as turning each mistake that the agent makes into a permanent fix in the harness. The mistakes depend on your codebase, your tools and your habits. A rule such as “always run this linter before you stop” is a great fix for one project and only noise for another. Thus, a harness that has grown around the mistakes of one team can feel clumsy to the next team.

The second reason is the model. Claude Code assumes a strong model and mostly stays out of its way. Mingbird starts from the opposite assumption. It is built for small open models, 2 to 9B parameters, that run on a laptop, and it steps in often. It refuses irreversible actions. It sends one corrective prompt when it detects a loop. Before it accepts that the work is done, it re-reads the task and runs the task’s own checks (Mingbird paper). Editing files shows the same pattern. To change a file, the model must quote the exact text to replace. A strong model quotes it correctly. A weak model makes small errors, so the harness must accept text that is almost the same, or ask for another format: the full new file, the old and new lines, or a diff. Aider has thirteen edit formats and chooses one for each model (Harness Engineering study).

The third reason is context, the text that the model receives on each call. The window has a limit, so a long session must remove or compress old text, and each method loses some information. One developer tracked 162 days of their own Claude Code use. Five steps after a summary, 41.9% of their messages corrected the agent. At other times, the rate was 17.7% (LangWatch). Thus, people build their own methods. Actor-gatherer, for example, adds a second program that keeps the full history and puts the related old text back before each call, even if the agent does not ask for it. On tests where the key fact is in an early user message, its authors report scores 27 to 36 points higher (actor-gatherer). The best method also depends on the size of the window. A separate study compared five methods (Fan et al., arXiv 2609.20804). With a 32k-token window, the best and the worst method differed by 35.7 points. With a 128k window, they differed by only 2.7 points.

The fourth reason is how the harness loads tools. Each tool needs a description in the prompt, and harnesses have from 1 tool (Mini-SWE-Agent) to more than 109 (OpenClaw). Many harnesses now send only the tool names, and load the full description when the model needs the tool. One paper reports that this makes the Claude Code prompt approximately 40% smaller. In my own experiments, however, the same change made the total cost 7% higher. A saving on paper does not always reach the bill, and each team must measure its own case.

The last reason is about business. A harness has become a platform worth owning. Harnesses turned into the frameworks that people build on, such as the Claude Agent SDK and the Codex SDK, and framework makers such as LangChain now ship harnesses of their own. Owning the harness also means that you control the updates. Codex downloads its prompts and settings from OpenAI’s servers, so its behaviour can change on the day a new model ships.

These are good reasons to want a harness that fits you. They are not, by themselves, reasons to write one from scratch. To see why, we first look at what all these harnesses share.

3. Under the hood they are quite similar

The loop that most harnesses share

Inside, the harnesses look alike. The authors of the study from section 1 read the code of eleven harnesses (Harness Engineering, arXiv 2609.00006). They found the same structure in all of them:

  • The same seven parts. Every harness builder makes a decision about the same seven parts: the loop, the link to the model, the tools, memory and context, safety, helper agents and extensions. For some parts, the decision is “none”.
  • The same loop. Nine of the eleven harnesses use one plain loop. They build a prompt, call the model, run the tools that the model asks for, add the results, and do it again. The other two harnesses add a layer on top. They keep the loop.
  • The same way to shorten the context. Seven of the eleven harnesses ask the model for a summary when a conversation gets near the limit. Then they replace the history with the summary.
  • The same way to find information. All eleven harnesses read project notes from Markdown files, such as AGENTS.md or CLAUDE.md. All of them find code with plain text search. None of them searches code by meaning.
  • The same instructions. Six harnesses tell the model, in almost the same words, to do only what the user asks.
  • The same extensions. Most harnesses load skills: folders of written instructions that the agent reads when it needs them.
  • The same team orchestration. If they use helper agents, one lead agent gives work to the helpers.

Thus, a new harness mostly builds the same parts again. But “quite similar” is not “the same”. In the next section, I show the real differences and why they are important.

4. How they actually differ

Where harnesses make different choices

Harness builders make different choices in a small number of places. Each choice involves one of the core features:

  • Different fixed prompts. Each call starts with the same fixed text: the instructions and the tool descriptions. Claude Code sends approximately 27k tokens of it. Pi sends approximately 2k. On the same tasks, the two harnesses solved within 2 points of each other, but Claude Code cost approximately twice as much (HarnessTax).
  • Different numbers of calls. On one SWE-bench task with the same model (Claude Opus 4.5), mini-SWE-agent made 32 model calls and cost $0.72. live-SWE-agent made 20 calls and cost $0.35. Both harnesses solved the task. I measured this from their public traces (SWE-bench experiments).
  • Different limits on old text. Strands Harness cuts each tool result above approximately 1,500 tokens, and summarizes the conversation at 85% of the window. On Terminal-Bench 2.1 with the same model, it cost $56 against $248 for Claude Code, at equal accuracy (Strands Harness).
  • Different cache care. The provider charges approximately one tenth of the price for text that starts the same as the previous prompt. Claude Code divides its instructions into a fixed part and a changing part, so the fixed part stays cheap. Hermes writes the date without the time and freezes its memory file for the session. The study’s authors report that Pi records, for each turn, the money lost to avoidable cache misses.
  • Different stop rules. Hermes does not let the model stop after a code edit until the tests run again. OpenCode asks the user after three identical calls. Claude Code, Codex and Pi do not check for a stuck loop.
  • Different safety. Codex and Gemini CLI run commands in an operating-system sandbox on three platforms. Hermes and OpenCode are equally large and have no sandbox. Claude Code checks a risky command in three steps: fixed rules, then an AI classifier, then the user. Pi’s authors refuse a partial sandbox, because users can think that it is a security wall.
  • Different helper agents. Claude Code gives a helper the same instructions as the lead agent, so the cache stays cheap. Codex lets the user choose how much history a helper sees. Hermes never gives a helper a tool that the lead agent does not have. Aider has no helpers.
  • Different long-term memory. In Codex, a background agent writes the memories. In Gemini CLI, an agent writes them and a person reviews them. Hermes lets the model write two small files, and freezes them for the session. OpenClaw runs a memory helper before each reply.

These choices are important because they impact results and performance:

  • Results. In one controlled test, a change of harness moved scores by 8.5 to 13 points. A change of model moved them by only 2.5 to 5 points (Zhang et al., arXiv 2605.23950). In another test, the best and worst of six harnesses differed by 23.8 points on the same tasks and models (Harness-Bench).
  • Quality over time. In spring 2026, users said for a month that Claude got worse. Anthropic found three harness changes, not a model change. One was a cache bug. Users felt it as forgetfulness (Anthropic postmortem).
  • Cost. The harness decides what to send again on each call. Thus, it sets most of the bill. In one of my own Claude Code sessions, the model’s own output was only 16% of the bill.

Thus, the harnesses differ mainly in their settings: how much fixed text to send, when to summarize, when to stop, and what to allow. These values change the results, the quality and the cost. Here, I argue for a better path.

In our Harness Engineering workshop, we deep dive into these settings, and how changing them affects cost and performance. Sign up here.

5. Need Configurability, not Duplication

From separate harnesses to one configurable harness

We do not need more harnesses. We need one harness that we can configure. Such a harness must have these properties:

  • One shared core. The core does the work that all harnesses do (section 3). It builds the prompt, calls the model, runs the tools and keeps a record.
  • Each difference is a setting. You set the fixed prompt, the cache rules, the stop rule, the safety level, the helper agents, the memory, the context method and the edit format (sections 2 and 4). You write a setting as data. You do not change the core.
  • One change at a time. You can change one setting and keep all the other settings. Thus, you can see the effect of that one setting.
  • A full record. The harness keeps a log of each call. You can replay a session and examine it.
  • A forecast before you change. The harness tells you what a setting will do to the cost before you use it.

Some harnesses already take the first steps:

  • Pi puts almost every feature in an extension. The core is small (Harness Engineering study).
  • Mistral Vibe writes each turn rule as a small plug-in: the turn limit, the cost limit, the token limit, automatic summaries and read-only mode. The authors of the study recommend this design when a harness has three or more turn rules.
  • Hermes writes its safety rules as data. It also freezes a minimum set of blocked commands when it starts, so no injected text can turn them off.
  • Add-ons such as caveman and TokenEconomizer (section 1) change the settings of a harness from outside. Their users change settings, and do not build a new harness.

A personal note. To explore configurability, I build muharness, a simulator of harnesses. It is not a harness – does not call a model, and does not change files. It is a simulator of harnesses: one small core and a set of settings. I wanted to know if this design works. Some of my observations:

  • I grouped harness features by three questions. Does the feature change what the model sees? Does it change what the harness does with the reply? Does it change what stays after a crash? Each of the 29 design patterns from the study (section 3) fit one of these questions. I did not need a fourth question.
  • To check if muharness can estimate session costs faithfully, I replayed real logs of two harnesses on the same core, with no change to muharness core. For Claude Code (275 calls), each token count was within 10 tokens. The total estimated cost was 0.6% less than the bill. For Codex (43 calls), the prompt sizes were exact, and the cache read estimates were 0.5% to 2.1% too high. So, muharness was able to estimate costs quite accurately just from logs and current prices.
  • I wrote other harnesses and harness features as settings of muharness and simulated them: the 90-line harness from the study, mini-swe-agent, Mingbird. The core largely remained same, with some feature additions. I will discuss those elsewhere.
  • muharness can calculate the cost of a setting change from a recorded session, without a new run. The result is exact when the model responses are same, for example a different cache lifetime or price. When the setting change, for example a new summary rule, may lead to different responses, the result is only an estimate but still useful.

So, do we really need so many harnesses?

I don’t think so. The harnesses are quite similar underneath. They differ in a small set of choices, and each choice can be a setting. Today, a team that wants a different setting often writes a new harness. That is slow, and good ideas spread only by copying. A better direction is one configurable harness, and a simulator that shows the cost of each setting before you change it. muharness is a first step toward that simulator.

Until that harness exists, I recommend asking one question before you build: which setting do I want to change? If an existing harness, or an add-on, lets you change it, you do not need a new harness. If none does, build that one setting, and share it.


Learn to tune a harness. In our Harness Engineering workshop, we work with the loop, the tools and the checks around a model. You see what a harness costs, how much you can trust it, and which settings fit your model and your task. Join the waitlist.

Have a system that needs a second opinion?