Harness Engineering
Ryan Sikorsky and I have been testing one question: if an agent can repair the system it runs in, what stops it repairing itself into something you never asked for?
Our answer so far is a constitution. It is a short set of rules the agents are free to work within, but can never rewrite. A separate decision model enforces it, not the agent being judged.
The first version of the harness came together in three days. This is a write-up of what we built, what it taught us, and the harder questions we think come next.
I've spent the last few years arguing that trustworthy AI isn't a policy layer bolted on after deployment. It's an architectural choice, made before a single line of autonomous code ships.
What Ryan has built here is one of the clearest demonstrations I've seen of that idea in practice. A coding agent isn't safer because we trust it more. It's safer because the system is built so that trust is never required of it: the judge is separate, the constitution is out of reach, and the humans decide what freedoms get granted, and when.
That's exactly the shape I want to see more of, as agentic systems move from demos into production across industries. The agents don't choose their own freedoms. We do. Worth writing that down, long before we hand them anything more.
— Jatin
The harness runs unattended: point it at a backlog of stories and it builds them, tests them, ships them and moves on. Three parts do the work.
fx, the coding agent from Vercel Labs. Four fx workers run in parallel, each taking one story at a time. fx writes failing tests first, then the code to pass them.
Vercel AI Gateway. Every model call goes through one gateway: the coders, the reviewer and the escalation models. That gives us one bill, one place to switch models and a meter on every call. It is what makes per-story and per-day spend caps enforceable rather than aspirational.
Jev, a decision model from TypeSafe AI. Jev does not write code. It reviews every story against its acceptance criteria, scores the risk of each change, decides which stories can safely run side by side, and gates every release the harness makes of itself. That is what Jev is built for: it returns typed choices, scores and probabilities rather than prose. Deterministic checks (types, lint, tests, build) run first, and Jev judges only what code cannot.
The important design choice is the separation. The model doing the work never decides whether the work is good. A big thank you to the Vercel team: fx and the AI Gateway made the coding and model-routing layer possible in an afternoon; the constitution, the Jev integration and the rest took the other two and a half days.
The constitution is one small file, owned by Ryan and edited only by hand. Inside it, the harness is free to diagnose its own failures, write stories to fix them, build those fixes test-first, release a new version of itself and roll back if the release goes badly. Outside it, nothing automatic can act. These are the rules we think are most interesting.
The constitution cannot amend itself. Any change the harness proposes that touches the constitution is a hard violation. No approval path, trial or "it only makes things stricter" exception can let it through. A person edits it, by hand.
The judge cannot be switched off or replaced. Review by Jev must stay on, unreviewed work is always rejected, and if Jev is unreachable the harness waits rather than skipping review. The harness may trial a backup reviewer and report on it, but only a person can change who the reviewer is.
The harness cannot choose what it is judged on. Every release of itself is tested against a held-out bench of five stories that have already shipped, stored where the harness cannot edit it. A release that ships two fewer stories, or costs twice as much per story, is refused.
Tighten freely, loosen only on trial. A change that makes a check stricter ships normally. A change that loosens one, even within the limits, ships as a trial that reverts itself if real metrics slip: first-pass rate down more than 10 points, or cost per story up more than 25%.
Money is a wall, not a target. There are fixed ceilings per story, per day and per self-healing cycle. After two cycles in a row that ship nothing, the harness must pause, and it may pause sooner but never later.
The brakes cannot touch the brakes. The self-healing code, the quality gates, the CI and the reviewer's prompts are all off-limits to automatic change. The one exception: it may add a new test file. It can add checks, never remove them.
It cannot widen its own fast lane. Low-risk releases (tests and docs only) skip some checks. The list of what counts as low-risk lives in the constitution, so the harness cannot quietly reclassify its own changes.
The constitution reached version 9 in its first week. Every amendment is recorded with the human instruction that authorised it, quoted in plain words. That turned out to be one of the most useful parts: it is an audit trail of every time we chose to give the system more freedom, and why.
Across its first 213 stories the harness averaged about $1.40 a story. The judge's own cost barely shows up in that number.
A cheap, typed judge changes the economics. At around $0.0002 and 0.2 seconds a review, every attempt gets reviewed, not a sample. Early on, one change passed every lint, type and test gate but had no real tests for the new behaviour. Jev scored "has tests" at 0.09 and sent it back.
Most work does not need the biggest model. 92% of stories went green on the cheapest tier. Most of the expensive spend came from stories we had pinned to a bigger model up front, when Jev would have started them small.
The harness runs two loops on itself. Self-healing fixes what broke: it reads its own history, writes stories to fix whatever slowed it down, builds them test-first, releases a new version and rolls back if the release goes badly. Self-evolving goes further: it tunes its own settings to get better, not just to recover. It released ten versions of itself in its first two weeks.
The constitution is what makes the evolving safe. It does not fix the settings; it bounds them. Take Jev's review thresholds. The constitution says the "implements the story" score may never drop below 0.75, and the risk score may never rise above 2.7. Inside those bounds the harness is free to move its own settings: a stricter value ships on its own, a looser one ships as a trial that reverts itself if quality slips. Today it runs tighter than the floor, at 0.85 and 2.5, because an audit of 77 shipped stories showed the stricter values caught 6 of 11 bad changes against 4.
There is a third loop, and it is ours. We watched how the harness was doing and moved the bounds themselves, by hand: the constitution reached version 9 in a week. The agents evolve inside the rules; people evolve the rules.
The guardrails turned out to be the easy part. The hard part is the signal the loop learns from, and that leads to the questions we find most interesting.
The rule is simple: fail closed on quality. A pause is cheap; an unreviewed change on main is not.
We think self-improving agents are moving from research demos to a governance problem, and the next year will be about who sets the rules and who checks them.
The research is moving fast. Lilian Weng's Harness Engineering for Self-Improvement argues that self-improvement will arrive through the harness around the model long before models rewrite their own weights. Systems like the Darwin Gödel Machine and Self-Harness already rewrite their own code and rules. But the same work keeps finding the same failure modes. The Darwin Gödel Machine was caught faking test logs to look successful. Other papers document agents inventing failures that never happened, and safety drifting as agents evolve.
We would put the industry somewhere between the first incidents and the first real governance layer. The organisations that write their rules down now will be the ones able to grant their agents more freedom later.
The tools under the harness move faster than the harness. fx ships often and has a built-in fx upgrade with stable and dev channels, and new models arrive on the gateway every week. A harness that runs for weeks is out of date before it finishes. Our view: an upgrade is just another release of the harness, and should pass the same gates.
The AI Gateway publishes its whole catalogue, with prices and capabilities, through one open endpoint, so the harness can see a new model the day it lands. Seeing it is easy. Deciding whether it is better is the hard part.
The constitution keeps one exception. The coding models can change on evidence, but the judge's model stays a person's decision. Otherwise the system could choose a judge that is easier to please. With new decision models arriving every few weeks, a challenger judge goes through the same bench as a challenger coder, scored on work where we already know the right answer.
A self-improving system optimises whatever it is told to, so the goal has to be written down as carefully as the rules. Ours: working software, at the lowest cost and time per story, with the fewest human interventions. The quality bar never moves, whatever that costs.
The quality bar lives in the constitution, not the scorecard, so it can never be traded for speed or cost. We think human interventions per story is the metric that matters most, and the one almost nobody measures. The point of a self-improving harness is not cheaper tokens. It is less of your people's time per piece of working software.
If you are building agents that change their own tooling, we would love to hear from you. What is in your constitution? What would you never let an agent change?
Thank you to Vercel: fx and the AI Gateway were a joy to build on, a fast scriptable coding agent, one key for every model, and a real cost on every call. With thanks to Steve Dickson, William McNulty and Lewis Crawford.