Token Price Is the Wrong Number: What a Merged Feature Actually Costs Across a Dozen Coding Agents

Veli-Matti Vanamo
July 6, 2026

15 minutes to read

Tags: AIClaudeCodeGitHubCopilotOpenCodeOpusSonnetAgenticCodingBenchmarksTokenEconomicsEngineeringLeadership

The short version

I keep getting asked which model and which coding tool a team should actually adopt, and the honest answer is that the numbers everyone quotes don’t answer it. So I built an evaluation that measures the thing an engineering organisation actually spends: the total cost and total time to a merged, deployable feature, one that clears a real senior-review bar, not a generation that merely looks plausible.

One real feature. A dozen models. Three command-line tools (Claude Code, GitHub Copilot, OpenCode). Every arm held to the same production PR-review gate. Here is what came out of it:

• The same feature cost about $7 to about $70 to merge, and took from 26 minutes to nearly three hours, depending on the model and the harness. The token price was a minor part of that spread.
• The harness alone moved the cost by roughly 2.5×, same model, same feature, same gate. A good deal of that turned out to be a default reasoning-effort setting I’d left on.
• A cheaper model can cost more to merge, because its weaker output buys more rounds of paid senior review. The lowest-sticker-price model produced one of the most expensive merged features.
• Cost and time don’t rank the same. The fastest route to a mergeable pull request was not the cheapest one.
• “Merged” is not “correct.” A separate, deterministic test oracle caught the review gate happily merging code that passed as few as 12 of 38 of the team’s own unit tests.
• A single measurement is a hypothesis, not a fact. Most of the striking early results didn’t survive a second, careful run.

The rest of this post is how I set that up, the full results, the methodology in enough detail that you could rebuild it, and three experiments on where a strong model earns its keep. I intend to open-source the harness once it’s a little less temperamental.

---

The premise: measure the actual cost and time to a merged feature

It has been a busy few weeks in AI: Sonnet 5 and Fable 5 both landed, and a run of capable open-weight models with them. Releases arrive weekly now, and the benchmark scores cluster so tightly that a leaderboard barely tells them apart. So I set out to make sense of their capabilities the only way that means anything to me: by measuring what actually matters. What prompted it was mundane, though. The token cost of agentic coding keeps climbing across every harness, and with it the pressure to reach for cheaper models, or even locally run ones.

Almost everyone watches the token price, but to my mind it only tells part of the story. What matters to an engineering organisation is the total cost and the total time to a merged feature: complete, reviewed, deployable, ready to ship, and not a generation that looks plausible.

For me that was actually quite easy to measure. We already run an agentic software-development lifecycle in which the decision to merge is delegated to a trusted, automated pull-request reviewer, designed, refined and battle-tested over months in production. It runs on Opus 4.8, but the model underneath isn’t the point; the point is that I have a standing bar every change must clear, whether a person wrote it or an agent did. That gives me what I need to evaluate models and harnesses continuously, which is the other problem I wanted to solve. The frontier won’t sit still. New models arrive faster than we can adopt them, and the harnesses driving them (Claude Code, GitHub Copilot, OpenCode) move faster still; Claude Code alone ships several releases a day. So there’s a recurring decision for anyone running an engineering organisation: when is it worth changing the model or harness your engineers use every day?

Public benchmarks tell me very little here, and models and harnesses are known to optimise for them. SWE-bench Pro/Verified might report one model at 83% and another at 81%, but it isn’t obvious what those two points buy you in practice. Do your engineers ship sooner? Does a feature cost less to get to production, or take fewer rounds of review? A single percentile can’t tell you, because it measures capability on someone else’s tasks, once, and folds model, harness and workflow into one number.

There are easy ways to get lost in model evaluation, too, and I’ve been caught by both. The first is the constant news cycle of benchmarks, model cards and releases, which now cluster open-weight and frontier-lab models ever closer together. The second is quieter and more personal: the vibe of a model. Opus 4.8 gives me the vibe I want; it feels like a collaborator, and I reach for it far more than I should. A faster and cheaper path to a merged feature was there the whole time, in choosing the model for the job in front of me rather than defaulting to the one I like. But as an engineer, vibe was hard to shake, and only data convinced me.

I concluded I needed more information than a leaderboard could give me, and that the most reliable way to get it was to measure my own work directly.

What I set up

The design is deliberately simple. Take a real feature from my own backlog, an engineering ticket I have already specified, built and run in production, rather than a synthetic exercise. Because the finished version exists, there is a ground truth to compare against, and because the work is mine, it exercises my stack, my conventions and my definition of done. The headline challenge is an RFC 8628 OAuth 2.0 device-authorisation flow on a frozen Nuxt 4 and Drizzle repository, a genuine API surface with a real security boundary (one-time codes, server-side expiry, no replay, hashed secrets, auth at the grant point). It’s the kind of feature that should get bounced if it’s done carelessly.

Freeze the repository at the commit before the feature existed, with the history stripped back to a single commit so the agent cannot read the later solution out of the log. The agent begins where an engineer began: the codebase as it stood the day before, plus the ticket.

Give that same specification to each harness-and-model combination, in every case through the command-line interface, and hold everything else constant. The only variable under test is the pairing of model and harness.

Then measure the two figures an organisation actually spends to ship: the total cost to a merged feature, and the total time to a merged feature.

How the evaluation runs. One variable, the model-and-harness pairing, against a constant gate and, on a second challenge, a deterministic test oracle.

What “merged” means here

It’s worth being concrete about what that production reviewer actually checks, because it is what “merged” rests on. It looks holistically at security, modularity, reusability, documentation and ordinary code quality, and it judges whether the change meets the project’s rules, the architecture’s intent and the feature specification. It is closer to a senior reviewer than a linter, and it sets a high bar. The evaluation reuses that review, pinned to a constant model so that it serves as a fixed instrument.

The agent implements the feature, runs its own internal review, opens a pull request, and the Opus gate reviews it, requests changes, the agent fixes them, and the loop continues until the gate is satisfied, which counts as a merge, or until it gives up. A merged feature therefore means what it means in my real process: the full specification, faithfully implemented, having survived a genuine senior review. Every figure below is the cost, or the time, of clearing that bar.

One ruler for every arm

The major agentic coding harnesses have moved to usage-based, per-token billing, and an agentic run consumes a great many tokens, so the bill has created real pressure to reach for cheaper models, to route through a broker such as OpenRouter, or to self-host on your own hardware. Each is reasonable if the per-token figure is the one that matters. It usually isn’t, because the per-token price is the cost of a generation, not of a shipped feature.

Different CLIs also report cost differently: Claude reports a billed, cache-aware dollar figure, OpenCode reports none, and Copilot bills in “AI credits.” To compare them you need one instrument, so I compute cost harness-agnostically from tokens × published rates (cache-aware: input, cached-read and output priced separately). Self-hosted and local models are a real, priced $0; the hardware is capex. I then split every run’s cost into two buckets, which is the insight that makes the results legible:

• coding $: everything the implementer does (implementation, internal review and all fixes).
• Opus PR $: the constant gate (the external review cycles).
• total $: the two summed.

That separates “what the coding agent costs” from “what the review instrument costs,” and it’s what surfaces the patterns below.

The results

Across three harnesses, a dozen models and three internal-review styles, the table has twenty-three model-and-harness arms and twenty-nine runs in all, since several cells were run more than once. “Cyc” is the number of external review-and-fix cycles needed to merge.

Model	Harness	Review	Cyc	Coding $	Opus PR $	Total $	Time
Sonnet 4.6	Claude Code	reflect	1	7.02 / 8.46	4.78 / 3.54	11.80 / 12.00	43 / 51m
Sonnet 4.6	Claude Code	skill	1	8.68	3.31	11.99	53m
Sonnet 4.6	Claude Code	off	3	6.52	8.63	15.15	36m
Sonnet 4.6	Copilot	skill	2	4.93	6.81	11.74	88m
Sonnet 4.6	Copilot	reflect	2	8.66	8.17	16.83	84m
Sonnet 4.6	OpenCode	skill	2	20.63	8.74	29.38	65m
Sonnet 5	Claude Code	skill	1	19.05 / 18.71	3.99 / 4.58	23.04 / 23.29	72 / 71m
Sonnet 5	Copilot	skill	1	9.19 / 6.45	4.96 / 4.21	14.15 / 10.65	107 / 49m
Opus 4.8	Claude Code	reflect	1	27.04	2.89	29.93	26m
Opus 4.8	Claude Code	off	1	28.37	4.13	32.49	28m
Opus 4.8	Claude Code	skill	2	60.94	9.03	69.97	51m
Fable 5	Claude Code	skill	1	32.08	3.90	35.98	48m
Fable 5	Copilot	skill	1	18.90	4.93	23.82	38m
GPT-5.5	Copilot	skill	1 / 2	7.98 / 4.94	4.42 / 8.10	12.40 / 13.04	83 / 58m
GPT-5.4-mini	Copilot	skill	2 / 1	1.34 / 1.11	5.62 / 4.45	6.96 / 5.56	52 / 56m
Gemini 3.5 Flash	Copilot	skill	5	1.42	16.78	18.20	55m
MAI-Code-1-Flash	Copilot	skill	2 / 4	0.05 / 0.08	8.27 / 16.95	8.32 / 17.03	78 / 90m
MiniMax-M3	Claude Code (ccr)	reflect	3	10.27	13.51	23.78	73m
MiniMax-M3	Claude Code (ccr)	skill	2	12.55	7.65	20.20	105m
MiniMax-M3	OpenCode	skill	5	5.98	15.90	21.88	88m
Kimi-K2.7	Claude Code (ccr)	reflect	3	18.08	10.94	29.02	84m
Qwen3-Coder-Next	Claude Code (ccr)	reflect	3	8.22	13.39	21.61	69m
GLM-5.2	Claude Code (ccr)	reflect	1	13.34	4.82	18.16	176m

Cost is computed the same way for every arm, so the three command-line tools are comparable even though each reports cost differently, or not at all. The Copilot arms are priced at the underlying model’s token rates. The exception is MAI-Code-1-Flash, which is Copilot-exclusive and has no published token rate, so it’s priced directly in AI credits at premium_requests × $0.01. “reflect” and “skill” are two internal-review styles, discussed later, and “off” removes the internal review. None of these figures appears on a public benchmark, and each is the kind of thing worth knowing before asking engineers to change tools.

Cost to the same merged feature, split into coding cost and the constant review gate. The spread is roughly ten-fold.

The harness moves the cost, on the same model

Holding the model and the task constant and varying only the command-line tool, the cost of a merged feature moves by roughly 2.5×. Sonnet 4.6 merges for $11.99 on Claude Code, for $11.74 to $16.83 on Copilot, and for $29.38 on OpenCode, same model, same feature, same gate. With nothing else changed, the difference is the harness: its system prompts, tool surfaces, turn efficiency and how much it churns before converging (the OpenCode figure rests on a single run). Choosing a harness from a model leaderboard would not surface any of this, because the model is the constant.

The “harness tax”: same model, same feature, cost moved by the tool alone.

The same pattern shows up on the newer frontier models. Sonnet 5 merges the device-auth feature for around $12 on Copilot against about $23 on Claude Code, and Fable 5 for $23.82 against $35.98. Because I compute cost at identical per-token rates for both tools, a gap like that isn’t a price difference but a token-count difference: Claude Code simply spends more tokens to reach the same merge.

I’d rather offer a thesis than pretend to certainty, so I ran a small test. The cleanest cut is the implement phase, which is the same task on both harnesses. There, for Sonnet 5, output per turn is almost constant, around 650 to 680 tokens, whether I run Claude Code or Copilot and whether I set the reasoning effort high or low. So the gap isn’t about how hard the model thinks on each step. It’s about two other things: how many turns the agent takes, and how much context it carries on each one. Claude Code’s default runs hotter on both.

The test was to pin Claude Code’s effort to low and re-run the same implement. It nearly halved the cost, from about $10 to $5.24, by cutting the turns from roughly 200 to 129 and the context carried with them, and the feature still merged, in two external cycles. That lands Claude Code in Copilot’s default cost range. So most of the gap looks like an effort setting I’d left on its default rather than an immovable property of the tool, though a structural residual remains: even at low effort Claude Code carried more context per turn. The honest caveats are the usual ones: this is a single low-effort run, I haven’t yet checked whether low effort costs correctness rather than only money, and both harnesses self-report their own token usage. But the direction is clear enough to act on, and it’s the same lesson as the model choice: the default is rarely the cost-optimal setting, and no one is charging you for noticing.

A note on OpenCode, because it’s why the evaluation settled on two harnesses rather than three. What drew me to it was the premise: a genuinely open, model-agnostic harness looked like the natural home for self-hosted open-weight models. The early runs didn’t bear that out. For the open-weight models it was no cheaper than the routes I already knew, it took more review cycles to converge, and it was the most expensive harness for the same model (Sonnet 4.6 at $29.38 against $11.99 on Claude Code). I want to be transparent that this is not a verdict on OpenCode: my workflow is shaped around the other two, I haven’t used it in anger, and its showing here may reflect my unfamiliarity as much as the tool. It warrants a proper look later.

A cheaper model can cost more per merged feature

Of all the findings, this one bears most directly on the habit of reading the per-token price.

Model (Copilot)	Coding $	Opus gate $	Cycles
GPT-5.5	$7.98	$4.42	1
GPT-5.4-mini	$1.34	$5.62	2
Gemini 3.5 Flash	$1.42	$16.78	5
MAI-Code-1-Flash	~$0.05	$8.27-16.95	2-4

As the coder gets cheaper, its output needs more review, so the gate pays more to bring it up to the bar. MAI shows this at its extreme: its coding is close to free at around five cents, so its whole cost to merge is the Opus gate. Because its output takes several rounds to satisfy a senior review, the gate cost ran from $8 to $17; on the dearer of its two runs the gate alone, $16.95, was more than Sonnet 4.6’s entire merged feature on Claude Code. The model with the lowest sticker price produced one of the more expensive merged features. I wouldn’t read too much into the precise ordering on two runs, where GPT-5.5 and GPT-5.4-mini overlap, but Gemini Flash is a clear high-gate case, and the general pattern holds: the per-token price and the cost to merge can point in opposite directions.

Cost shifting off the coder and onto the gate as the model gets cheaper.

Cost and time do not rank the models the same way

The times are worth reading as a separate column, because the fastest merge and the cheapest merge are not the same run. The quickest in the matrix is Opus 4.8 on Claude Code with reflect, at 26 minutes and one cycle: the frontier model is expensive per token but converges in a single cycle, so it reaches a mergeable pull request faster than any other arm, at around $30. Sonnet 4.6 sits between about 36 and 90 minutes depending on the harness and the review style; Fable 5 merges in 38 minutes on Copilot but 48 on Claude Code; Sonnet 5 ranges from 49 to 107 across its runs. The open-weight arms cluster at the slow end, 69 to 176 minutes. If the constraint is cost per feature, Sonnet on Claude Code is the choice; if it’s getting a change in front of a reviewer as soon as possible, a frontier model on a fast harness is. There is no single best combination, only a cost axis and a time axis, and an organisation sits at a particular point on each.

Cost against time. The two axes don’t agree on a winner.

Open-weight models: cheaper APIs, or your own hardware

Open-weight models have been getting steadily better at agentic coding, and that matters here because of where they can run. You can reach them through a cheaper provider API such as OpenRouter or Azure AI Foundry, or you can host them yourself. Workstations with enough memory to hold a capable model are starting to arrive, and many organisations already run GPU clusters. So for an open-weight model the interesting question is not only how well it codes, but where you’d run it and what that would cost.

For this evaluation I ran the open-weight arms through OpenRouter, driven by two harnesses, OpenCode directly and Claude Code through a router (ccr) pointed at the same endpoint, and priced them at OpenRouter’s published rates. The computed coding cost ran from about $6 to $18 per run (the actual invoice was around $8 to $10), and with the Opus gate on top the totals came to $18 to $29, over 69 to 176 minutes. All four merged. The models were GLM-5.2, MiniMax-M3, Kimi-K2.7 and Qwen3-Coder-Next; these sit a step below the frontier commercial models, and each still cleared my senior-review bar on a real feature, which is worth noting in its own right.

GLM-5.2 is the one to call out. It gave the strongest open-weight result on cost and review cycles, merging in a single external cycle at $18.16, in the same range as the commercial mid-tier though not cheaper than it. It was, however, the slowest run in the group at 176 minutes, and that slowness looks more like a property of this particular path than of the model. GLM is reported to run considerably better through Z.ai’s own Anthropic-compatible endpoint than through OpenRouter and the router I used, and I didn’t test that endpoint for this piece. It’s near the top of the list to measure directly.

Open-weight models, rented. In the cost range of the commercial mid-tier, but slow.

What if the tokens were free?

The question underneath all of this is simpler than the rental figures: what if the tokens cost nothing? If you self-host Kimi, Qwen or GLM on hardware you already own, and you conveniently forget what that hardware amortises to, the coding cost falls to zero. So set it there and see what’s left.

What’s left is the gate and the clock, and neither falls out with the tokens. The external Opus review is a constant, paid instrument on every arm, and for the weaker models it’s the part that runs up, because they need more rounds to satisfy it. Zero the tokens and Qwen still costs $13.39 in review alone over 69 minutes, Kimi $10.94 over 84, MiniMax-M3 $7.65 over 105; GLM is the exception at $4.82, having merged in a single cycle, but it took 176 minutes to do it. Qwen’s review bill, with the model itself free, is higher than Sonnet 4.6 costs in total on Claude Code ($11.99, tokens included), and Sonnet reached the merge faster. You stop paying for the model and start paying for its mistakes, in review cycles or in wall-clock, and often both. The token price was never the thing to optimise. Total cost and time to a merged feature were.

Whether a merged feature is also a correct one

There is a fair objection to all of the above: the definition of “merged” rests on one instrument, the Opus gate, which for all its rigour is still a model’s judgement. I trust it, since it’s the same gate that decides what ships for us in production, but I didn’t want to lean the whole argument on a single model’s opinion.

So I built a second challenge, on a different frozen repo, with a feature whose behaviour can be graded objectively, not by another model, but by the team’s own deterministic unit tests, run against the agent’s code after the fact. Pass or fail against the interface contract; no opinion to negotiate with. Those 38 tests check behavioural invariants such as a double-count guard on billing, a security invariant that requires an instance identifier, and byte-identical output on re-encoding. Because the tests are deterministic, they can be re-run against any commit in a run’s history for free, with no model calls at all.

The gate and the objective tests disagree, and the disagreement is the point. The Opus gate merged implementations across a wide range of objective pass-rates, several of them low: on the complex feature the tests separate the models clearly: Opus 30/38, GPT-5.5 18/38, Sonnet 16/38, while the gate merged all of them in a single cycle. An LLM reviewer alone will approve code the team’s own CI would fail. And re-scoring every commit in a run shows why the pairing matters: the review-and-merge loop is close to fidelity-neutral. The objective score at the raw implementation commit equals the score at the final merged commit, within a test or two, across about thirty runs. The review cycles change the code so that it passes review; they do not move the behavioural tests.

That’s not a knock on the gate. It’s good at exactly what a holistic review should do (structure, security posture, documentation, readability), and it improves all of those things the tests can’t see. It simply isn’t a correctness oracle, and shouldn’t be relied on as one. Cost to merge needs the gate; whether the specification was implemented correctly needs the tests; and you only learn where they part company by running both and watching them disagree. That variability in what an agentic review will let through is real, and closing it is on my list.

The gate merges across a wide band of objective pass-rates, and barely moves the score. “Merged” and “correct” are different questions.

Three experiments in where you put the strong model

A question that comes up whenever you assemble an agentic workflow is how much review to build into the loop, how strong the models running it should be, and where the strong one sits, reviewing the work after it’s written, or planning it before. I ran three experiments on that, each measured against the cost figures above and, where the question is about quality rather than cost, the objective tests from the previous section.

One: does an internal review earn its place? The first holds the model fixed, the same cheap model both codes and reviews, and varies only the internal review the agent runs on its own work before opening the pull request. Three styles: “skill” is a structured, adversarial self-review that produces a findings verdict; “reflect” is a looser pass in which the model re-reads its own diff against the bar and improves it; “off” is no internal review. The review earns its place. For capable models skill costs about the same as or less than reflect, and both cost less than off: Sonnet merges in one external cycle with an internal review, against three without one. The economics are simple: a pre-emptive self-fix is cheap, whereas a fix the external Opus gate has to draw out costs a full review cycle at Opus rates. A good internal review converts expensive external cycles into cheap internal ones.

Two: does a stronger reviewer do better? If a self-review helps, does making the reviewer a strong model help more? So I kept the cheap model as the coder and had Opus 4.8 run the internal review, on the objective-test challenge where the effect on correctness is visible. The data splits the premise in two. On convergence it’s partly true, and it depends on the coder: Opus-as-reviewer cut Sonnet 5 from six external cycles to two, for a cheaper run; but for a model that already converged on its own, it only added Opus’s cost. On the result itself it did not hold: the objective score was essentially unchanged whether the reviewer was the cheap model or Opus. This is the fidelity-neutral finding again: a stronger reviewer changes what passes review, and how quickly, but not what passes the tests, because the coder’s own implementation sets the fidelity. A strong reviewer buys cheaper convergence; it doesn’t buy correctness.

Three: does a stronger planner do better? The third experiment puts the strong model as early as it can go: as the architect, planning the work up front, with a cheap model implementing against the plan and reviewing itself, and the external gate unchanged. A plan shapes the code that gets written rather than critiquing code already written, so it could lift a cheap coder’s output where post-hoc review did not. Whether it does is genuinely coder-dependent. The effect tracks how strong the coder is, and the clearest version is at the top: Sonnet 5, reviewing its own work, scored around twelve out of thirty-eight on most runs, code that passed the Opus gate but failed most of the objective tests, the merged-but-not-correct gap in miniature. With an Opus plan in front of it, the same model landed a steady twenty-seven to twenty-nine across four runs on both harnesses, while its own unaided score swung anywhere from eleven to thirty-three. So the plan does two things for a capable coder: it lifts the objective score by a wide margin, and it removes the variance, turning a coin-flip into a reliable result. The mid-tier coders gain a smaller two or three points, and at the weak end there’s nothing to report: the score swung by fourteen points from one run to the next, with and without a plan alike, too noisy to call.

The plan isn’t free: it adds the Opus planning fee, a few dollars a run, which for the cheapest coders is most of the bill. Set against experiment two, the two placements do different jobs. As a reviewer the strong model reliably buys cheaper convergence but not correctness; as an architect it buys a capable coder a markedly steadier and more correct result at a premium, and buys a weak one nothing. Neither is the across-the-board win a workflow diagram implies, because both sit downstream of the constant Opus gate that already does the heavy quality work. If there’s a rule in it, it’s that a plan is worth its premium only for a coder strong enough to execute it; below that line, the strong model does more good reviewing the work than planning it.

Where the strong model earns its keep, as reviewer, and as architect.

A measurement is a hypothesis, not a fact

The same pace of change carries a warning about reading any of these numbers, mine included: a single measurement is a hypothesis, not a fact, and the more striking it is the more caution it warrants. The models and harnesses underneath keep moving. A freshly released model’s objective score swung between eleven and thirty-one out of thirty-eight across runs I couldn’t otherwise control, and the command-line tools ship new output shapes several times a day. Harness updates shuffle the deck or quietly break a review skill between one run and the next. On ground that unstable, a number that runs once has nothing to disagree with it. The findings I’d stand on here are the ones that survived being run again, on a fresh checkout, more than once: that the gate doesn’t buy fidelity, that the harness moves the cost of the same model, that a cheaper model can cost more to merge. To trust a result at all I had to run weekly sets of repeated iterations and account for the volatility. The striking one-off is usually variance.

That is the real case for owning the instrument rather than reading a leaderboard. A leaderboard runs once and reports a point; it can’t tell you that the model or the harness it measured has moved since, or that the result wouldn’t repeat. One you run yourself, on your own work and more than once, can. Distrust the striking result, and keep the one that holds up.

Building one for your own context

You don’t need my repository, only your own work. The approach:

1. Choose a feature you’ve already shipped, so you have ground truth and it reflects your kind of work.
2. Freeze the repository at the commit before it, with history stripped so the solution can’t leak.
3. Reuse your real merge bar as the gate, pinned to a constant reviewer, so “merged” means what it means in your process.
4. Give the same specification to each harness and model you’re weighing, through their command-line tools.
5. Measure cost and time to merge, computed the same way for every arm from token counts, split into coding cost and gate cost, and read them as two axes rather than one.
6. Add a second, objective instrument, your own tests, for the questions the gate can’t answer, and score at the commit level to see whether the review cycles are buying fidelity or only approval.
7. For open-weight models, price the rental through a provider and separately consider self-hosting, weighing model size against your hardware and the capability your features require.
8. Guard against silent failures, and replicate before you believe a result, particularly a striking one. Most of mine didn’t survive that step.
9. Report every run, and re-run over time, since the frontier you measured last month has already moved.

The output isn’t a leaderboard. It’s a statement you can act on: for my kind of feature, a given model on a given harness merges in so many cycles, at a certain cost and time, past my real review bar, and here is how much that changed when I re-ran it. That’s the substance of an adoption decision, and the only way to obtain it is to measure your own work.

One closing, practical note. The harness still has a few dependencies on internal applications and repositories baked in, and it’s a little temperamental besides. I’m working to strip those out and steady it so it stands on its own, and I intend to release it, open source, on my personal GitHub for anyone who’d like to run it against their own work or simply inspect how it does what it does.

Share on Twitter (X) Share on LinkedIn RSS Feed