The Northern Summer Frontier Models: Opus 5, GPT-5.6 and Kimi K3 on the Same Bench
Veli-Matti Vanamo
July 26, 2026
15 minutes to read
The Northern Summer frontier models
The frontier labs have shipped quickly these past few weeks. This is what the current wave costs to get a real feature merged, measured on the same bench as the last one, and why that bench now has to measure something it didn’t before.
The northern summer brought Claude Opus 5, GPT-5.6 in three context tiers, and Kimi K3, with Fable 5 not long before them. Enough of those are direct successors to models I’d already measured that it seemed worth turning on the benchmark to understand what changed beyond the model card and impressive SWEBench numbers.
Briefly, for anyone who hasn’t read the first piece: the bench hands one real ticket from my own backlog to a coding agent working on a frozen production repo, then holds the result to a local replica of the pull-request review gate we actually use, looping fixes until the gate passes it or the run gives up. What it reports is the cost and the time to a merged feature, not to a first draft, which is a different and considerably less flattering number. Cost is computed from tokens the same way for every arm, so models running under different command-line tools stay comparable.
The idea is: measure the time and cost to a real, merged feature in a real project - not synthetic benchmarks.
The results
Opus 5 appears twice in this chart. Once with the harness running its internal self-review, once without it.
With internal review on, Opus 5 produces the dearest merge I’ve measured on this feature, at $39.70. With that pass turned off it merges the same feature for $14.49, below Opus 4.8 and below Sonnet 5, and every one of those runs still cleared the gate on the first cycle with nothing blocking. Same model, same harness, same ticket, same reviewer, and 2.7 times the bill. More on that later.
Here’s the per-run full detail:
| Model | Harness | Review | Cyc | Coding $ | Opus PR $ | Total $ | Time | Gate |
|---|---|---|---|---|---|---|---|---|
| Opus 5 | Claude Code | on | 1 / 1 | 37.45 / 37.60 | 2.30 / 2.05 | 39.75 / 39.65 | 70 / 73m | Opus 5 |
| Opus 5 | Copilot | on | 1 / 1 | 30.00 / 28.04 | 2.87 / 2.86 | 32.87 / 30.90 | 81 / 79m | Opus 5 |
| Opus 5 | Claude Code | off | 1 / 1 / 1 | 15.90 / 11.36 / 9.17 | 2.12 / 2.52 / 2.39 | 18.02 / 13.88 / 11.57 | 31 / 33 / 29m | Opus 5 |
| Fable 5 | Claude Code | on | 1 | 36.93 | 2.51 | 39.44 | 49m | Opus 5 |
| Fable 5 | Claude Code | off | 2 | 21.19 | 3.70 | 24.89 | 38m | Opus 5 |
| Opus 4.8 | Claude Code | on | 2 | 20.31 | 3.01 | 23.32 | 51m | Opus 4.8 |
| Kimi K2.7 | Copilot | on | 1 / 1 | 8.65 / 12.94 | 1.23 / 1.85 | 9.88 / 14.80 | 77 / 174m | Opus 4.8 |
Opus 5 has outgrown the review pass I built for it
My production agent workflow has, in the past, put an explicit adversarial self-review into the loop, between the agent writing the code and the pull request being opened. I built that step because it did valuable work: the AI Developer would often ship half a feature, or miss the security boundary, and the self-review caught it before it reached the PR reviewer in GitHub. It was faster for the AI Agent to self-review in a loop, rather than having to do 5-15 rounds in GitHub PR, so I built self-review as de facto standard into my agents.md/claude.md loops, and created my own skills for doing it efficiently.
But the frontier appears to have moved. Opus 5 passes the PR gate on 1st go consistently, with the self-review pass running or not. Adding the self-review step on Opus 5 costs up to $25 more without meaningfully improving the outcome. To prove this, I ran five Opus 5 runs, three of them with the pass disabled, and every single one merged on the first cycle.
Opus 5 appears to critique itself as it works, at least when running in xhigh effort. Ask it to do it again, explicitly, as a separate skill, and it digs deeper, finding genuine enhancements - but ones that weren’t necessarily required to achieve the goal for a merge. Opus 5 demands higher quality when self-reviewing; all of the findings were genuinely good - but they were polish, enhancements and edge case protections. Arguably - they were scope creep.
It’s interesting to me that Fable 5 does not behave this way. In Fable 5 with the self-review pass disabled it did not clear the gate first time, it came back with two blocking findings and needed a second cycle. So the pass was doing real work for Fable and near enough none for Opus 5.
That makes this an Opus 5 observation rather than a general frontier model observation. I have not done loop optimisation or skill changes to account for the “harsh self-critic” behaviour yet, but I know I’ll be focusing on that in my production work. I’ve already been burned by the “Opus 5 is a harsh self-critic” in my production workflows outside the bench, so I know it’s not an isolated bench observation.
Let’s dig deeper.
What the review pass was actually doing
Turning it off cost roughly a third of the money and half the wall clock, and the gate couldn’t tell the two conditions apart. The obvious reading is that the pass is waste, and I held that view for a few hours, until I read the code.
The finding Opus 5 kept raising against itself concerned the endpoint that records a person approving a device.
It sits behind the repo’s ordinary requireAuth, and requireAuth tries a bearer personal access token before
it considers a browser session, so a token can perform a consent action that’s meant for a human.
Two of the three unreviewed runs shipped exactly that, and the third left a comment explaining why: “Mirrors server/api/oauth/authorize.post.ts”. It was right to. The existing consent endpoint in that repo, the one the ticket asks the model to follow, gates the same way, so the unreviewed runs were being faithful to the convention they’d been handed, and the reviewed runs added a rejection the incumbent endpoint doesn’t have.
The review wasn’t catching a mistake, then, but noticing that the pattern it had been given carries a weakness, and improving on it, which is a real contribution and worth something, though not progress towards a merged feature, since a review gate is supposed to pass code that matches the codebase’s conventions and mine correctly did. Cost to merge, the number this whole exercise is built around, can’t see convention improvement at all and simply books it as overhead. And… arguably - it is scope creep.
The review prompt can’t separate an auth weakness from a stale index
Here’s what each model does when handed the same self-review skill and pointed at its own work:
| Opus 4.8 | Fable 5 | Sonnet 5 | Opus 5 | |
|---|---|---|---|---|
| Self-review findings marked blocking | 4.0 | 4.0 | 5.5 | 10.5 |
| Share dismissed as merely low | 79% | 80% | 72% | 56% |
| Findings the gate blocked on | 1.0 | 0.0 | 0.0 | 0.0 |
This isn’t a capability gradient. Fable 5 sits on 4.0, level with Opus 4.8, and Opus 5 sits at more than double everything else. Fable produced identical figures under Claude Code and under Copilot, twenty findings and four blocking on both, which is about as clean a replication as this bench ever gives me. Opus 5 replicates too, at twenty-five findings and twelve blocking on Claude Code against twenty-nine and twelve on Copilot. Both numbers are solid, and nothing orders them by model strength. This is Opus 5 specifically, acting as a harsh critic of its own work.
Of the four findings Opus 5 ranked highest, three look genuine to me: the consent weakness, which it found twice, and a rate limit that would break a compliant client’s polling. The fourth was a documentation index that hadn’t been updated. The practical shape of the problem is that my review prompt was written for models that needed correcting, a stronger model applies it to itself with more force, and it can’t sort an authorisation weakness from a stale index, so you pay to fix both at the same priority.
Which suggests an enhancement I need to look into: the review skill should be scoped per model, not written once and pointed at whatever is driving. A descriptive skill, the kind that enumerates what to look for, suits a model that needs correcting. A goal-oriented one, anchored on what merging actually requires, suits a model that is already critical enough and needs bounding rather than prompting. Which of those a given model wants is evidently not something I can read off its capability tier, so I’ll have to measure it per model. One for the todo list.
What the northern summer models moved
The two Opus rows share a predecessor and a cycle count, and differ only in whether the self-review ran. Either way Opus 5 clears the gate in one cycle where 4.8 needed two; the pass decides whether that costs seventy per cent more than 4.8 or nearly forty per cent less. On capability Opus 5 looks like Fable 5 to me, both of them merging first time with nothing blocking, at half Fable’s token price.
GPT-5.6-terra costs about the same as GPT-5.5 and also drops a cycle. Kimi isn’t in this chart because K2.7 and K3 ran under different tools, and that turned out to matter more than the version, which is the next section.
Kimi K3 moves the needle, but the harness matters more now
Both Kimi models merge this feature, and the version bump costs a little more. On this ticket the bench can’t see what you get for that, since both land in the same place; the extra capability is something I’ve seen in normal workflows rather than something these runs measure.
| Model | Harness | Cyc | Coding $ | Opus PR $ | Total $ | Time | Diff |
|---|---|---|---|---|---|---|---|
| K2.7 | Copilot (native) | 1 / 1 | 8.65 / 12.94 | 1.23 / 1.85 | 9.88 / 14.80 | 77 / 174m | 2,276 / 2,484 |
| K3 | Kimi Code | 1 | 13.06 | 2.26 | 15.32 | 78m | 2,428 |
| K3 | Copilot (BYOK) | 3 | 13.53 | 5.96 | 19.49 | 121m | 7,311 |
K3 at its best costs about a quarter more than K2.7 averaged, merges first time exactly as K2.7 did, and produces a diff of almost identical size. It did that against the stricter of my two review gates, where K2.7 was judged by the older one, so if anything that gap is flattered towards K2.7. For a version bump on an open-weights model, paying a quarter more and keeping a first-cycle merge is a fair trade.
The interesting number is the last two rows. That’s the same model, the same provider, the same token prices and the same reviewer, and the only thing I changed is which command-line tool drove it. Under Kimi Code it merged first time with a 2,428 line diff. Under Copilot’s bring-your-own-key path it took three gate cycles and produced 7,311 lines, most of them work the reviewer then asked it to undo. The coding cost was within fifty cents either way, so the model did roughly the same amount of thinking. It just spent it differently.
Two settings explain most of that gap, and both are worth checking in whatever you drive your own agents with. The first is the context window the harness actually gives the model, which isn’t always the one the model supports: a tool that doesn’t recognise a model id will quietly fall back to a safe default, and a short-context agent on a large repo loses track and edits inconsistently. The second is whether the agent keeps its session between the review and the fix. If it doesn’t, every fix re-reads the diff cold with no memory of the review it’s answering, and you get exactly the churn above. Neither is visible in any output the tool gives you. Both look like the model being sloppy.
One more, if you route open-weights models through OpenRouter: a model name there isn’t one deployment. The K2.7 slug resolves to fourteen, at different quantisations and with output limits ranging from 16k tokens to 262k, picked per request on price and availability. You can be served a different one mid-session. Pin the provider in a preset before you conclude anything about a model, or you’re measuring a pool.
New frontier needs new workflows
If you’re picking between this wave’s models for real work, the spread between them is smaller than the spread between running one of them well and running it badly. That’s the part I didn’t expect.
I find that uncomfortable to publish, because the self-review step is mine and I’ve been recommending it to other people. But a workflow is a hypothesis about where the model is weak, and the models and the frameworks continue to move. It speaks to the whole point of why I built this bench for our internal work: we need to continuously measure the frontier, so we know when it moves enough for our harnesses, workflows and loops to evolve with them.
Method, harness and the full run data are in the original post. Every
figure here comes from results/run_data.jsonl, filtered to one challenge and one review style; the generating
script is docs/blog/figures/make_figures_northern_summer.py.
Two things to read the numbers against. The review gate changed model partway through this work, from Opus 4.8 to Opus 5, so the previous-wave rows were judged by the more forgiving of the two and the per-run tables name which gate saw what. Every comparison in the post runs the conservative way round, with the newer models facing the stricter reviewer. And the Kimi K3 harness comparison is one run each way.