6 months, 527 AI code reviews, 4 agents

Every change we ship is written by one AI agent and read by agents from other vendors before it can merge. Sugra is built by a dozen agents that write, review and block each other’s merges, with no engineering team, so whether that reading earns its cost is not an academic question here. We argued about it, then stopped arguing and started writing every verdict down: who reviewed, with which model, what they found, how severe, and what happened to the finding afterwards.

One disclosure before any number. The practice has run six months; verdicts have been written to files for the last stretch of it. Every count below comes from 527 reviews recorded between 9 July and 15 August 2026. Older cases are real and cited, but they survive in commits and pull request comments rather than as records, so they are told as stories and never counted.

The headline is not about the reviewers. It is about the code they were handed.

61 of 227 changes - 27 percent - carried a serious defect after being written, tested, and believed finished.

Roughly one in four. Not drafts and not first attempts: changes whose author considered them done.

The second thing the files say is that the readers disagree with each other, and fast. One morning a change went up that carried a signed handoff between two of our own surfaces. Google’s Gemini read it and returned APPROVE_WITH_CHANGES - two cosmetic notes, ship it. Thirty-nine seconds later, on the same commit, OpenAI’s Codex returned BLOCK, with an exploit it had already run by hand:

GET /consent/token is a signing oracle: the cookie it reads is fully client-supplied, so anyone can send Cookie: sugra_cookie_consent=a0m1 and receive an HMAC-authenticated marketing grant to hand to a third party. Confirmed empirically with curl before review. The signature proves our server minted the token, not that the bearer consented.

Eleven minutes after that block, the next commit landed with the message drop the cross-domain consent handoff after review. The mechanism was not patched. It was deleted, and the change merged without it.

Two readers, one commit, thirty-nine seconds apart. One said ship it. The other proved the endpoint would mint a consent grant for anyone who asked.

First, the number the skeptic will quote

“Four out of five of those reviews are a waste of time.” Here is where that comes from, and it is worth stating before anything else.

Across 527 recorded reviews, here is what a single review produced:

More than half of all reviews returned an empty finding list. Add the cosmetic ones and 78 percent produced no finding that mattered. Quote that number first, then take the correction, because calling those reviews wasted is a category error.

A review does not produce findings. It produces a verdict about the state of a change, and “nothing here” is a verdict. A drug trial that records no adverse events did not fail - recording that is what the trial was for. A batch that clears inspection was not inspected for nothing. The empty reviews are confirmations, and the confirmations are most of what you are buying. You do not get to keep only the reviews that find something, any more than you get to run only the trials that will show an effect.

It holds on two conditions, both measurable.

A clean verdict is worth exactly what the reader’s ability to detect was worth. In this data the same model at a higher effort setting finds roughly twice as much, and in one month 47 percent of reviews ran under configurations that produced zero serious findings across the entire month. A negative result from a reader that could not have found the defect is not evidence that the code is clean. It is the absence of evidence, which is the same failure as a check that cannot fail.

Negatives only accumulate if they are independent. Four clean verdicts from readers that share a blind spot are worth about one clean verdict. That is the entire case for using different vendors, stated in the language of confirmation rather than of catching: independence is what turns a second “nothing here” into more confidence instead of an echo.

So the scoreboard is not 20 percent useful and 80 percent wasted. It is one review in five that changed the code, and four in five that told us, with a detection power we can quantify, there was nothing to change.

We paid for mistakes too. Of 581 findings, 72 were refuted with evidence. One reviewer fabricated two external URLs and reported our correct ones as broken; curl -I showed ours returning 200 and its inventions 404, on a path that had already pulled 23,280 records in production. Another reviewed the wrong pull request entirely and confidently described code that was not in the diff, three separate times - models are confidently wrong, and search does not fix it. A third assumed the worst-case behaviour of a helper without opening the helper, which produced four false criticals.

The reviews are not what is being audited here. The confidence is.

One honest weakness: severity is assigned by the reviewer that raises the finding, so the party whose value we are measuring grades its own work. The only counterweight is that every finding’s outcome is recorded afterwards, which is how 72 ended up marked refuted. Treat 27 percent as our accounting, not a neutral audit.

The defects were not stylistic. They included a forgeable consent grant, an account-takeover path, silently wrong numbers on a paid data API, a rate limiter that could be reset by an empty bucket, and a decoder that deleted other products’ last-good values when one product’s data was malformed.

Why a second pass by the same model is not a second opinion

The finding that surprised us most, and the one to keep even if you throw out the rest:

A reviewer from the same vendor as the author approves the code while stating something factually false about it.

Case one. A helper was supposed to push a blocking SEC network call off the event loop. It did this:

result = fn(*args)
await asyncio.to_thread(lambda: result)

The blocking call executes on the first line, on the event loop. The to_thread wraps a value that has already been computed. It is a no-op that type-checks, runs, and passes every functional test - all 31 of them stayed green, because test stubs return instantly and the stall only appears under real network latency.

Anthropic’s Claude, reviewing code written by Anthropic’s Claude, approved it with the words:

no blocking call slipped into the event loop

OpenAI’s Codex flagged it as MAJOR. The shipped fix dispatches the callable, not the result:

result = await asyncio.to_thread(fn, *args)

Case two. A fix computed an HTTP Retry-After as the whole seconds remaining until UTC midnight, using int(). Two separate Claude review passes approved it, and one of them explained that rounding down was the safe direction because it makes the client wait longer.

It does the opposite. Retry-After is a duration, not a deadline: a smaller number means the client retries sooner. Rounding down lands an honest client a fraction of a second before the rollover, while the old day’s counter is still over limit, and it eats one more avoidable 429. Google’s Gemini caught it. The shipped code rounds up:

seconds = (midnight - now).total_seconds()
return str(max(1, math.ceil(seconds)))

A second pass by the same model is not a second opinion. It is the same opinion, louder - and it can generate a confident justification for the defect it just missed.

You may have noticed which vendor is missing from the tables below. Claude is the author here, not the reviewer: it wrote 157 of the 227 changes, and the rule is that a change is reviewed by vendors other than the one that wrote it. Hence 10 review records against roughly 200 each for the others, and hence both cases above being Claude reading Claude, from before that rule was mechanised. How this fleet came to exist is its own story.

This is not a claim that Claude reviews badly. Any model reading its own vendor’s output shares that vendor’s blind spots, and all four do it here: Gemini missed findings Codex caught, Codex missed findings Gemini caught, Grok missed findings both caught. Claude simply had the most chances to demonstrate it. Across its 10 reviews - five on changes its own vendor wrote - it returned APPROVE every time, with zero serious findings.

Would one good reviewer have been enough?

No, and this is the number that decides it. There were 202 serious findings that survived scrutiny. Keep only one of the three reviewers:

Keep only	Serious findings it would have raised	Affected changes left with nothing
Gemini	106 of 202 (52.5%)	20 of 61
Codex	70 of 202 (34.7%)	34 of 61
Grok	26 of 202 (12.9%)	47 of 61

On 41 of the 61 affected changes, every serious finding came from exactly one vendor that no other vendor matched.

Read that as an upper bound on what you would lose, not a forecast: a reviewer running alone would get more rounds and more attention per change, so it would close some of the gap. What the table establishes is that the gap exists and is large.

There is no best reader. Each catches and misses: Gemini missed 10 findings Codex caught, Grok missed 8 Gemini caught, Codex missed 7 Gemini caught. The value is not a vote and not redundancy. It is coverage of blind spots that do not overlap.

It also happens fast. In 17 episodes across 11 changes, code one vendor had already cleared had a serious defect found by another on the same commit. Fourteen came within sixty seconds of the clean verdict; the tightest was one second.

These are not four passes in a queue

The obvious objection is that the table measures position, not skill: if one reader goes first and the next reads the cleaned-up result, the last one in the chain would find nothing because nothing is left.

The timestamps settle it. Reviewers are dispatched together against the same commit. The median spread inside a dispatch is seven seconds, and 168 of the 300 gaps between consecutive reviews of one change are under a minute. Nothing gets fixed in seven seconds; every reader in a batch sees identical bytes. Nor is anyone systematically last: across multi-reviewer batches, Gemini is dispatched first 42 percent of the time, Codex 37, Grok 43, and the share of each vendor’s reviews falling in the first round is 81, 68 and 75 percent. Grok’s low yield is a property of the reader, not of its place in a line.

Order does exist between rounds, and that is the harder catch: reading code that has already been fixed once. 12 findings had their defect in the fix for a previous finding - a hardening that survived the hardening, a round-one guard that still failed, a first attempt that turned out to be inert. The change that opens this article went six rounds: an ordering guard too coarse to separate two decisions in the same second, then a replacement that stamped when the request was handled rather than when the person clicked. Neither survived, and the mechanism was removed rather than repaired.

Which reader is best, and does the model version matter?

Less than the effort setting does, and that is the most useful thing in this data.

Model version	Reviews	Findings per review	Serious findings that stood	Critical	Wrong criticals
gemini-3.1-pro-high	110	1.34	64	14	8
grok-4.5	106	1.14	21	0	1
gpt-5.4	82	0.77	33	8	3
gemini-3.1-pro	39	0.74	9	3	4
gpt-5.6-sol	30	0.67	4	0	1
gpt-5.5	22	0.91	7	2	0
gpt-5.4-codex	15	1.33	10	0	0
gpt-5.1-codex-max	15	1.20	3	0	0
grok-4.5-heavy	12	1.25	5	1	0
gemini-3.5-flash-high	9	0.67	4	2	0
gemini-3.5-flash	8	0.38	2	1	0

Read the pairs, not the ranking. The same model at a higher effort setting roughly doubles serious findings per review, replicated three times:

• gemini-3.1-pro 0.23 per review, gemini-3.1-pro-high 0.58
• gemini-3.5-flash 0.25, gemini-3.5-flash-high 0.44
• grok-4.5 0.20, grok-4.5-heavy 0.42

The first two ran harder changes at high effort, so part of that gap is the work. The third is clean: the multi-agent heavy mode ran on an easier slice and still doubled the yield. If you take one operational decision from this article, take that one - turning up the effort on a reviewer you already run is cheaper than adding a vendor and buys more.

Two caveats before quoting the table. Model assignment was not random, so the raw ranking is not a leaderboard. And the label is self-reported by the tooling: 37 distinct label strings for about a dozen real models, and 67 reviews - 13 percent - carry a CLI version or an agent name instead of a model version.

What only this catches

Grouped by class, with one real example each. Every one of these was caught before it merged.

Forgeable authority. The consent signing oracle above. And an OAuth sign-in that auto-linked a Google account by email address without requiring email_verified, so registering an account against someone else’s unverified address linked you into theirs. Round three of a review, on code that had passed two rounds. Nobody would have needed to break anything: you do not get hacked, you get logged into.

Silently wrong numbers. This is the class that deserves a full example, because it is the one that never announces itself.

A solar position calculation measured azimuth from south instead of north. Both are real conventions; the older astronomical literature uses south, most modern geospatial work uses north, and code that mixes them produces an answer that is off by 180 degrees in a way that looks entirely reasonable. Every downstream value derived from that angle was wrong. Nothing raised, nothing crashed, no test failed, because there is no assertion you would naturally write that separates a correct sun position from one that is exactly backwards - both are plausible numbers in the right range with the right units, moving smoothly through the day.

That is invisible to every gate but one: a reader who knows the domain well enough to ask which convention the code assumes, then checks. Types will not catch it, tests written against the same assumption will not, and a reader checking that the code does what it says will not, because it says what it does perfectly.

Two more of the same shape: an interconnection map keyed on the wrong column of a live feed, so the from and to fields never attached to the flows they described; and lightning strikes counted for coordinates outside the observing satellite’s field of view. None of these fail. They return a plausible number, which on a data API is worse than an error, because an error gets noticed.

Guards that cannot fire. A workflow gate whose shell block, run under bash -e, printed nothing at all - a check that had been merged and was silently passing everything. A review gate that ran the pull request’s own version of the judge, so a pull request could tamper with the thing judging it and report green. A parser whose malformed entries were skipped in silence, so a BLOCK verdict could drop out of the count and let an older APPROVE authorise a merge. None of these fail loudly, which is the whole problem: a mismatch of 21 fields against 20 sat behind a wildcard for months before it took out 8.5 million machines in 78 minutes.

Money. A rate-limit cutover that took the wrong counter, so a key that had spent its daily quota could be refreshed by an empty account bucket. A checkout guard that cleared on a callback firing before the page navigates, so a second click on a slow connection opened a second Stripe session. Defects on a payment path are not theoretical: a single flag cost 440 million dollars.

Data destruction. A decoder that wrote all products into one blob: when one product’s variable was missing, the partial write overwrote the blob and deleted every other product’s last-good values.

The oldest example is the plainest. In April a review replaced a Redis KEYS scan with scan_iter and made a rate-limit increment atomic. KEYS blocks the whole server while it runs - on a shared instance with a growing keyspace, that is an outage with a scheduled arrival time.

Where it fails, plainly

Multi-model review substitutes for nothing, and treating it as the last gate is how you get hurt.

Four adversarial passes on one change found four real bugs and none noticed a missing Azure role assignment; the first deploy failed with an authorization error. Six rounds on another found nine real defects and all six missed a schema mismatch that returned a 500 on every request. That reason is structural: a review diffs your branch against main, not your code against the state of your live data, so anything that breaks only where code meets production state is invisible to it. On a third change, three bugs were found by production and nothing else - not the tests, not any reviewer. That is the class needing internal state reconstructed from external outputs, not another reader.

The sharpest miss in six months was our own merge gate.

A pipeline step was supposed to fail the build when a check did not pass. It ran under a shell option that aborts on the first failing command, and because of where the block’s exit code was read, that status never reached the step deciding the outcome. The gate printed nothing and reported success. On everything.

It passed review. It merged. Then it sat there approving every change that came past, because a gate that always says yes is indistinguishable from a gate with nothing to complain about. It was found only when a reviewer was pointed at the gate itself as the change under review, ran the block by hand, and saw it produced no output at all.

None of that is visible in a diff. Every line does what it says. The failure lives between a shell option and the place a status is read, and it appears only when you execute the thing instead of reading it. Reviewers that had just found real bugs in the same change read straight past it.

The rule we wrote down afterwards: a gate that cannot check must fail, not skip. A check that cannot fail is worse than no check at all, because it also removes the suspicion that would have made someone look.

And in the incident that made us rewrite a runbook, a reviewer dispatched read-only did the work instead: created a branch, applied a diff, opened a duplicate pull request and claimed the task. A reviewer with write access is not a reviewer - worth settling before you hand any agent the keys, which is what a team of always-on agents forces on everyone eventually.

What it costs, and what we would cut

The practice is effective. It is not efficient, and the same records show exactly where the loose spending is.

58 percent of all reviews - 307 of 527 - went to changes that turned out to have nothing wrong with them. That is the confirmation half of the ledger, not waste: you cannot know in advance which change is the one in four, and a regime that only ran where defects existed would not be verification.

What deserves attack is paying for readers whose confirmations mean little. Three such leaks are measurable.

The cost of a reviewer does not match its contribution. Our third vendor took 23 percent of the review spend and produced 13 percent of the serious findings. Not zero - it was the only reader on seven changes - but the ratio says reduce depth there, not breadth.

Risk tiering predicts correctly and then responds wrongly. The share of changes carrying a serious defect climbs from 17 percent at the lowest tier to 36 percent at the highest, so the classification works. But the cost per defect found gets worse as you climb - 7.9 reviews per defect in the middle tier against 10.6 in the top one - because our answer to “riskier” was “more readers” when the data says the cheaper lever is “better readers”.

Configuration drift is expensive and invisible. In one month, 47 percent of reviews ran under model configurations that produced zero serious findings for the entire month. Not an unlucky week: 147 reviews, and nearly half of them through readers that found nothing at all while a measurably better configuration existed and went unused.

So the verdict, in one line: keep the cross-vendor rule absolutely, spend on effort rather than on breadth, cap the rounds, and never let this be the last gate before production. And keep asking whether each rule earns its place, because a machine enforces whatever rule you hand it, perfectly, without ever asking whether it is a good one.

How we measured

Every verdict is a file committed to the repository carrying the change, at the head of its pull request: change id, reviewer’s pilot and model id, UTC timestamp, a verdict of APPROVE, APPROVE_WITH_CHANGES or BLOCK, a judgement on six fixed dimensions, and the findings. Each finding carries a severity from S1 to S4, a dimension, its evidence, and the required fix. The outcome is written back into the record afterwards, which is how a finding becomes refuted rather than merely disputed.

A change is reviewed by vendors other than the one that wrote it; how many depends on a risk tier, from one non-author reviewer at the lowest to all three plus a human at the highest.

The numbers come from 668 record files. A change spanning several repositories saves the same verdict once per repository, so collapsing those mirrors gives 527 distinct reviews over 227 changes, recorded between 9 July and 15 August 2026. Cross-vendor review started earlier - a second vendor from 3 May, a third from 15 June, a fourth from 12 July - and that period survives in commits and pull request comments rather than as records. “Serious” means an S1 or S2 whose record does not mark it refuted, declined, withdrawn or not reproducible; it never means we agreed with it, only that the record does not say otherwise.

We publish the method because the conclusion matters less than the measurement. Run more than one model over your own diffs, record the verdicts this way for a month, and you can answer the only question that counts for your own codebase: how often is your finished code actually finished.

For us it was one in four.