--- title: "Two of six" description: "We hid six bugs in one change and asked twelve reviewers to find them. GPT-6 Astra found two. GPT-5.6 Sol found all six. Turning the thinking setting up did not help." seo_title: "GPT-6 Astra found two of six bugs" seo_description: "We hid six bugs and asked twelve reviewers. GPT-6 Astra found two of the six. GPT-5.6 Sol found all six. Thinking longer did not help." slug: two-of-six status: published published_at: 2026-09-09 author: Arman Obosyan author_url: https://sugra.systems/about section: general primary_keyword: ai code review which model seeded defects flagship vs workhorse hero_image: /blog/images/posts/two-of-six-hero.jpg hero_alt: "Dark Field Notes cover: source code dissolving into two streams of light, a thin filament and a thicker amber ribbon. The headline reads Two of six." og_image: /blog/images/posts/two-of-six-hero.jpg tags: - ai - agents - engineering - code-review --- Every change we ship is written by one model and read by models from other companies before it can merge. There is no engineering team: most of the code is written with Fable 5.1, and Codex, Grok and Gemini sit on the other side of that gate. The question is which reader catches what the writer missed. In [August we published 527 of those reviews](/blog/527-ai-code-reviews). The cheap-looking lesson was to keep the reviewer you already run and turn the thinking setting up: same model, higher setting, roughly twice the serious findings. Three weeks later we hid six bugs in a small Python service. Twelve reviewers, one change, one run each. **GPT-6 Astra found two of the six. GPT-5.6 Sol found all six.** Spark found the same single bug at high thinking and at low. Grok, asked to think harder without its heavy mode, went from four of the six down to three. That is the whole argument. The rest of this piece is how we know, and where we almost lied to ourselves. ## How much of this we actually do From 10 March to 10 September 2026 the GitHub org merged **2,516 pull requests**. On product and docs that is **3,295 commits** and **1.53 million lines added**. [Kumar, Bararia and Raj](https://arxiv.org/abs/2606.15689) ranked models on 150 reviews. [SWE-bench](https://arxiv.org/abs/2310.06770) is 2,294 generation tasks harvested from other people's issues. Our merged-PR count is about seventeen times Kumar's sample. SWE-bench is a different kind of test, writing fixes rather than finding bugs, and even there the public set is 2,294 tasks against 2,516 live merges in six months of one company. This ranking is still twelve runs on one change. The volume is why a reader can treat it as coming from a machine that actually ships, not from a weekend notebook. The seats are Claude Max at 20x, ChatGPT Pro, and Grok Heavy. We do not drop to a cheaper model to stay inside the meter. Local `/usage` meters, different windows, volume rather than a bill: Claude Code **112.4 billion tokens** all time (favorite Opus 4.8; 108.8 billion of that is cache read), Codex **5.71 billion** lifetime, Grok Build **2.26 billion** for 1-10 September only, which is when that CLI started keeping files. ![Three usage cards: Claude Code /usage 112.4 billion all time, Codex /usage 5.71 billion lifetime, Grok Build 2.26 billion for 1 to 10 September](/blog/images/posts/two-of-six/token-volume.jpg) *Figure 1. Token volume on three seats. The windows are not the same length and are not a bill. Cache read is most of the Claude total.* ![GitHub merged PRs 2,516 against Kumar's 150 reviews, labelled 17x, then 2,516 live merges against SWE-bench's 2,294 harvested generation tasks, then 1.53 million lines added](/blog/images/posts/two-of-six/coding-volume.jpg) *Figure 2. Six months of shipping next to the public slices we cite. 2,516 live merges against 150 reviews, and against 2,294 harvested generation tasks.* ## Six bugs, on purpose A tiny service: parse a time window, fetch a series, cache it. Two files. Six defects, from the kind any reviewer should see to the kind that only shows up if you hold both files in your head. | tag | what we hid | | --- | --- | | A | a safety check runs after the thing it was meant to protect, so empty input throws the wrong error | | B | every retry is swallowed, and the empty result is stored as a success | | T0 | a timestamp of zero, which is a valid time, gets dropped as if it were missing | | TTL | the cache advertises a lifetime, and nothing reads it | | AL | the cache hands back its own list, so a caller can corrupt what everyone else sees | | NIE | a placeholder raises an error the retry loop does not catch | A seventh line was bait: `# TODO: optimise` on code that was already fine. Flagging it is a false alarm. Claude wrote this test and did not sit on the scoreboard. Production code is Fable 5.1. Each reviewer got one run. Codex, Grok and Google's Antigravity CLI (we call it Agy) do not all see the change the same way. Native Codex can open the repository. One Astra run was handed the diff as text and never opened the second file. Grok was given the text, then the same text under a JSON schema, then in heavy mode. Agy got the prompt on the command line. ## Who found them Twelve runs, three vendors. Codex takes six of the twelve slots because that roster was the one we could meter. The score is how many of the six hidden bugs they named. Extra bugs in the filler are real, and they have their own column: mixing them in is how a run gets quoted as nine findings on a test with six bugs. A zero in false alarms means we did not name one, not that the reviewer is precise. | vendor | model | seconds | of 6 | extras | false alarms | | --- | --- | --- | --- | --- | --- | | Codex | gpt-5.6-sol, native, high thinking | 275 | **6** | 0 | 0 | | Grok | grok-4.6, JSON schema | 44 | 4 | 1 | 1 | | Grok | grok-4.6 heavy, high thinking | 350 | 4 | 1 | 0 | | Agy | Gemini 3.8 Flash (High) | 71 | 4 | 1 | 0 | | Codex | gpt-5.6-luna, native, high thinking | 186 | 4 | 0 | 0 | | Grok | grok-4.6 default | 243 | 4 | 0 | 0 | | Codex | gpt-6-astra, our prompt, text only | 32 | 3 | 0 | 0 | | Grok | grok-4.6 high thinking, no heavy | 274 | 3 | 0 | 0 | | Codex | gpt-6-astra, native, high thinking | 58 | 2 | 0 | 0 | | Agy | Gemini 3.1 Pro (High) | 42 | 2 | 0 | 0 | | Codex | gpt-5.3-codex-spark, native, high thinking | 138 | 1 | 0 | 0 | | Codex | gpt-5.3-codex-spark, native, low thinking | 39 | 1 | 0 | 0 | Grok's schema run flagged a real whitespace bug we had not planted, and also the TODO. Heavy Grok and Flash each found one extra we had written ourselves. ![Horizontal bars grouped by vendor. OpenAI: gpt-5.6-sol native xhigh at 6, luna 4, astra stdin 3, astra native 2, spark xhigh 1, spark low 1. xAI grok-4.6 schema, heavy and default at 4, xhigh at 3. Google Gemini 3.8 Flash High at 4, Gemini 3.1 Pro High at 2](/blog/images/posts/two-of-six/planted-recall.jpg) *Figure 3. Of the six bugs we hid, how many each reviewer named. Same change, one run each.* GPT-5.6 Sol, through Codex's own reviewer, was the only OpenAI run that caught both of the two-file bugs: the safety check in the wrong place, and the cache handing out its own list. It caught all six. GPT-6 Astra is the model OpenAI currently calls most capable. Allowed to open the repo, it found two. Handed the diff as text, it found three, because it saw the hunk and the uncaught placeholder, and it never opened the other file. That is a difference in how we asked, not a second score for the same model. Spark found B, the swallowed retry, at both settings. 397,747 tokens at high thinking, against 88,953 for Sol on six bugs, both counted from the same local table. Spark never saw the hard bugs, so it cannot tell you whether thinking longer works. Google's names pointed the wrong way. Gemini 3.8 Flash found four of the six in 71 seconds; Gemini 3.1 Pro found two in 42. Flash is the fast class, Pro is supposed to be the strong one. Google has not shipped a newer general-purpose Pro since 3.1 Pro Preview, and the recent reasoning and coding releases have been Flash: 3.8 Flash is the one they call their best at that job. We had switched to Flash the day before because it was newest. Another session called that a downgrade because Flash sounds cheap. The 71-second run decided it. ## Who saw which bug ![Matrix of twelve reviewers against six hidden defects written in plain English: guard after the split, failure cached as success, timestamp 0 dropped, TTL never enforced, cache returns a shared list, placeholder not caught](/blog/images/posts/two-of-six/defect-matrix.jpg) *Figure 4. Amber is a hit. An open circle is a miss. Only the JSON-schema run flagged the TODO bait.* B is the easy one: every reviewer found it. A and AL are the ones that need two places in your head, and they are what separate the board. We almost published "nine findings in 44 seconds" for Grok's schema run. What we can still name is four of the six, one extra, one TODO, and some stubs whose text was never saved. The clock is real: 44 seconds against 243 for ordinary prose, and the answer arrives as JSON a merge gate can read. Default Grok also found four of the six, a different four. Schema changed which bugs it saw. It did not double the score. ## The lever from August ![Two groups of paired bars. Live pull requests: Gemini 3.1 Pro 0.23 vs 0.58, Gemini 3.5 Flash 0.25 vs 0.44, Grok 4.5 0.20 vs 0.42. This test: gpt-5.3-codex-spark 1 vs 1, grok-4.6 4 vs 3](/blog/images/posts/two-of-six/effort-did-nothing.jpg) *Figure 5. On live pull requests, more thinking roughly doubled serious findings. On these six bugs it did not.* August was observational. We pointed stronger settings at harder changes. The sentence we took to production was: "turning up the effort on a reviewer you already run is cheaper than adding a vendor and buys more." On this change Spark is a floor: one bug at high, one at low. Grok's default found four of the six; high thinking without subagents found three. Heavy mode added a fifth finding, and that fifth was an extra we had not planted. OpenAI, running their own reviewer, [wrote in December](https://alignment.openai.com/scaling-code-verification) that extra thinking mostly improved calibration rather than finding more bugs. That is the same knob, and it does not carry the August sentence onto this task. OpenAI list prices as of September 2026: Sol is $4 / $20 per million tokens, Astra $10 / $50. Sol used 16 percent more tokens than native Astra and is 2.5 times cheaper per token. Google and xAI are subscriptions, so there is no honest third column. ## When the change got large The small test is about 2 KB. We then buried the same six bugs in twelve pages of ordinary adapter code. The reviewer that could open the repository kept five of the six and missed only the placeholder. Grok, given the large diff as text, dropped from four of the six to two. [SWE-Review](https://arxiv.org/abs/2607.06065) says the same thing from the other direction: a diff is not enough evidence by itself. Let the reader see the tree. ## After 15 August On a change that could still email a user after they had opted out, succeeded, or been deleted, Codex returned BLOCK. Gemini, on the same first pass, returned APPROVE_WITH_CHANGES with nothing serious open. Thirty-nine seconds apart, one vendor stopped it and the other did not: the same split as the consent-grant case in the 527 post. A second clean verdict from the same company as the writer is still an echo. Two other notes, short: a history rebuild that could write 100 percent available for a day with no telemetry, and three bugs in one change that had been introduced by the previous round's fix. The test fake still passed 60 of 60 with the transaction dropped, so the fake was deleted. ## What we got wrong that afternoon Four of our own claims died the same day. We reported a broken review path that was not broken. The probe called an internal function. Production already did the conversion. The documented command, run as written, worked in 42 seconds. We recommended a flag the native reviewer accepts, exits zero on, and ignores. It returns its own format anyway. We read a run's token count while the run was still writing it, and published a number 32 percent low. We pointed four reviews at a stale local branch, so they read fourteen files from other people's merges as if they were ours. Three of the four were caught by the adversarial reviewer the measurement had just recommended, on its first real use. This is one change and one run each. Rankings on a single diff invert. Claude wrote the test. The large-diff comparison is two sizes, not a curve. Grok's "nine findings" are not on disk. The six bugs are synthetic. [Kumar, Bararia and Raj](https://arxiv.org/abs/2606.15689) already showed a smaller Claude beating a more capable sibling on review, and that recall falls as the diff grows. Repeat this ranking on real historical diffs before treating the default as settled. ## What we do now Switch the OpenAI reviewer to Sol, and write that measurement next to the default, so the next person cannot put Astra back because it is newer. Ask for structured output when a gate has to consume the verdict, and do not call that a miracle of finding bugs until the issue list is saved. Let the reviewer open the repository when the change is large. Keep the split: Fable 5.1 writes, a different company reads, and we spend on the model that can see the two-file bugs, not on the thinking knob of the one that cannot. The label on the box, the thinking slider, and the public leaderboard are poor substitutes for measuring the exact reviewer you actually run.