**One seeded bug, 26 AI agents: all passed the tests, all stayed broken**

> A 10-minute smoke test for AI coding agents. Five seeded Python bugs, one checker the agent sees and one it never does — five agents across two labs and three model tiers all pass the first and fail the same case in the second. - vyang472/five-bugs five-bugs 26 agents, from a 4-bit quantized 7B model up to Opus 5, were given the same one-line bug and its test suite. All 26 made the tests pass. All 26 stayed broken. They wrote four different regexes to do it: re . sub ( r"<\[^\>\]\*\>" , "" , html ) \# 14x Codex CLI, Opus 5, 4 repeats, 8 local re . sub ( r"<\[^\>\]+\>" , "" , html ) \# 8x Claude Code, Sonnet 5, 1 repeat, 5 local re . sub ( r"<.\*?\>" , "" , html ) \# 3x local re . sub ( r"<.+?\>" , "" , html ) \# 1x Haiku 4.5 All four turn the sentence a < b and c \> d into a d . Nobody was wrong to write them. The test suite they were handed contained only well-formed HTML, so that was the entire specification, and every one of them satisfied it exactly. Scaling the model across two orders of magnitude did not help, because the model was never the binding constraint. Handing one agent a better test fixed it on the first try. Verify that in two seconds, without running an agent or spending anything: git clone https://github.com/vyang472/five-bugs cd five-bugs python3 scripts/verify.py What this repo is Five single-file Python bugs. Each ships with the checker the agent is given, and a second checker, written afterwards, that no agent ever sees . The gap between those two is the whole experiment. Everything was run on one laptop: the tasks, the patch every agent actually wrote, raw timings, token counts and costs, and the scripts that produced them. Nothing here asks you to trust a number I typed. How the blind spot showed up Claude Code and Codex CLI both fixed all five bugs, 5/5 on the checker they were given. It looked like a tie and a clean sweep. Then I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again. task agent visible hidden t1 claude PASS PASS t1 codex PASS PASS t2 claude PASS PASS t2 codex PASS PASS t3 claude PASS PASS t3 codex PASS PASS t4 claude PASS FAIL t4 codex PASS FAIL t5 claude PASS PASS t5 codex PASS PASS Four of five held. On the fifth, both agents failed in exactly the same way. The bug was a greedy regex: re . sub ( r"<.+\>" , "" , html ) \# the seeded bug So I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and Opus 5. Five agent configurations, two labs, three model tiers. They wrote three different patterns: agent patch hidden Claude Code (default) <\[^\>\]+\> FAIL Codex CLI <\[^\>\]\*\> FAIL Haiku 4.5 <.+?\> FAIL Sonnet 5 <\[^\>\]+\> FAIL Opus 5 <\[^\>\]\*\> FAIL Three distinct fixes. One identical wrong answer: strip\_tags ( "a < b and c \> d" ) \# -\> "a d" (all five) None of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given, and all five satisfied it exactly. So I ran two more experiments to find out whether the model or the spec was the binding constraint. Is it reproducible, or did I get unlucky? Same task, same weak checker, same prompt, five more independent runs: run visible hidden pattern 1 PASS FAIL <\[^\>\]+\> 2 PASS FAIL <\[^\>\]\*\> 3 PASS FAIL <\[^\>\]\*\> 4 PASS FAIL <\[^\>\]\*\> 5 PASS FAIL <\[^\>\]\*\> Ten runs on this task tonight across two labs and three model tiers. Ten hidden failures. Not variance — a fixed point. What happens if I fix the spec instead of the model? I handed the agent the hidden checker as its visible one. Nothing else changed: same default model, same prompt, same one attempt. All five tasks passed, t4 included. It wrote: re . sub ( r"</?\[A-Za-z!?\]\[^\>\]\*\>" , "" , html...

**来源信息**
- **来源**：Hacker News：AI 热帖
- **分类**：ai-models
- **发布时间**：2026-09-16 09:48（北京时间）
- **原文**：[打开原文](https://github.com/vyang472/five-bugs)