Independent runs of Epoch AI’s MirrorCode evaluation will determine the maximum repository-scale software project current coding agents can complete with limited human intervention.

state: expiredheat: lowuncertainty: highconvergesscott: highcoding-agents agent-benchmarksEpoch AI

What is this?

MirrorCode is a long-horizon software-engineering benchmark built by Epoch AI and co-developed and funded with METR; one secondary snippet instead attributes a collaboration to Meta, but the supplied primary-adjacent evidence supports METR. It asks coding agents to autonomously reimplement target programs from observable behavior and tests, across 25 programs in areas such as Unix utilities, interpreters, bioinformatics, and cryptography. Preliminary results report that Claude Opus 4.6 rebuilt the roughly 16,000-line Go toolkit gotree, a task contributors estimated would take a skilled unaided engineer 2–17 weeks, though the snippets do not establish that independent replications have yet determined a general maximum project scale.

Why it matters to Scott

MirrorCode independently operationalizes several of Scott’s load-bearing positions: repository-scale capability should be measured through long-horizon execution, executable behavioral specifications, and observable test-based convergence rather than patch-level answers. Epoch AI’s benchmark creates a strong dated-receipts and validation opportunity for his long-running-agent architecture, although the supplied evidence does not yet show independent runs establishing a general maximum project scale.

ip:framework.long-running-agentsip:source.breaking-the-1hr-barrierip:concept.benchmarking-the-wrong-unitip:concept.characterisation-testingip:concept.measurable-convergencedev:concept.resumable-agent-job-control-planeradar:concept.coding-agentsradar:concept.long-horizon-agentsradar:concept.agent-evaluationradar:concept.agent-benchmarksradar:concept.benchmark-integrityradar:cursor-sqlite-doc-reconstruction

queries asked of Scott's wikis

repository-scale coding-agent evaluations
autonomous coding harnesses and human intervention
behavioral specifications versus source-code context
long-horizon agent reliability and inference scaling
benchmark validity for agent-completed software projects
measuring coding agents by task horizon

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (8) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	What's the largest software project AI can complete on its own?	yusufozkan	106	105
🟧 echo.blog ⭐	Introduces MirrorCode as an evaluation of the largest software projects AI agents can complete autonomously.	Epoch AI	—	—
🟧 hn	Ask HN: I had Codex and GPT 5.6 Sol running for 12 days, 870k+ LOC. Now what?	iagooar	1	2
🟧 hn	I Spent 2B Tokens Writing a C++ Compiler So You Don't Have To	lemming	2	0
🟠 reddit	Update on the dating app I built with Claude Code: 30 agents run the day to day now, I still make every call, and I printed 350 posters myself ClaudeAI	kyle77745	0	16
🟧 hn	LLM DeepSWE Pareto Frontier	ijidak	1	0
🟧 hn	Neal: Codex and Claude in a loop shipped a 549-commit migration	navels	1	0
🟠 reddit	I made Claude Souls ClaudeAI	vrdrift	1	3

Interpretation history

2026-08-15T17:29:59Z
Repeated checks have produced only unvalidated project anecdotes, with no independent MirrorCode run or published result narrowing the autonomous project-scale boundary. Expire the active episode and reopen if substantive benchmark results or replication appear.
2026-08-13T16:35:50Z
The agent-built native game adds another concrete but unvalidated project anecdote; without inspectable artifacts, intervention logs, or independent evaluation, it does not establish limited-intervention completion or narrow MirrorCode’s project-scale boundary. The case remains cold pending an independent MirrorCode run or substantive published benchmark results.
2026-08-13T16:23:51Z
evidence attached: reddit.post.1vndp67 — A concrete agent-built native game project offers weak but relevant evidence about the repository-scale software projects coding agents can complete.
2026-08-11T16:44:46Z
Repeated staleness adds no independent MirrorCode run or validated limited-intervention completion; practical anecdotes still do not establish an autonomous project-scale boundary, so the case remains cold pending published results.
2026-08-09T16:34:28Z
The staleness review and minor engagement increase add no independent MirrorCode run, validated limited-intervention completion, or evidence of an autonomous project-scale boundary. Keep the case cold while awaiting benchmark replication or substantive published results.
2026-08-07T16:26:18Z
The 549-commit migration further supports the practical plausibility of sustained repository-scale agent work, but the sparse report does not establish project complexity, quality, completeness, or limited human intervention. It is neither an independent MirrorCode run nor evidence of a maximum autonomous project scale, so the case remains cold pending validated benchmark results.
2026-08-07T16:21:39Z
evidence attached: hn.story.49212155 — A concrete 549-commit migration adds independent evidence about coding agents completing substantial repository-scale work.
2026-08-06T20:31:27Z
The added comments offer more practitioner anecdotes about large AI-assisted projects, but they still involve substantial human architecture or exhibit uncontrolled agent drift. No independent MirrorCode run or validated limited-intervention completion has appeared, so the case’s meaning remains unchanged and cold pending benchmark replication.
2026-08-06T19:22:08Z
The sparse DeepSWE frontier listing concerns capability and efficiency but does not test long-horizon repository completion, limited human intervention, or independently replicate MirrorCode. The case remains a promising benchmark hypothesis without evidence establishing an autonomous project-scale boundary.
2026-08-06T19:21:32Z
evidence attached: hn.story.49200656 — A DeepSWE Pareto-frontier result may provide relevant evidence about current coding-agent capability and efficiency, though the sparse listing limits confidence.
2026-08-05T09:25:20Z
No substantive evidence has appeared since the last review: the minor engagement increase adds neither an independent MirrorCode run nor a validated limited-intervention completion. Keep the case cold until replication or benchmark results establish an autonomous project-scale boundary.
2026-08-04T23:28:18Z
The latest activity adds no identifiable independent MirrorCode run or validated autonomous project completion; discussion continues to amplify repository-scale anecdotes without establishing limited-intervention capability or an upper bound. Repeated non-substantive updates now warrant cooling the case until replication or benchmark results appear.
2026-08-04T22:26:18Z
The 30-agent deployment adds another independent indication that repository-scale agent work is operationally feasible, but its creator still makes every consequential decision, underscoring that project size and agent count do not establish limited-intervention autonomy. No independent MirrorCode run or validated upper bound has appeared, so rising discussion increases attention without advancing maturity.
2026-08-04T22:21:31Z
evidence attached: reddit.post.1vfogeb — A real deployment reportedly has Claude Code coordinating 30 agents and maintaining a 70k-line project, materially contextualizing the scale and human-oversight limits of autonomous coding workflows.
2026-08-04T12:25:25Z
The compiler report adds a second independent anecdote that very large token budgets can sustain repository-scale agent work, but it provides no validation of completeness, quality, or limited human intervention. It therefore broadens practical plausibility without establishing MirrorCode’s maximum-project claim or resolving its generalization limits.
2026-08-04T12:21:35Z
evidence attached: hn.story.49167134 — Independent real-world report of a coding agent spending 2B tokens building a substantial compiler materially informs the frontier of autonomous project scope.
2026-08-04T01:22:00Z
No substantive new evidence is identifiable: the attached material still provides neither an independent MirrorCode run nor a validated completed-project result. The case remains promising but unchanged, with amplification outpacing evidence that could establish a maximum autonomous project scale.
2026-08-04T00:24:27Z
The latest change is only marginal engagement on existing discussion; no independent MirrorCode run or validated project completion has appeared. The case remains relevant but its meaning has not advanced beyond the benchmark’s initial promise and unresolved validity limits.
2026-08-03T21:23:18Z
No independent MirrorCode replication or validated completed-project result has appeared; the added activity remains anecdotal amplification and does not narrow the benchmark’s scope or validity uncertainty.
2026-08-03T20:28:04Z
The new long-running-agent report makes repository-scale autonomy more plausible in practice, but unvalidated LOC volume is not an independent MirrorCode run and does not establish completed-project scale. Growing discussion also sharpens the benchmark’s central validity question: reproducing behavior may not generalize to developing novel software.
2026-08-03T20:21:47Z
evidence attached: hn.story.49160339 — An independent real-world report of agents producing 870k lines over 12 days bears directly on repository-scale autonomous software development, though validation remains limited.
2026-08-03T17:24:36Z
grounded: converges/high — MirrorCode independently operationalizes several of Scott’s load-bearing positions: repository-scale capability should be measured through long-horizon executio
2026-08-03T17:22:16Z
case created — This is a bounded new coding-agent evaluation with a first-party artifact and enough discussion to merit follow-up.

Decision trace

08-16 03:30expireRepeated checks have produced only unvalidated project anecdotes, with no independent MirrorCode run or published result narrowing the autonomous project-scale boundary. Expire the active episode and
08-16 03:29alert_silentThis review contains no new consequential event, only continued staleness; Scott can wait until an independent run or substantive MirrorCode result appears.
08-16 03:29alert_routeThis review contains no new consequential event, only continued staleness; Scott can wait until an independent run or substantive MirrorCode result appears.
08-15 15:27drop_targetsquiet through full ladder or over cap 8
08-14 02:35repriceThe agent-built native game adds another concrete but unvalidated project anecdote; without inspectable artifacts, intervention logs, or independent evaluation, it does not establish limited-intervent
08-14 02:35alert_silentThis self-report is suggestive but does not materially change the benchmark hypothesis or create a time-sensitive decision for Scott; it can wait for inspectable results or independent validation.
08-14 02:35alert_routeThis self-report is suggestive but does not materially change the benchmark hypothesis or create a time-sensitive decision for Scott; it can wait for inspectable results or independent validation.
08-14 02:24alert_silentA low-engagement self-report of two agents producing a native C++ game prototype in roughly seven hours is suggestive but lacks inspectable code, executable evaluation results, intervention logs, or i
08-14 02:24alert_routeA low-engagement self-report of two agents producing a native C++ game prototype in roughly seven hours is suggestive but lacks inspectable code, executable evaluation results, intervention logs, or i
08-14 02:23attachA concrete agent-built native game project offers weak but relevant evidence about the repository-scale software projects coding agents can complete.
08-14 02:22propose_attachA concrete agent-built native game project offers weak but relevant evidence about the repository-scale software projects coding agents can complete.
08-12 02:44repriceRepeated staleness adds no independent MirrorCode run or validated limited-intervention completion; practical anecdotes still do not establish an autonomous project-scale boundary, so the case remains
08-12 02:44alert_silentNo new consequential event occurred; this is only a scheduled staleness check, so Scott can wait for an independent run or substantive benchmark result.
08-12 02:44alert_routeNo new consequential event occurred; this is only a scheduled staleness check, so Scott can wait for an independent run or substantive benchmark result.
08-10 02:34repriceThe staleness review and minor engagement increase add no independent MirrorCode run, validated limited-intervention completion, or evidence of an autonomous project-scale boundary. Keep the case cold
08-10 02:34alert_silentThe new delta is only repetitive engagement on existing evidence and does not change the benchmark’s status or Scott-relevant implications; it can wait for a substantive independent result.
08-10 02:34alert_routeThe new delta is only repetitive engagement on existing evidence and does not change the benchmark’s status or Scott-relevant implications; it can wait for a substantive independent result.
08-08 02:26repriceThe 549-commit migration further supports the practical plausibility of sustained repository-scale agent work, but the sparse report does not establish project complexity, quality, completeness, or li
08-08 02:21attachA concrete 549-commit migration adds independent evidence about coding agents completing substantial repository-scale work.
08-08 02:21propose_attachA concrete 549-commit migration adds independent evidence about coding agents completing substantial repository-scale work.
08-07 06:31repriceThe added comments offer more practitioner anecdotes about large AI-assisted projects, but they still involve substantial human architecture or exhibit uncontrolled agent drift. No independent MirrorC
08-07 06:20mark_dirtycomment_update
08-07 05:22repriceThe sparse DeepSWE frontier listing concerns capability and efficiency but does not test long-horizon repository completion, limited human intervention, or independently replicate MirrorCode. The case
08-07 05:21attachA DeepSWE Pareto-frontier result may provide relevant evidence about current coding-agent capability and efficiency, though the sparse listing limits confidence.
08-07 05:21propose_attachA DeepSWE Pareto-frontier result may provide relevant evidence about current coding-agent capability and efficiency, though the sparse listing limits confidence.
08-05 19:25repriceNo substantive evidence has appeared since the last review: the minor engagement increase adds neither an independent MirrorCode run nor a validated limited-intervention completion. Keep the case cold
08-05 19:20mark_dirtyengagement_update
08-05 09:28repriceThe latest activity adds no identifiable independent MirrorCode run or validated autonomous project completion; discussion continues to amplify repository-scale anecdotes without establishing limited-
08-05 09:20mark_dirtyengagement_update
08-05 09:20mark_dirtycomment_update
08-05 08:26repriceThe 30-agent deployment adds another independent indication that repository-scale agent work is operationally feasible, but its creator still makes every consequential decision, underscoring that proj
08-05 08:21attachA real deployment reportedly has Claude Code coordinating 30 agents and maintaining a 70k-line project, materially contextualizing the scale and human-oversight limits of autonomous coding workflows.
08-05 08:20propose_attachA real deployment reportedly has Claude Code coordinating 30 agents and maintaining a 70k-line project, materially contextualizing the scale and human-oversight limits of autonomous coding workflows.
08-04 22:25repriceThe compiler report adds a second independent anecdote that very large token budgets can sustain repository-scale agent work, but it provides no validation of completeness, quality, or limited human i
08-04 22:21attachIndependent real-world report of a coding agent spending 2B tokens building a substantial compiler materially informs the frontier of autonomous project scope.
08-04 22:21propose_attachIndependent real-world report of a coding agent spending 2B tokens building a substantial compiler materially informs the frontier of autonomous project scope.
08-04 11:22repriceNo substantive new evidence is identifiable: the attached material still provides neither an independent MirrorCode run nor a validated completed-project result. The case remains promising but unchang
08-04 11:20mark_dirtyengagement_update
08-04 10:24repriceThe latest change is only marginal engagement on existing discussion; no independent MirrorCode run or validated project completion has appeared. The case remains relevant but its meaning has not adva
08-04 10:20mark_dirtyengagement_update