Vero evaluations will determine whether AI agents can autonomously produce complete software repositories whose required behavior is established through machine-checked formal verification.
state: expiredheat: lowuncertainty: mediumconvergesscott: highcoding-agents formal-verification agent-harnessesVeroVerina
What is this?
Vero is presented as a repository-level benchmark for evaluating whether AI coding agents can jointly synthesize software implementations and machine-checked proofs that the implementations satisfy their specifications. Unlike function-level evaluations, it uses multi-module repository tasks intended to test cross-module dependencies and long-horizon reasoning. The supplied snippets do not identify Vero’s authors or establish Verina’s role, and Vero should not be confused with Scale Labs’ separately described VeRO agent-optimization harness.
Why it matters to Scott
Vero independently operationalizes Scott’s Evaluation-Driven Development and Spec-Driven Development positions by testing whether agents can turn machine-checkable specifications into repository-scale implementations with accompanying proofs. Its results could validate or constrain his load-bearing claim that generated code can become a replaceable derived artifact when governed by binding verification, while extending radar coverage beyond separate repository-scale coding and proof-synthesis evaluations.
ip:concept.evaluation-driven-developmentip:concept.spec-driven-developmentip:concept.verification-loopsip:concept.spec-as-assetradar:concept.coding-agent-benchmarksradar:concept.formal-verificationradar:mirrorcode-autonomous-project-scoperadar:mathcode-mathematical-coding-agent
queries asked of Scott's wikis
- repository-level coding-agent evaluation harnesses
- formal verification as an agent-generated correctness layer
- joint code and proof synthesis
- machine-checkable specifications for autonomous coding
- long-horizon multi-module agent benchmarks
- verified software generation and proof-carrying code
Measured heat
no measured readings yet — the hourly heat pass fills this in
How the heat travelled
no chain yet — the hourly chain pass fills this in
Evidence (2) — ⭐ canonical anchor
| source | object | author | score | comments |
|---|
| 🟧 hn | Vero: Can AI Agents Build Formally Verified Software Repositories? | matt_d | 1 | 0 |
| 🟧 echo.paper ⭐ | The original paper introduces Vero as “the first benchmark to evaluate joint implementation and proof synthesis at the repository level,” us | Zhe Ye, Hantao Lou, Yuechun Sun, Peiyang Song, Zhengxu Yan, Timothe Kasriel, Qingyang Zhang, Kaiyu Yang, Soonho Kong, Jingxuan He, and Dawn Song | — | — |
Interpretation history
2026-08-25T06:31:34Z
Vero remains a concrete released benchmark, but no replication, adoption, or new results have appeared to develop the case beyond its original self-reported package; ongoing monitoring no longer earns attention until substantive validation surfaces.
2026-08-23T05:29:58Z
Vero merits watching because the benchmark, curation pipeline, evaluation harness, and repository-level results form a concrete released package, but nothing new independently validates the reported capability. This reobservation adds no development beyond the original release.
2026-08-23T05:28:56Z
grounded: converges/high — Vero independently operationalizes Scott’s Evaluation-Driven Development and Spec-Driven Development positions by testing whether agents can turn machine-checka
2026-08-23T05:27:22Z
origin walked (codex/luna, conf 0.98): anchor hn.story.49406158 -> echo.paper.66153d7ce8 by Zhe Ye, Hantao Lou, Yuechun Sun, Peiyang Song, Zhengxu Yan, Timothe Kasriel, Qingyang Zhang, Kaiyu Yang, Soonho Kong, Jingxuan He, and Dawn Song
2026-08-23T05:26:16Z
case created — The released project defines a bounded and checkable agent-software capability claim that goes beyond ordinary test-based code generation.
Decision trace
- 08-25 16:31expireVero remains a concrete released benchmark, but no replication, adoption, or new results have appeared to develop the case beyond its original self-reported package; ongoing monitoring no longer earns
- 08-25 16:31alert_silentThe staleness trigger carries no new evidence or consequential event; any independent replication, external implementation, or materially expanded result can reopen the case through normal briefing.
- 08-25 16:31alert_routeThe staleness trigger carries no new evidence or consequential event; any independent replication, external implementation, or materially expanded result can reopen the case through normal briefing.
- 08-23 15:29repriceVero merits watching because the benchmark, curation pipeline, evaluation harness, and repository-level results form a concrete released package, but nothing new independently validates the reported c
- 08-23 15:29alert_silentThe release was already identified and routed; this look contains only an unchanged reobservation, with no replication, new results, or adoption that needs attention before the next briefing.
- 08-23 15:29alert_routeThe release was already identified and routed; this look contains only an unchanged reobservation, with no replication, new results, or adoption that needs attention before the next briefing.
- 08-23 15:29alert_shadowVero is a concrete new benchmark, dataset, and evaluation harness for testing whether agents can implement and formally verify complete multi-module Lean 4 repositories. Its reported best result—27 of
- 08-23 15:29alert_routeVero is a concrete new benchmark, dataset, and evaluation harness for testing whether agents can implement and formally verify complete multi-module Lean 4 repositories. Its reported best result—27 of
- 08-23 15:28groundVero independently operationalizes Scott’s Evaluation-Driven Development and Spec-Driven Development positions by testing whether agents can turn machine-checkable specifications into repository-scale
- 08-23 15:27promote_anchororigin walk conf 0.98
- 08-23 15:26createThe released project defines a bounded and checkable agent-software capability claim that goes beyond ordinary test-based code generation.
- 08-18 22:29expireVero has not developed beyond the original paper mention: no methods, artifacts, scale results, implementations, or independent replication have surfaced within the observation horizon. The case shoul
- 08-18 22:29alert_silentThe only change is a negligible engagement increment with no new technical evidence, so there is nothing consequential to surface before a routine briefing.
- 08-18 22:29alert_routeThe only change is a negligible engagement increment with no new technical evidence, so there is nothing consequential to surface before a routine briefing.
- 08-16 21:28repriceNo new evidence has arrived: Vero remains an unvalidated research claim without disclosed scale results, usable artifacts, or independent replication. The hot coding-agent neighborhood does not change
- 08-16 21:28alert_silentThis is only an unchanged reobservation and adds no capability result, implementation, or replication that would warrant attention before a routine briefing.
- 08-16 21:28alert_routeThis is only an unchanged reobservation and adds no capability result, implementation, or replication that would warrant attention before a routine briefing.
- 08-16 21:26alert_silentThe evidence establishes that a paper presenting Vero exists, but supplies no methods, results, scale measurements, disclosed harness, or replication outcome. Without a concrete capability result or u
- 08-16 21:26surface_candidateThe evidence establishes that a paper presenting Vero exists, but supplies no methods, results, scale measurements, disclosed harness, or replication outcome. Without a concrete capability result or u
- 08-16 21:26alert_routeThe evidence establishes that a paper presenting Vero exists, but supplies no methods, results, scale measurements, disclosed harness, or replication outcome. Without a concrete capability result or u
- 08-16 21:26groundVero extends Scott’s spec-driven, verification-loop approach from test-gated coding toward repository-scale implementations carrying machine-checked proofs, while its need for disclosed harnesses and
- 08-16 21:23createThe linked paper is a concrete research artifact on agent-generated verified software, but it currently has only one low-engagement observation and no independent validation.