Independent use will determine whether Computer Anthology’s continuously evolving terminal-task family provides durable agent measurements that resist saturation better than static benchmarks.

state: expiredheat: lowuncertainty: highknownscott: mediumagent-benchmarks terminal-agents coding-agents computer-useVetto AIComputer Anthology

What is this?

Computer Anthology is presented in the case as a Vetto AI benchmark family that continuously evolves terminal tasks for evaluating AI agents. The supplied web snippets do not independently document Computer Anthology itself or establish results from outside adopters; they only support the broader motivation that static, manually curated benchmarks can saturate or diverge from authentic workflows, while terminal and long-horizon evaluations test sustained execution, recovery, and maintainability. Whether Computer Anthology delivers more durable measurements therefore remains an unverified hypothesis in the supplied material.

Why it matters to Scott

Scott already argues for repeatable, contamination-aware evaluation of agents on authentic tool-using paths in “Evaluation-Driven Development,” “Benchmarking the Wrong Unit,” and “Reflexive Agent Design.” Computer Anthology could become a useful external evaluation surface for his Ask terminal agent, but absent independent adoption or results, it currently adds an unverified implementation rather than new evidence about whether continuously evolving benchmarks resist saturation.

ip:concept.benchmarking-the-wrong-unitip:concept.evaluation-driven-developmentip:framework.reflexive-agent-designdev:project.askradar:concept.agent-benchmarksradar:concept.agent-evaluationradar:concept.benchmark-integrityradar:concept.agent-harnesses

queries asked of Scott's wikis

evolving benchmarks and benchmark saturation
coding-agent evaluation on authentic workflows
terminal-agent harnesses and task verification
long-horizon agent reliability and recovery
benchmark contamination and durable measurements
agent evaluation beyond pass/fail tests

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	Computer Anthology: A continuously evolving benchmark family for AI agents	rigelbm	29	11
🟧 echo.blog ⭐	Introduces Computer Anthology as a continuously evolving benchmark family for AI agents performing terminal tasks.	Vetto AI	—	—

Interpretation history

2026-08-15T22:24:24Z
Repeated review windows have produced only minor launch-thread engagement, with no independent adoption, comparative results, implementation, or evidence that evolving tasks resist saturation. The episode has faded as an unvalidated benchmark launch and should be reopened only if external usage or durability data emerges.
2026-08-13T21:32:35Z
The staleness trigger adds no evidence and arrived sooner than the previously chosen weekly cadence. The benchmark remains a plausible but untested launch; revisit only if independent use, comparative results, or task-evolution data appear.
2026-08-11T20:41:30Z
The latest change is only minor launch-thread amplification, with no independent use, implementation, comparison, or durability results. The design remains plausible but unvalidated; move monitoring to a weekly cadence rather than repeatedly revisiting stale engagement.
2026-08-09T19:42:05Z
Another review window passed without independent adoption, implementation, or durability results. The launch remains a plausible but untested benchmark design, so defer further attention until external users publish results or comparisons.
2026-08-07T19:33:31Z
No independent use, implementation, or durability evidence has appeared; the small engagement increase is repetitive amplification rather than validation. Keep the case open on a longer horizon for external benchmark results or adoption.
2026-08-04T17:25:32Z
The added observation provides no independent adoption, implementation, or durability results; it remains launch testimony plus unchanged, modest discussion. The benchmark’s saturation-resistance hypothesis is still untested, so attention can cool pending external use.
2026-08-04T15:30:39Z
grounded: known/medium — Scott already argues for repeatable, contamination-aware evaluation of agents on authentic tool-using paths in “Evaluation-Driven Development,” “Benchmarking th
2026-08-04T15:28:26Z
case created — This is a distinct benchmark launch with a concrete evolving-task design whose usefulness and resistance to saturation can be independently evaluated.

Decision trace

08-16 08:24expireRepeated review windows have produced only minor launch-thread engagement, with no independent adoption, comparative results, implementation, or evidence that evolving tasks resist saturation. The epi
08-16 08:24alert_silentThe new delta is engagement-only and adds no consequential evidence; elapsed silence does not merit Scott’s attention.
08-16 08:24alert_routeThe new delta is engagement-only and adds no consequential evidence; elapsed silence does not merit Scott’s attention.
08-14 07:32repriceThe staleness trigger adds no evidence and arrived sooner than the previously chosen weekly cadence. The benchmark remains a plausible but untested launch; revisit only if independent use, comparative
08-14 07:32alert_silentNothing consequential changed, and elapsed silence is not alert-worthy; independent adoption or durability evidence can wait for routine review.
08-14 07:32alert_routeNothing consequential changed, and elapsed silence is not alert-worthy; independent adoption or durability evidence can wait for routine review.
08-12 06:41repriceThe latest change is only minor launch-thread amplification, with no independent use, implementation, comparison, or durability results. The design remains plausible but unvalidated; move monitoring t
08-12 06:41alert_silentNo consequential evidence has arrived, and a small engagement increase does not change the benchmark’s status; wait for an external adopter, published results, or a direct comparison with static bench
08-12 06:41alert_routeNo consequential evidence has arrived, and a small engagement increase does not change the benchmark’s status; wait for an external adopter, published results, or a direct comparison with static bench
08-10 05:42repriceAnother review window passed without independent adoption, implementation, or durability results. The launch remains a plausible but untested benchmark design, so defer further attention until externa
08-10 05:42alert_silentNo new consequential evidence arrived; staleness alone does not warrant interrupting Scott, and independent usage evidence is more likely to emerge on a weekly than hourly cadence.
08-10 05:42alert_routeNo new consequential evidence arrived; staleness alone does not warrant interrupting Scott, and independent usage evidence is more likely to emerge on a weekly than hourly cadence.
08-08 05:33repriceNo independent use, implementation, or durability evidence has appeared; the small engagement increase is repetitive amplification rather than validation. Keep the case open on a longer horizon for ex
08-08 05:33alert_silentThe new delta is only a minor engagement change and does not alter the benchmark’s untested status, so it can wait for substantive independent use.
08-08 05:33alert_routeThe new delta is only a minor engagement change and does not alter the benchmark’s untested status, so it can wait for substantive independent use.
08-05 03:25repriceThe added observation provides no independent adoption, implementation, or durability results; it remains launch testimony plus unchanged, modest discussion. The benchmark’s saturation-resistance hypo
08-05 03:20mark_dirtyengagement_update
08-05 01:30groundScott already argues for repeatable, contamination-aware evaluation of agents on authentic tool-using paths in “Evaluation-Driven Development,” “Benchmarking the Wrong Unit,” and “Reflexive Agent Desi
08-05 01:28createThis is a distinct benchmark launch with a concrete evolving-task design whose usefulness and resistance to saturation can be independently evaluated.