Terminal-Bench-Science’s maintainers claim their benchmark reproducibly measures AI agents on realistic scientific research workflows, enabling capability comparisons beyond synthetic tasks.

state: expiredheat: lowuncertainty: highconvergesscott: mediumscientific-agents agent-benchmarksTerminal-Bench-Science

What is this?

Terminal-Bench Science is an announced extension of Terminal-Bench, maintained through the Harbor Framework repository and Terminal-Bench team, for evaluating AI agents on complex computational workflows drawn from natural-science research. Its organizers are soliciting scientists to contribute tasks, aiming to create comparable terminal-based evaluations and spur “Claude Code / Codex for Science”-style systems. The supplied snippets establish the benchmark’s goals and contribution process, but provide no completed results or independent evidence that it yet measures scientific-agent capabilities reproducibly.

Why it matters to Scott

Terminal-Bench Science independently converges with Scott’s evaluation-driven and model-plus-harness position, and directly touches his trace-backed agent-comparison work by proposing comparable evaluations on representative workflows. It creates a potential dated-receipts and implementation-comparison opportunity, but remains only an announced benchmark without results or independent reproducibility evidence.

ip:concept.evaluation-driven-developmentip:concept.model-plus-harness-benchmark-unitdev:concept.trace-backed-agent-comparisonradar:bixbench3-biology-agent-workflowsradar:concept.agent-evaluationradar:concept.scientific-ai

queries asked of Scott's wikis

real-world agent evaluation harnesses and reproducibility
benchmark-driven agent development and Goodhart effects
long-horizon terminal agents and error recovery
scientific workflow automation with coding agents
agent benchmark integrity contamination and continuous validation
portable adapters for cross-agent capability comparisons

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	Terminal-Bench-Science: Evaluating AI agents on scientific research workflows	matt_d	115	36
🟧 echo.blog ⭐	Announces a benchmark for evaluating AI agents on scientific research workflows.	Terminal-Bench-Science	—	—

Interpretation history

2026-08-31T15:37:09Z
After repeated checks and 48 hours without artifacts, results, methodology, or independent use, the announcement has faded without substantiating its reproducibility claim. Reopen only if the maintainers release an executable benchmark or external teams report results.
2026-08-29T15:33:15Z
The latest comments repeat anecdotal model comparisons and correctness concerns without adding tasks, methodology, results, or independent reproduction. The benchmark remains a relevant proposal, not yet evidence of reproducible scientific-agent measurement.
2026-08-28T14:41:50Z
The refreshed comments add concerns about correctness and anecdotal model comparisons, but no methodology, released tasks, results, or independent reproduction. Repeated discussion does not change the case from an announced benchmark awaiting substantive artifacts or external use.
2026-08-28T09:31:23Z
The refreshed discussion remains anecdotal model preference and broad enthusiasm, adding no methodology, artifacts, benchmark results, or independent reproduction. The case still represents a relevant benchmark proposal rather than evidence that realistic scientific-agent workflows are being measured reproducibly.
2026-08-28T01:32:46Z
Refreshed comments merely endorse realistic workflow evals or make unsupported model comparisons; they add no artifacts, methodology, results, or independent reproducibility evidence. The case remains an announced benchmark awaiting implementation or external use.
2026-08-28T00:28:02Z
No substantive evidence has appeared beyond the original benchmark announcement; modest engagement does not validate its task design, results, or reproducibility. The case remains a relevant proposal awaiting artifacts or independent use.
2026-08-28T00:26:32Z
grounded: converges/medium — Terminal-Bench Science independently converges with Scott’s evaluation-driven and model-plus-harness position, and directly touches his trace-backed agent-compa
2026-08-28T00:24:34Z
case created — This is a concrete first-party benchmark release addressing a consequential and distinct scientific-agent evaluation gap.

Decision trace

09-01 01:37expireAfter repeated checks and 48 hours without artifacts, results, methodology, or independent use, the announcement has faded without substantiating its reproducibility claim. Reopen only if the maintain
09-01 01:37alert_silentNo new consequential fact emerged; staleness alone does not warrant an alert, and any future artifact or independent reproduction can start a fresh episode.
09-01 01:37alert_routeNo new consequential fact emerged; staleness alone does not warrant an alert, and any future artifact or independent reproduction can start a fresh episode.
08-30 01:33repriceThe latest comments repeat anecdotal model comparisons and correctness concerns without adding tasks, methodology, results, or independent reproduction. The benchmark remains a relevant proposal, not
08-30 01:33alert_silentOnly engagement and repetitive commentary changed; no consequential fact emerged that warrants attention before the next briefing.
08-30 01:33alert_routeOnly engagement and repetitive commentary changed; no consequential fact emerged that warrants attention before the next briefing.
08-29 00:41repriceThe refreshed comments add concerns about correctness and anecdotal model comparisons, but no methodology, released tasks, results, or independent reproduction. Repeated discussion does not change the
08-29 00:41alert_silentNo consequential new fact emerged; the discussion remains unsupported commentary and can wait while the case awaits benchmark artifacts, published results, or independent validation.
08-29 00:41alert_routeNo consequential new fact emerged; the discussion remains unsupported commentary and can wait while the case awaits benchmark artifacts, published results, or independent validation.
08-29 00:21sensor_dirtycomment_update
08-28 22:22sensor_dirtyengagement_update
08-28 20:21sensor_dirtyengagement_update
08-28 19:31repriceThe refreshed discussion remains anecdotal model preference and broad enthusiasm, adding no methodology, artifacts, benchmark results, or independent reproduction. The case still represents a relevant
08-28 19:31alert_silentNo consequential new fact has emerged; repetitive, unsupported comments can wait for the next briefing while the case awaits released tasks, results, or independent use.
08-28 19:31alert_routeNo consequential new fact has emerged; repetitive, unsupported comments can wait for the next briefing while the case awaits released tasks, results, or independent use.
08-28 19:21sensor_dirtycomment_update
08-28 18:21sensor_dirtycomment_update
08-28 17:21sensor_dirtycomment_update
08-28 15:21sensor_dirtyengagement_update
08-28 14:21sensor_dirtycomment_update
08-28 13:21sensor_dirtycomment_update
08-28 12:21sensor_dirtyengagement_update
08-28 11:32repriceRefreshed comments merely endorse realistic workflow evals or make unsupported model comparisons; they add no artifacts, methodology, results, or independent reproducibility evidence. The case remains
08-28 11:32alert_silentThe new discussion is repetitive and unsubstantiated, so it does not change the benchmark’s credibility or create a time-sensitive implication for Scott.
08-28 11:32alert_routeThe new discussion is repetitive and unsubstantiated, so it does not change the benchmark’s credibility or create a time-sensitive implication for Scott.
08-28 11:21sensor_dirtycomment_update
08-28 10:28repriceNo substantive evidence has appeared beyond the original benchmark announcement; modest engagement does not validate its task design, results, or reproducibility. The case remains a relevant proposal
08-28 10:28alert_silentThe only change is minor engagement, with no methodology, benchmark results, implementation, or independent validation; it can wait for the next briefing.
08-28 10:28alert_routeThe only change is minor engagement, with no methodology, benchmark results, implementation, or independent validation; it can wait for the next briefing.
08-28 10:26alert_silentThe benchmark announcement is relevant to Scott’s evaluation work, but the visible evidence provides no task design, harness, results, artifacts, or reproducibility details that would change what he s
08-28 10:26surface_candidateThe benchmark announcement is relevant to Scott’s evaluation work, but the visible evidence provides no task design, harness, results, artifacts, or reproducibility details that would change what he s
08-28 10:26alert_routeThe benchmark announcement is relevant to Scott’s evaluation work, but the visible evidence provides no task design, harness, results, artifacts, or reproducibility details that would change what he s
08-28 10:26groundTerminal-Bench Science independently converges with Scott’s evaluation-driven and model-plus-harness position, and directly touches his trace-backed agent-comparison work by proposing comparable evalu
08-28 10:24createThis is a concrete first-party benchmark release addressing a consequential and distinct scientific-agent evaluation gap.