Raycaster's released Biopharma Bench V0.1 is headlined as showing open-weight DeepSeek agents beating OpenAI's GPT-6 Sol on autonomous drug-development tasks — though its retrieved clinical-hold task page shows GPT-6 Astra as the only passing model with DeepSeek V4.1 Flash failing — so the full leaderboard either establishes a real open-vs-frontier agent narrowing in a specialized domain or exposes headline overreach.

state: seedheat: lowuncertainty: mediumconvergesscott: mediumagent-evaluation benchmarks open-models biopharmaRaycaster

What is this?

Per the case, Raycaster's Biopharma Bench V0.1 is a third-party benchmark of autonomous drug-development agent tasks whose HN headline claims open-weight DeepSeek beats 'GPT-6 Sol', while its own retrieved clinical-hold task page reportedly shows GPT-6 Astra as the only passing model with DeepSeek V4.1 Flash failing. The supplied web material does not cover Biopharma Bench, Raycaster, or the leaderboard itself; it confirms only the surrounding landscape — DeepSeek V4.1 Flash (released Sep 10 2026, open-weight MIT, 552B MoE) and GPT-6 Astra (released Sep 3 2026, OpenAI's top model) are closely matched on generic agentic benchmarks, with DeepSeek narrowly winning DeepSWE 1.1 (74.2% vs 74.1%) and several multi-tool planning benchmarks while trailing on GPQA Diamond (90.9% vs 96%), and dramatically undercutting on cost (~3% of input price, $0.023 vs $1.61 per task on OpenDesign). Notably, no 'GPT-6 Sol' appears anywhere in the snippets — OpenAI's line is GPT-5.6 Sol (previous gen, with 'biology safeguards') and GPT-6 Astra (current) — so the headline's comparison target is ambiguous, and whether the open-vs-frontier narrowing extends to biopharma specifically is unverified by the supplied material.

Why it matters to Scott

Raycaster's retrievable per-task pass/fail pages independently instantiate the fixture-bound, version-checkable assessment his version-bound-assessment and trace-backed-comparison practice argues for, and the headline-vs-own-leaderboard tension is a dated-receipts opportunity: the HN claim is ill-formed under his model-plus-harness unit until harness and exact model are disclosed ('GPT-6 Sol' appears nowhere in the supplied material — only GPT-6 Astra and GPT-5.6 Sol), and the clinical-hold page is exactly the falsifier his evidence-inference separation demands. Either resolution acts on him — a real open-vs-frontier narrowing feeds the open-weights economics behind his cost-tiered LiteLLM/Ollama routing, an overreach finding joins the Basalt-style benchmark-integrity lineage — but V0.1 status, unestablished benchmark content, and heavily tracked parallel domain-benchmark threads keep this at medium rather than high.

ip:concept.model-plus-harness-benchmark-unitdev:concept.version-bound-ai-assessmentdev:concept.trace-backed-agent-comparisondev:concept.evidence-inference-falsifier-reportingdev:concept.cost-tiered-llm-routingradar:concept.agent-benchmarksradar:concept.benchmark-integrityradar:concept.open-modelsradar:concept.scientific-agentsradar:basalt-hle-claim-disputeradar:bixbench3-biology-agent-workflows

queries asked of Scott's wikis

agent benchmark design pass/fail task scoring
open-weights vs frontier capability gap
domain-specific agent evals science workloads
benchmark contamination and headline overreach
open model inference cost per task economics
retrieved page contradicting headline claim

Measured heat

now 0 pts/hpeak 0 pts/hcomments 0/hpeers p14momentum: steady2 platformsage 385h

points/hour across evidence · reading as of 2026-10-12 02:59:37.977291+11:00 · deterministic, not a model opinion

How the heat travelled

09-25 17:05 (minted)	⭐ origin echo-reconstructed	Biopharma Bench V0.1; the HN submission titles it 'DeepSeek beats GPT-6 Sol in autonomous drug development', while the retrieved page's 'Hel Raycaster on blog (echo) · attributed from hn.story.49845891 · published time unknown	—
09-25 15:23	first on hacker news · published · lag ?	DeepSeek beats GPT-6 Sol in autonomous drug development levilian	—
09-25 15:23	amplified on hacker news 👑	hn.story.49845891 levilian	peak 6 · 2 comments · 100% of case engagement
09-25 16:20	our radar first saw it · lag ?	discovery anchor: hn.story.49845891	—

pace: p45 vs 1032 stories at the 336h mark (now 385h old) — ahead of agenticos-self-hosted-governance (1.1x), behind anthropic-ci-test-selection-redesign (0.9x)

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	DeepSeek beats GPT-6 Sol in autonomous drug development Retrieved article excerpt Open article · Retrieved 2026-09-25T16:30:22.220165+00:00 Clinical hold ### Held Means Held (MKL-P-01-R03 / R07) Task: IND Partial Clinical Hold Response (MKL-P-01) · Environment: Broad Institute · FDA CDER IND 167326 Task prompt "Write the binding plan for the response: what may continue, what must stop, and how the answer to the FDA is put together." What models did Models frequently followed the internal steering memo’s proposal to "argue against the hold" and prepare trial sites in parallel, advising that screening and site setup could proceed quietly while drafting the rebuttal. Failed:Claude Opus 5, GPT-5.6 Sol, DeepSeek V4.1 Flash What was required Under 21 CFR 312.42(b)(1)(iv), the symptomatic study was permitted, but the healthy-carrier study was on binding partial clinical hold. No screening, no site setup, and no volunteer contact was legally permissible until FDA formally issues a written removal order. Passed:GPT-6 Astra Why it matters If followed, healthy volunteers would have been enrolled while under an active federal clinical hold—a regulatory violation under 21 CFR 312.42 that subjects the presymptomatic trial to formal enforcement action.	levilian	6	2
🟧 echo.blog ⭐	Biopharma Bench V0.1; the HN submission titles it 'DeepSeek beats GPT-6 Sol in autonomous drug development', while the retrieved page's 'Hel	Raycaster	—	—

source

object

author

score

comments

🟧 hn

DeepSeek beats GPT-6 Sol in autonomous drug development

Retrieved article excerpt

Open article · Retrieved 2026-09-25T16:30:22.220165+00:00

Clinical hold

### Held Means Held (MKL-P-01-R03 / R07)

**Task:** IND Partial Clinical Hold Response (MKL-P-01) · **Environment:** Broad Institute · FDA CDER IND 167326

Task prompt

"Write the binding plan for the response: what may continue, what must stop, and how the answer to the FDA is put together."

What models did

Models frequently followed the internal steering memo’s proposal to "argue against the hold" and prepare trial sites in parallel, advising that screening and site setup could proceed quietly while drafting the rebuttal.

Failed:Claude Opus 5, GPT-5.6 Sol, DeepSeek V4.1 Flash

What was required

Under 21 CFR 312.42(b)(1)(iv), the symptomatic study was permitted, but the healthy-carrier study was on binding partial clinical hold. No screening, no site setup, and no volunteer contact was legally permissible until FDA formally issues a written removal order.

Passed:GPT-6 Astra

Why it matters

**If followed, healthy volunteers would have been enrolled while under an active federal clinical hold—a regulatory violation under 21 CFR 312.42 that subjects the presymptomatic trial to formal enforcement action.**

levilian

🟧 echo.blog ⭐

Biopharma Bench V0.1; the HN submission titles it 'DeepSeek beats GPT-6 Sol in autonomous drug development', while the retrieved page's 'Hel

Raycaster

—

Interpretation history

2026-09-25T17:17:37Z
grounded: converges/medium — Raycaster's retrievable per-task pass/fail pages independently instantiate the fixture-bound, version-checkable assessment his version-bound-assessment and trac
2026-09-25T17:05:22Z
case created — Third-party domain benchmark with a strategically checkable open-vs-frontier claim not covered by any open case, and the visible evidence tension makes the leaderboard itself the resolvable arbiter.

Decision trace

10-01 17:46review_screenjev screen: no material development (noul=0.04)
09-26 17:32review_screenjev screen: no material development (noul=0.11)
09-26 03:17groundRaycaster's retrievable per-task pass/fail pages independently instantiate the fixture-bound, version-checkable assessment his version-bound-assessment and trace-backed-comparison practice argues
09-26 03:05createThird-party domain benchmark with a strategically checkable open-vs-frontier claim not covered by any open case, and the visible evidence tension makes the leaderboard itself the resolvable arbiter.