Basalt Labs' claimed 99.44% HLE result will be shown to rely on a misleading model identity, evaluation setup, or served-model substitution.
state: expiredheat: lowuncertainty: highconvergesscott: mediumbenchmark-integrity model-identity ai-scams hleBasalt LabsDeepSeekQwen
What is this?
Basalt Labs is alleged to have claimed a 99.44% score on Humanity’s Last Exam (HLE) using tools; HLE is a 3,000-question, multimodal academic benchmark whose reported frontier scores remain below 60% in the supplied results. The evidence title further alleges that Basalt released a Qwen2.5-7B-Instruct-based model while serving DeepSeek on its website, raising possible model-substitution and evaluation-integrity concerns. However, the supplied web snippets neither independently document Basalt Labs, its claimed result, nor the alleged identity mismatch, so the scam hypothesis remains unverified here.
Why it matters to Scott
If verified, the alleged served-model substitution would reinforce Scott’s requirement to declare model versions, preserve evaluation/production parity, and audit real execution rather than trust vendor demos. It also bears directly on his provider-side execution benchmark and multi-provider gateways by suggesting that served-model identity should become an explicit verification control, but the allegation is currently uncorroborated.
ip:framework.12-factor-agents-frameworkip:concept.capability-auditdev:project.remote-execdev:technology.litellmdev:technology.openrouter
queries asked of Scott's wikis
- benchmark integrity and reproducible model evaluations
- served-model identity and provenance verification
- benchmark contamination tool use and evaluation leakage
- AI product claims audit and scam detection
- model substitution in hosted inference APIs
- HLE benchmark limitations and evaluation methodology
Measured heat
no measured readings yet — the hourly heat pass fills this in
How the heat travelled
no chain yet — the hourly chain pass fills this in
Evidence (1) — ⭐ canonical anchor
Interpretation history
2026-07-22T02:22:52Z
The allegation has produced no independent verification after the observation window; the slight engagement change is stale amplification rather than new substance. Expire the episode without treating the claim as proved or disproved.
2026-07-20T04:21:34Z
grounded: converges/medium — If verified, the alleged served-model substitution would reinforce Scott’s requirement to declare model versions, preserve evaluation/production parity, and aud
2026-07-20T01:36:51Z
No independent evidence has emerged to substantiate the model-substitution allegation; the discussion remains repetitive amplification of the original suspicion. The case is still testable but has lost near-term momentum.
2026-07-19T11:24:20Z
case created — The post makes specific, testable allegations about an extraordinary benchmark score and a served-model mismatch.
Decision trace
- 07-22 12:22expireThe allegation has produced no independent verification after the observation window; the slight engagement change is stale amplification rather than new substance. Expire the episode without treating
- 07-20 14:21groundIf verified, the alleged served-model substitution would reinforce Scott’s requirement to declare model versions, preserve evaluation/production parity, and audit real execution rather than trust vend
- 07-20 11:36repriceNo independent evidence has emerged to substantiate the model-substitution allegation; the discussion remains repetitive amplification of the original suspicion. The case is still testable but has los
- 07-20 11:23mark_dirtyengagement_update
- 07-19 21:24createThe post makes specific, testable allegations about an extraordinary benchmark score and a served-model mismatch.