Benchmark author mauricekleine's Nonobench v1.2 reports that among 43 tested LLMs no open-weight model solves the new 20×20 Hard mode (0/10) while GPT-6 Astra posts the first perfect 15×15 run (30/30) and Opus 5.5 scores 8/10; if the open-weight shutout holds as open-weight releases are added, it stands as a measured open-frontier gap on long-constrained verifiable reasoning, and any open model clearing 20×20 refutes it.

state: watchingheat: lowuncertainty: mediumconvergesscott: mediumllm-benchmarks open-weight-modelsmauricekleine

What is this?

Nonobench is a reproducible community benchmark run by independent developer Maurice Kleine that scores LLMs on nonogram puzzles — one attempt per puzzle, no tools, answers machine-checked against the clues — with a public leaderboard at v1.2 covering 43 models. Its headline claims: no open-weight model has solved any 20×20 Hard-mode puzzle (0/10; best open entry DeepSeek V4 Pro sits at 83.3% overall), while OpenAI's GPT-6 Astra posted the first perfect 15×15 run and Anthropic's Opus 5.5 leads Hard at 8/10 — a shutout that held in a 2026-09-30 follow-up in which 11 of 15 models (including Astra, at 5/10) solved zero 20×20s. The supplied web results corroborate the landscape but not the benchmark: the model names and release dates (GPT-6 Astra 2026-09-03, GPT-6.1 Sol 2026-09-29, Opus/Sonnet 5.5 in late September) match release trackers, and independent 2026 reporting (Lumiere, NerdLevelTech, Artificial Analysis figures) measures a persistent open-vs-closed frontier gap that is widest on long-horizon and written-from-scratch benchmarks, with harness effects flagged as a known confound. But zero coverage of Nonobench or Kleine surfaces anywhere, so the specific 0/10 shutout remains a single-source claim — falsified if any open model clears 20×20 raw, while a harnessed-only clear would instead refute the 'open models can't' reading.

Why it matters to Scott

The Hard-mode follow-up sharpens this from an open-gap data point into an instrument for Scott's benchmark-unit argument: the case's own raw-vs-harnessed falsification split operationalizes his model-plus-harness unit claim (with Nonobench's tool-free, one-attempt design as the wrong-unit counterfactual), and Opus taking 8/10 at 'high' while Sol (max) and Astra (xhigh) trail at 7 and 5 is measured evidence that reasoning-effort burn doesn't convert on constraint puzzles. Stays medium — still single-source and quiet, consequential only on a trigger (any open 20×20 clear, and especially a harnessed-only one, which would be dated receipts for the harness-unit position over the 'open models can't' reading).

ip:concept.model-plus-harness-benchmark-unitip:concept.benchmarking-the-wrong-unitip:concept.inference-time-scalingip:concept.capability-symmetryradar:concept.open-modelsradar:concept.open-weight-modelsradar:concept.benchmark-integrityradar:concept.llm-evaluationradar:pcss-zebra-puzzle-reasoning-transferradar:minizinc-mcp-solver-toolsradar:ship-harness-benchradar:claude-sonnet-55-token-economicsradar:qwen38-27b-reasoning-effort

queries asked of Scott's wikis

open-weight vs frontier capability gap position
harness or scaffold as part of the model eval
reproducible verifiable LLM benchmark design
benchmark contamination fresh held-out evals
LLM constraint satisfaction puzzle reasoning limits
reasoning token cost per solved task economics

Measured heat

now 0 pts/hpeak 11 pts/hcomments 0/hpeers p50momentum: steady2 platformsage 386h

points/hour across evidence · reading as of 2026-10-12 02:59:37.977291+11:00 · deterministic, not a model opinion

How the heat travelled

09-25 14:00	⭐ origin echo-reconstructed	Nonobench is Kleine's own LLM nonogram benchmark leaderboard; the v1.2 results it publishes are the source of every claim in the Reddit titl Maurice Kleine on other (echo) · attributed from reddit.post.1wrh7b2	—
09-27 10:51	first on r/LocalLLaMA · published · +44.9h	Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode mauricekleine	—
09-30 14:07	first on r/ClaudeAI · published · +120.1h	Claude Opus 5.5 leads my nonogram benchmark's Hard mode: 8 of 10 random 20x20 puzzles, while 11 of 15 models solve not even one mauricekleine	—
10-04 07:57	first on r/MachineLearning · published · +210.0h	Nonobench: an open benchmark of 49 LLMs on nonogram puzzles, public and open source [P] mauricekleine	—
09-27 10:51	amplified on r/LocalLLaMA 👑	reddit.post.1wrh7b2 mauricekleine	peak 28 · 26 comments · 66% of case engagement
09-30 14:07	amplified on r/ClaudeAI	reddit.post.1wu5w1c mauricekleine	peak 9 · 3 comments · 15% of case engagement
10-04 07:57	amplified on r/MachineLearning	reddit.post.1wxa2bs mauricekleine	peak 8 · 8 comments · 20% of case engagement
09-27 11:20	our radar first saw it · +45.3h	discovery anchor: reddit.post.1wrh7b2	—

pace: p65 vs 1032 stories at the 336h mark (now 386h old) — ahead of gfx906-expert-pool-admission-fix (1.0x), behind ordewell-editable-multi-runner-plans (1.0x)

Evidence (4) — ⭐ canonical anchor

source	object	author	score	comments
🟠 reddit	Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode LocalLLaMA Retrieved article excerpt Open article · Retrieved 2026-09-27T11:24:15.544855+00:00 # Prove your humanity We’re committed to safety and security. But not for bots. Complete the challenge below and let us know you’re a real person. [Reddit, Inc. © "2026". All rights reserved.](https://www.redditinc.com/) [User Agreement](https://www.reddit.com/help/useragreement) [Privacy Policy](https://www.reddit.com/help/privacypolicy) [Content Policy](https://www.reddit.com/help/contentpolicy) [Help](https://support.reddithelp.com/hc/en-us)	mauricekleine	28	26
🟧 echo.other ⭐	Nonobench is Kleine's own LLM nonogram benchmark leaderboard; the v1.2 results it publishes are the source of every claim in the Reddit titl	Maurice Kleine	—	—
🟠 reddit	Claude Opus 5.5 leads my nonogram benchmark's Hard mode: 8 of 10 random 20x20 puzzles, while 11 of 15 models solve not even one ClaudeAI	mauricekleine	9	3
🟠 reddit	Nonobench: an open benchmark of 49 LLMs on nonogram puzzles, public and open source [P] MachineLearning	mauricekleine	8	8

Interpretation history

2026-10-04T08:24:24Z
Open-sourcing converts the single-source watch into an inspectable one: the method (Standard 30 puzzles 5×5–15×15 from the CC BY 4.0 dataset; Hard ten unique-solution 20×20s, five not line-logic-solvable) is now fully replicable, so an outside run is the reachable second evidence line rather than a hypothetical. Roster 43→49 with headline results unchanged extends the open-weight 20×20 shutout under more models. Still zero outside coverage or independent replication and no periphery expansion — the new post itself scored 1 — so it stays a quiet falsifiable watch, not corroborated.
2026-10-04T08:23:29Z
evidence attached: reddit.post.1wxa2bs — Same benchmark now public and open source with roster expanded 43→49 and headline results unchanged — adoption/scrutiny substrate the case's resolution depends on.
2026-09-30T14:46:34Z
grounded: converges/medium — The Hard-mode follow-up sharpens this from an open-gap data point into an instrument for Scott's benchmark-unit argument: the case's own raw-vs-harnessed falsif
2026-09-30T14:37:34Z
Same-author Hard-mode follow-up extends the series: 15 models on the 20×20s, 11 solve zero, Opus 5.5 still #1 (8/10), new GPT-6.1 Sol 7/10, and Astra's perfect 15×15 doesn't transfer (5/10 Hard) — the shutout survived its first extension to the newest frontier models, but there is still no open model close, no second evidence line, and no periphery expansion. Moved seed→watching because the benchmark is now demonstrably an actively iterating series (January post → v1.2 → Hard-mode update within days), which is what makes the falsification watch actionable; evidence itself remains single-source.
2026-09-30T14:25:24Z
evidence attached: reddit.post.1wu5w1c — Independent update from the same benchmark author's Nonobench Hard-mode runs (Opus 5.5 8/10, Sol 7/10, Astra 5/10) plus Sonnet 5.5 long-thinking cost parity — direct evidence for the open-weight gap case and its token-economics angle.
2026-09-28T01:42:23Z
Engagement-only update: two comment churns and a transient 4.5x velocity spike that has already decayed (now 0.67 pts/h, cooling). No new model runs, no independent corroboration, and the 20x20 open-weight shutout stands unchanged — the case's meaning is unchanged; it remains a quiet falsifiable watch with no periphery expansion.
2026-09-27T11:40:52Z
origin walked (opencode/cheap-glm, conf 0.78): anchor reddit.post.1wrh7b2 -> echo.other.a4b7d4b3b2 by Maurice Kleine
2026-09-27T11:33:54Z
grounded: converges/medium — Independently measured (though single-source — grounding found no coverage of Nonobench or mauricekleine, so the 0/10 is a claim to watch, not a fact) confirmat
2026-09-27T11:25:26Z
case created — Second release of a reproducible community benchmark with a specific falsifiable open-vs-frontier result and an active maintainer iterating per model additions; no open case tracks Nonobench.

Decision trace

10-07 06:29review_screenjev screen: no material development (noul=0.07)
10-07 06:24sensor_dirtycomment_update
10-04 19:24repriceOpen-sourcing converts the single-source watch into an inspectable one: the method (Standard 30 puzzles 5×5–15×15 from the CC BY 4.0 dataset; Hard ten unique-solution 20×20s, five not line-logic-solva
10-04 19:23attachSame benchmark now public and open source with roster expanded 43→49 and headline results unchanged — adoption/scrutiny substrate the case's resolution depends on.
10-04 19:23propose_attachSame benchmark now public and open source with roster expanded 43→49 and headline results unchanged — adoption/scrutiny substrate the case's resolution depends on.
10-01 00:46repriceSame-author Hard-mode follow-up extends the series: 15 models on the 20×20s, 11 solve zero, Opus 5.5 still #1 (8/10), new GPT-6.1 Sol 7/10, and Astra's perfect 15×15 doesn't transfer (5/10 H
10-01 00:46groundThe Hard-mode follow-up sharpens this from an open-gap data point into an instrument for Scott's benchmark-unit argument: the case's own raw-vs-harnessed falsification split operationalizes
10-01 00:25attachIndependent update from the same benchmark author's Nonobench Hard-mode runs (Opus 5.5 8/10, Sol 7/10, Astra 5/10) plus Sonnet 5.5 long-thinking cost parity — direct evidence for the open-weight
10-01 00:23propose_attachIndependent update from the same benchmark author's Nonobench Hard-mode runs (Opus 5.5 8/10, Sol 7/10, Astra 5/10) plus Sonnet 5.5 long-thinking cost parity — direct evidence for the open-weight
09-28 11:42repriceEngagement-only update: two comment churns and a transient 4.5x velocity spike that has already decayed (now 0.67 pts/h, cooling). No new model runs, no independent corroboration, and the 20x20 open-w
09-28 07:20sensor_dirtycomment_update
09-28 01:21sensor_dirtyvelocity_spike
09-27 22:20sensor_dirtycomment_update
09-27 21:40promote_anchororigin walk conf 0.78
09-27 21:33groundIndependently measured (though single-source — grounding found no coverage of Nonobench or mauricekleine, so the 0/10 is a claim to watch, not a fact) confirmation of the Model Barbell's open-fro
09-27 21:25createSecond release of a reproducible community benchmark with a specific falsifiable open-vs-frontier result and an active maintainer iterating per model additions; no open case tracks Nonobench.