HunterBench claims its live-infrastructure benchmark can reproducibly compare LLM and agent pentesting capabilities beyond isolated vulnerability-exploitation tasks, giving security engineers a more realistic basis for model selection.

state: expiredheat: lowuncertainty: highknownscott: lowagentic-security pentesting-benchmarks cyber-agentsHunterBench

What is this?

HunterBench is presented as a benchmark for comparing LLMs and AI agents on penetration testing against live infrastructure, aiming to move beyond isolated vulnerability-exploitation tasks toward realistic, end-to-end attack workflows. The supplied results support the need for controlled, reproducible evaluations on realistic targets: existing agents perform much worse autonomously than with human guidance, especially on complex chained attacks. However, the snippets do not identify who operates HunterBench or provide direct evidence for its methodology, results, or reproducibility claims, so those details remain unverified here.

Why it matters to Scott

The core position is already explicit in Scott’s “Capability Audit,” “Benchmarking the Wrong Unit,” and “Model-Plus-Harness Benchmark Unit,” while the radar already tracks realistic agent evaluation under “agent-benchmarks” and offensive-security agents under “cyber-agents.” HunterBench is a domain-specific example rather than a demonstrated extension or challenge, and the supplied evidence does not verify its methodology, reproducibility, or results sufficiently to affect Scott’s model-selection or benchmark work.

ip:concept.capability-auditip:concept.benchmarking-the-wrong-unitip:concept.model-plus-harness-benchmark-unitdev:concept.trace-backed-agent-comparisonradar:concept.agent-benchmarksradar:concept.cyber-agentsradar:concept.agent-evaluation

queries asked of Scott's wikis

live-environment benchmarks for autonomous agents
reproducible evaluation of stochastic agent systems
cyber-agent harnesses and human-in-the-loop autonomy
benchmarking end-to-end workflows versus isolated tasks
model selection through task-specific capability evaluations
agent observability and replay in stateful environments

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟠 reddit	Which LLM is actually best at pentesting? benchmark to find out LocalLLaMA	TomatoWasabi	39	26
🟧 echo.blog ⭐	Introduces a benchmark intended to compare current LLMs and AI agents on pentesting against live infrastructure.	HunterBench	—	—

Interpretation history

2026-09-02T14:38:33Z
After repeated checks and 48 hours without methodology, code, detailed results, or independent use, the initial launch discussion has faded without validating HunterBench’s claims. The case can close unless substantive technical evidence later creates a new episode.
2026-08-31T13:38:41Z
The refreshed discussion again raises harness and model-selection questions but provides no methodology, code, detailed results, or independent replication. It remains an unverified implementation of a benchmark pattern already familiar to Scott.
2026-08-31T06:30:03Z
The refreshed comments add user interest and benchmark-design questions but no technical artifacts, comparative methodology, or independent replication. The case remains an unverified domain-specific example of a benchmark pattern already familiar to Scott.
2026-08-31T04:29:02Z
The refreshed comments are repetitive discussion rather than validation of HunterBench’s methodology, reproducibility, or comparative findings. Without technical artifacts or independent use, it remains a low-relevance domain example of a familiar benchmark pattern.
2026-08-31T02:28:46Z
The refreshed comments remain informal discussion and add no methodology, artifacts, comparative data, or independent replication. HunterBench is still an unverified example of a benchmark pattern already familiar to Scott.
2026-08-31T01:30:02Z
The refreshed discussion remains speculative and adds no methodology, artifacts, comparative data, or independent use. HunterBench is still an unverified implementation of a benchmark pattern already familiar to Scott.
2026-08-31T00:31:53Z
Refreshed comments raise benchmark-design questions but add no methodology, artifacts, replication, or independent results. HunterBench remains an unverified domain example of a benchmark pattern already familiar to Scott.
2026-08-30T23:32:30Z
No substantive evidence arrived: the small Reddit score change does not validate HunterBench’s methodology, reproducibility, or comparative results. The case remains an unverified implementation of an already familiar benchmark pattern and can cool pending technical artifacts or independent use.
2026-08-30T23:27:55Z
grounded: known/low — The core position is already explicit in Scott’s “Capability Audit,” “Benchmarking the Wrong Unit,” and “Model-Plus-Harness Benchmark Unit,” while the radar alr
2026-08-30T23:24:31Z
case created — This is a concrete first-party security-evaluation artifact addressing a consequential and actively developing agent capability.

Decision trace

09-03 00:38expireAfter repeated checks and 48 hours without methodology, code, detailed results, or independent use, the initial launch discussion has faded without validating HunterBench’s claims. The case can close
09-03 00:38alert_silentThere is no consequential new delta—only elapsed staleness—and the benchmark remains an unverified example of a pattern already familiar to Scott.
09-03 00:38alert_routeThere is no consequential new delta—only elapsed staleness—and the benchmark remains an unverified example of a pattern already familiar to Scott.
09-01 02:21sensor_dirtyengagement_update
08-31 23:38repriceThe refreshed discussion again raises harness and model-selection questions but provides no methodology, code, detailed results, or independent replication. It remains an unverified implementation of
08-31 23:38alert_silentThe new comments do not materially change HunterBench’s credibility or practical usefulness; this can wait for a technical artifact, independent implementation, or replication.
08-31 23:38alert_routeThe new comments do not materially change HunterBench’s credibility or practical usefulness; this can wait for a technical artifact, independent implementation, or replication.
08-31 23:21sensor_dirtycomment_update
08-31 19:21sensor_dirtyengagement_update
08-31 16:30repriceThe refreshed comments add user interest and benchmark-design questions but no technical artifacts, comparative methodology, or independent replication. The case remains an unverified domain-specific
08-31 16:30alert_silentNo consequential new evidence changes HunterBench’s credibility or usefulness; it can wait for published methodology, code, detailed results, or independent validation.
08-31 16:30alert_routeNo consequential new evidence changes HunterBench’s credibility or usefulness; it can wait for published methodology, code, detailed results, or independent validation.
08-31 16:21sensor_dirtycomment_update
08-31 14:29repriceThe refreshed comments are repetitive discussion rather than validation of HunterBench’s methodology, reproducibility, or comparative findings. Without technical artifacts or independent use, it remai
08-31 14:29alert_silentNo consequential new evidence arrived; wait for published methodology, code, detailed results, or independent replication.
08-31 14:29alert_routeNo consequential new evidence arrived; wait for published methodology, code, detailed results, or independent replication.
08-31 14:21sensor_dirtycomment_update
08-31 12:28repriceThe refreshed comments remain informal discussion and add no methodology, artifacts, comparative data, or independent replication. HunterBench is still an unverified example of a benchmark pattern alr
08-31 12:28alert_silentNothing in the new discussion changes the benchmark’s evidentiary footing; it can wait for published methodology, code, detailed results, or independent validation.
08-31 12:28alert_routeNothing in the new discussion changes the benchmark’s evidentiary footing; it can wait for published methodology, code, detailed results, or independent validation.
08-31 12:21sensor_dirtycomment_update
08-31 11:30repriceThe refreshed discussion remains speculative and adds no methodology, artifacts, comparative data, or independent use. HunterBench is still an unverified implementation of a benchmark pattern already
08-31 11:30alert_silentThe new delta is repetitive discussion rather than evidence that the benchmark is reproducible or useful; it can wait for published methodology, code, detailed results, or independent validation.
08-31 11:30alert_routeThe new delta is repetitive discussion rather than evidence that the benchmark is reproducible or useful; it can wait for published methodology, code, detailed results, or independent validation.
08-31 11:21sensor_dirtycomment_update
08-31 10:31repriceRefreshed comments raise benchmark-design questions but add no methodology, artifacts, replication, or independent results. HunterBench remains an unverified domain example of a benchmark pattern alre
08-31 10:31alert_silentThe new delta is discussion-only and does not change the benchmark’s evidentiary footing; it can wait for published methodology, code, detailed results, or independent use.
08-31 10:31alert_routeThe new delta is discussion-only and does not change the benchmark’s evidentiary footing; it can wait for published methodology, code, detailed results, or independent use.
08-31 10:21sensor_dirtycomment_update
08-31 09:32repriceNo substantive evidence arrived: the small Reddit score change does not validate HunterBench’s methodology, reproducibility, or comparative results. The case remains an unverified implementation of an
08-31 09:32alert_silentThe new delta is engagement-only and does not change the benchmark’s evidentiary footing or implications for Scott; it can wait for methodology, published results, code, or independent replication.
08-31 09:32alert_routeThe new delta is engagement-only and does not change the benchmark’s evidentiary footing or implications for Scott; it can wait for methodology, published results, code, or independent replication.
08-31 09:31alert_silentA self-authored Reddit post establishes that HunterBench has been introduced as a live-infrastructure pentesting benchmark, but supplies no methodology, reproducibility evidence, result set, or indepe
08-31 09:31alert_routeA self-authored Reddit post establishes that HunterBench has been introduced as a live-infrastructure pentesting benchmark, but supplies no methodology, reproducibility evidence, result set, or indepe
08-31 09:27groundThe core position is already explicit in Scott’s “Capability Audit,” “Benchmarking the Wrong Unit,” and “Model-Plus-Harness Benchmark Unit,” while the radar already tracks realistic agent evaluation u
08-31 09:24createThis is a concrete first-party security-evaluation artifact addressing a consequential and actively developing agent capability.