Independent benchmarks and merge review will determine whether llama.cpp’s proposed AVX2 IQ kernels materially accelerate large-batch CPU prompt processing without meaningful perplexity loss.

state: expiredheat: lowuncertainty: highknownscott: mediumlocal-inference quantization llama-cppggml-org bartowski1182

What is this?

Bartowski1182 has opened a draft pull request to ggml-org’s llama.cpp proposing AVX2 batched-GEMM kernels for IQ-quantized models, aimed at speeding large-batch CPU prompt processing. The supplied background supports the underlying premise that prompt ingestion is batchable, can be a CPU bottleneck for long contexts, and is sensitive to quantization-specific kernels. However, the snippets do not provide direct benchmark or perplexity results for this PR, so its material speedup and quality impact remain claims requiring independent testing and merge review.

Why it matters to Scott

Scott already holds the relevant position in “Hardware-aware local inference”: numerical precision, hardware capabilities, and compilation kernels should be explicit runtime concerns, with changes validated through evaluation-driven development. This draft llama.cpp optimization could affect CPU prefill performance in his local-serving stack, but without benchmarks or quality measurements it is currently an implementation candidate to test rather than a finding that changes his view.

dev:concept.hardware-aware-local-inferenceip:concept.evaluation-driven-developmentdev:technology.ollamaradar:concept.llama-cppradar:concept.cpu-inferenceradar:concept.quantizationradar:llama-cpp-x86-vnni-q2-speedup

queries asked of Scott's wikis

CPU prompt-ingestion bottlenecks in local inference
quantization quality versus inference-speed tradeoffs
AVX2 kernel optimization for local LLMs
benchmarking methodology for llama.cpp changes
large-batch prompt processing and long-context latency
local inference hardware compatibility strategy

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟠 reddit	[Draft - Open PR] AVX2: Speed up large batch size prompt processing of IQ models by bartowski1182 · Pull Request #27402 · ggml-org/llama.cpp LocalLLaMA	pmttyji	58	16
🟧 echo.github ⭐	The earliest public artifact is the first commit in the PR branch, titled “Batched gemm for grid IQ quants.” The later PR explains the imple	Colin Kealty (bartowski1182)	—	—

Interpretation history

2026-08-26T22:38:43Z
Repeated checks have produced no reproducible benchmark, quality validation, implementation uptake, or upstream review movement, so the draft optimization has faded as an active episode. Reopen if the PR advances, merges, or gains representative independent benchmarks.
2026-08-24T22:30:03Z
The latest pass adds only minor engagement, with no independent benchmark, quality validation, implementation uptake, or upstream review movement. The optimization remains a plausible but dormant candidate awaiting substantive PR evidence.
2026-08-22T21:33:14Z
No new benchmark, quality check, upstream review, or implementation evidence has followed the initial informal reproduction. The optimization remains plausible but unvalidated, and the episode has cooled while awaiting substantive PR movement.
2026-08-20T21:30:04Z
A first independent user report now claims roughly 1.5–2× faster prefill on Ornith 1.5 35B in a constrained CPU/GPU setup, extending the signal beyond the author’s EPYC benchmark. The anecdote is underspecified and lacks quality checks, so it supports active watching but not corroboration yet.
2026-08-20T20:35:55Z
The refreshed comments remain reactions and broad speculation, not independent benchmarks, quality validation, merge review, or implementation uptake. The case still rests entirely on the author’s narrow EPYC large-batch results and has not strengthened.
2026-08-20T18:34:05Z
Refreshed discussion remains amazement and broad-use speculation rather than independent reproduction, merge review, or evidence on representative serving workloads. The case still rests on the author’s narrow EPYC large-batch measurements, so its meaning has not strengthened.
2026-08-20T12:45:28Z
No independent benchmark, merge progress, or broader implementation evidence has arrived; the small engagement increase is repetitive attention and does not strengthen the author-reported performance claim.
2026-08-20T12:33:42Z
grounded: known/medium — Scott already holds the relevant position in “Hardware-aware local inference”: numerical precision, hardware capabilities, and compilation kernels should be exp
2026-08-20T12:31:30Z
origin walked (codex/luna, conf 0.98): anchor reddit.post.1vtgyzf -> echo.github.d0377455b0 by Colin Kealty (bartowski1182)
2026-08-20T12:30:20Z
case created — The draft upstream pull request includes concrete performance and perplexity measurements for a distinct CPU inference optimization.

Decision trace

08-27 08:38expireRepeated checks have produced no reproducible benchmark, quality validation, implementation uptake, or upstream review movement, so the draft optimization has faded as an active episode. Reopen if the
08-27 08:38alert_silentThe only delta is continued staleness; there is no new technical event or actionable evidence worth interrupting Scott for.
08-27 08:38alert_routeThe only delta is continued staleness; there is no new technical event or actionable evidence worth interrupting Scott for.
08-25 08:30repriceThe latest pass adds only minor engagement, with no independent benchmark, quality validation, implementation uptake, or upstream review movement. The optimization remains a plausible but dormant cand
08-25 08:30alert_silentThis is an unchanged reobservation rather than a consequential technical delta; minor engagement does not justify interrupting Scott before reproducible benchmarks or merge progress arrives.
08-25 08:30alert_routeThis is an unchanged reobservation rather than a consequential technical delta; minor engagement does not justify interrupting Scott before reproducible benchmarks or merge progress arrives.
08-23 07:33repriceNo new benchmark, quality check, upstream review, or implementation evidence has followed the initial informal reproduction. The optimization remains plausible but unvalidated, and the episode has coo
08-23 07:33alert_silentThe only new condition is elapsed time without fresh evidence; there is no consequential delta for Scott before the next briefing.
08-23 07:33alert_routeThe only new condition is elapsed time without fresh evidence; there is no consequential delta for Scott before the next briefing.
08-21 11:21sensor_dirtyengagement_update
08-21 07:30repriceA first independent user report now claims roughly 1.5–2× faster prefill on Ornith 1.5 35B in a constrained CPU/GPU setup, extending the signal beyond the author’s EPYC benchmark. The anecdote is unde
08-21 07:30alert_silentThe independent reproduction is directionally meaningful but remains a single informal comment without commands, timings, quality measurements, or upstream review; it can wait for a reproducible bench
08-21 07:30alert_routeThe independent reproduction is directionally meaningful but remains a single informal comment without commands, timings, quality measurements, or upstream review; it can wait for a reproducible bench
08-21 07:21sensor_dirtycomment_update
08-21 06:35repriceThe refreshed comments remain reactions and broad speculation, not independent benchmarks, quality validation, merge review, or implementation uptake. The case still rests entirely on the author’s nar
08-21 06:35alert_silentThe new delta is repetitive discussion with no consequential technical or release change; it can wait for independent reproduction or upstream review progress.
08-21 06:35alert_routeThe new delta is repetitive discussion with no consequential technical or release change; it can wait for independent reproduction or upstream review progress.
08-21 06:21sensor_dirtycomment_update
08-21 04:34repriceRefreshed discussion remains amazement and broad-use speculation rather than independent reproduction, merge review, or evidence on representative serving workloads. The case still rests on the author
08-21 04:34alert_silentThe new delta is repetitive commentary and engagement, with no confirming benchmark, quality evaluation, implementation uptake, or merge-status change; normal review is sufficient.
08-21 04:34alert_routeThe new delta is repetitive commentary and engagement, with no confirming benchmark, quality evaluation, implementation uptake, or merge-status change; normal review is sufficient.
08-21 04:21sensor_dirtyengagement_update
08-21 03:21sensor_dirtyengagement_update
08-21 02:21sensor_dirtyengagement_update
08-21 00:21sensor_dirtycomment_update
08-20 23:21sensor_dirtyengagement_update
08-20 22:45repriceNo independent benchmark, merge progress, or broader implementation evidence has arrived; the small engagement increase is repetitive attention and does not strengthen the author-reported performance
08-20 22:45alert_silentThe only change is minor engagement growth, with no new evidence about reproducibility, quality impact, merge status, or performance on representative serving workloads; this can wait for normal revie
08-20 22:45alert_routeThe only change is minor engagement growth, with no new evidence about reproducibility, quality impact, merge status, or performance on representative serving workloads; this can wait for normal revie
08-20 22:42alert_silentThe open PR and author-run benchmarks are substantive and relevant for future CPU-prefill testing, but the gains are demonstrated only on an EPYC 9654 under an unusually large batch-size workload. It
08-20 22:42surface_candidateThe open PR and author-run benchmarks are substantive and relevant for future CPU-prefill testing, but the gains are demonstrated only on an EPYC 9654 under an unusually large batch-size workload. It
08-20 22:42alert_routeThe open PR and author-run benchmarks are substantive and relevant for future CPU-prefill testing, but the gains are demonstrated only on an EPYC 9654 under an unusually large batch-size workload. It
08-20 22:33groundScott already holds the relevant position in “Hardware-aware local inference”: numerical precision, hardware capabilities, and compilation kernels should be explicit runtime concerns, with changes val
08-20 22:31promote_anchororigin walk conf 0.98
08-20 22:30createThe draft upstream pull request includes concrete performance and perplexity measurements for a distinct CPU inference optimization.