llama.cpp contributor predatar claims PR #28086 raises IQ3-quantized MoE decode throughput on Apple Silicon Metal from about 65.6 to 73.9 tokens per second, potentially improving local sparse-model inference if merged.

state: watchingheat: lowuncertainty: highknownscott: lowlocal-inference llama-cpp apple-silicon inference-optimizationpredatarllama.cpp

What is this?

The supplied search answer reports that a proposed llama.cpp Metal optimization improves IQ3-quantized mixture-of-experts decode throughput on Apple Silicon from 65.6 to 73.9 tokens per second—about a 13% gain. The surrounding snippets establish that llama.cpp and MLX are competing local-inference paths on Apple Silicon and that performance varies substantially by model, quantization, workload, and hardware. However, the snippets do not independently verify predatar’s attribution or PR #28086, and the evidence title instead names PR #13388, “metal: optimize MoE for large batches,” so the exact primary artifact and merge status remain unclear.

Why it matters to Scott

This is another implementation-level example of Scott’s “Hardware-aware local inference” position, and the radar already tracks llama.cpp, Apple Silicon, MoE inference, and inference optimization. The claimed roughly 13% gain could become operationally useful, but the conflicting PR identifiers, unclear merge status, and lack of independent validation mean it does not yet change what Scott should build or argue.

dev:concept.hardware-aware-local-inferenceradar:concept.llama-cppradar:concept.apple-silicon-inferenceradar:concept.moe-inferenceradar:concept.inference-optimization

queries asked of Scott's wikis

local inference economics on Apple Silicon
GGUF and IQ quantization trade-offs
MoE inference optimization for local models
llama.cpp versus MLX backend strategy
benchmark methodology for local inference
Apple Silicon local agent deployment

Measured heat

now 0 pts/hpeak 0 pts/hcomments 0/hpeers p0momentum: steady2 platformsage 12530h

points/hour across evidence · reading as of 2026-10-12 02:59:37.977291+11:00 · deterministic, not a model opinion

How the heat travelled

05-07 14:00	⭐ origin echo-reconstructed	The original primary artifact is llama.cpp PR #13388, “metal : optimize MoE for large batches.” It describes remapping per-expert inputs, pe Georgi Gerganov (ggerganov) on github (echo) · attributed from reddit.post.1w46a73	—
09-01 08:55	first on r/LocalLLaMA · published · +11562.9h	Llama cpp metal moe optimization predatar	—
09-01 08:55	amplified on r/LocalLLaMA 👑	reddit.post.1w46a73 predatar	peak 6 · 7 comments · 69% of case engagement
09-03 14:13	amplified on r/LocalLLaMA	reddit.post.1w68lyj predatar	peak 4 · 2 comments · 32% of case engagement
09-01 09:20	our radar first saw it · +11563.3h	discovery anchor: reddit.post.1w46a73	—

Evidence (3) — ⭐ canonical anchor

source	object	author	score	comments
🟠 reddit	Llama cpp metal moe optimization LocalLLaMA	predatar	6	7
🟧 echo.github ⭐	The original primary artifact is llama.cpp PR #13388, “metal : optimize MoE for large batches.” It describes remapping per-expert inputs, pe	Georgi Gerganov (ggerganov)	—	—
🟠 reddit	llama cpp Metal moe improvements for decode + prefill + caching (tested on Qwen3-30B-A3B), looking for Qwen3.8-Flash-Next testers LocalLLaMA	predatar	4	2

Interpretation history

2026-09-09T16:30:03Z
This check adds no substantive evidence beyond engagement drift, leaving the Metal optimization package a contributor-reported lead rather than an established upstream improvement. Merge status and transferable performance gains remain unknown; the reconstructed older PR is separate testimony, not corroboration.
2026-09-07T16:27:15Z
The three-PR Metal effort remains a contributor-reported optimization lead, with no new evidence of reproducible gains or upstream availability. The older PR described in reconstructed testimony is a separate optimization, not corroboration of this package; absent substantive follow-up, a slower review cadence is appropriate.
2026-09-05T15:31:47Z
No substantive follow-up establishes broader performance gains or upstream adoption; the three-PR package remains a contributor-reported implementation lead. The reconstructed account of an older, differently numbered PR does not independently corroborate this effort, and current merge status remains unknown.
2026-09-03T14:36:47Z
The lead has broadened from a single benchmark claim into a coherent three-PR Metal optimization effort spanning MoE decode, prefill, and caching, which merits watching. It remains one contributor’s limited-hardware testing, with no independent reproduction, merge confirmation, or resolution of the original PR-attribution conflict.
2026-09-03T14:22:50Z
evidence attached: reddit.post.1w68lyj — Concrete llama.cpp Metal MoE prefill, decode, and caching optimizations provide additional implementation evidence for improving Apple Silicon local inference.
2026-09-01T13:42:54Z
The refreshed discussion adds only an intention to test, not a reproduction or clarification of the conflicting PR attribution. The case remains a narrow, contributor-reported benchmark awaiting a verifiable primary artifact, merge status, or independent results.
2026-09-01T09:31:34Z
No substantive new evidence clarifies the conflicting PR attribution, merge status, or benchmark reproducibility; the slight engagement increase is immaterial. The case remains an unvalidated contributor-reported optimization lead rather than an established llama.cpp improvement.
2026-09-01T09:28:56Z
grounded: known/low — This is another implementation-level example of Scott’s “Hardware-aware local inference” position, and the radar already tracks llama.cpp, Apple Silicon, MoE in
2026-09-01T09:26:26Z
origin walked (codex/luna, conf 0.94): anchor reddit.post.1w46a73 -> echo.github.cdb86f2faf by Georgi Gerganov (ggerganov)
2026-09-01T09:24:55Z
case created — A concrete upstream optimization with reported measurements is a distinct episode from the open llama.cpp CPU, NUMA, and CUDA optimization cases.

Decision trace

09-24 22:51review_dormantscheduled targets exhausted or 28 quiet days
09-24 22:51drop_targetsquiet through full ladder or over cap 8
09-10 02:30repriceThis check adds no substantive evidence beyond engagement drift, leaving the Metal optimization package a contributor-reported lead rather than an established upstream improvement. Merge status and tr
09-10 02:30alert_silentThere is no new implementation, access, or performance fact requiring Scott’s attention today. The narrow reported gains can wait for a briefing; an upstream availability change or independent benchma
09-10 02:30alert_routeThere is no new implementation, access, or performance fact requiring Scott’s attention today. The narrow reported gains can wait for a briefing; an upstream availability change or independent benchma
09-08 02:27repriceThe three-PR Metal effort remains a contributor-reported optimization lead, with no new evidence of reproducible gains or upstream availability. The older PR described in reconstructed testimony is a
09-08 02:27alert_silentThis staleness check supplies no new consequential delta. The reported narrow performance improvement does not currently change Scott’s implementation choices, so it can wait for a briefing rather tha
09-08 02:27alert_routeThis staleness check supplies no new consequential delta. The reported narrow performance improvement does not currently change Scott’s implementation choices, so it can wait for a briefing rather tha
09-06 01:31repriceNo substantive follow-up establishes broader performance gains or upstream adoption; the three-PR package remains a contributor-reported implementation lead. The reconstructed account of an older, dif
09-06 01:31alert_silentThis check adds no consequential implementation, availability, or benchmark fact. The narrow optimization package can remain in the briefing queue without interrupting Scott; independent measurements
09-06 01:31alert_routeThis check adds no consequential implementation, availability, or benchmark fact. The narrow optimization package can remain in the briefing queue without interrupting Scott; independent measurements
09-04 00:36repriceThe lead has broadened from a single benchmark claim into a coherent three-PR Metal optimization effort spanning MoE decode, prefill, and caching, which merits watching. It remains one contributor’s l
09-04 00:36alert_silentThe broader implementation package is useful evidence but remains unmerged and validated only by its author on two Apple systems; it can wait for an independent benchmark, upstream merge, or clearly a
09-04 00:36alert_routeThe broader implementation package is useful evidence but remains unmerged and validated only by its author on two Apple systems; it can wait for an independent benchmark, upstream merge, or clearly a
09-04 00:23alert_silentThe contributor now describes a coherent three-PR Metal optimization package covering MoE prefill, IQ2/IQ3 decode, and checkpoint reuse, but it remains unmerged and is explicitly seeking broader testi
09-04 00:23surface_candidateThe contributor now describes a coherent three-PR Metal optimization package covering MoE prefill, IQ2/IQ3 decode, and checkpoint reuse, but it remains unmerged and is explicitly seeking broader testi
09-04 00:23alert_routeThe contributor now describes a coherent three-PR Metal optimization package covering MoE prefill, IQ2/IQ3 decode, and checkpoint reuse, but it remains unmerged and is explicitly seeking broader testi
09-04 00:22attachConcrete llama.cpp Metal MoE prefill, decode, and caching optimizations provide additional implementation evidence for improving Apple Silicon local inference.
09-04 00:22propose_attachConcrete llama.cpp Metal MoE prefill, decode, and caching optimizations provide additional implementation evidence for improving Apple Silicon local inference.
09-01 23:42repriceThe refreshed discussion adds only an intention to test, not a reproduction or clarification of the conflicting PR attribution. The case remains a narrow, contributor-reported benchmark awaiting a ver
09-01 23:42alert_silentNo consequential new fact emerged: the comments ask about MLX and promise future testing but provide no measurements. This can wait for an actual reproduction, accessible PR evidence, or merge event.
09-01 23:42alert_routeNo consequential new fact emerged: the comments ask about MLX and promise future testing but provide no measurements. This can wait for an actual reproduction, accessible PR evidence, or merge event.
09-01 23:21sensor_dirtycomment_update
09-01 19:31repriceNo substantive new evidence clarifies the conflicting PR attribution, merge status, or benchmark reproducibility; the slight engagement increase is immaterial. The case remains an unvalidated contribu
09-01 19:31alert_silentNothing consequential changed since the prior review, and the claimed gain still lacks a verified primary artifact or independent reproduction. It can wait for a merge, accessible PR details, or addit
09-01 19:31alert_routeNothing consequential changed since the prior review, and the claimed gain still lacks a verified primary artifact or independent reproduction. It can wait for a merge, accessible PR details, or addit
09-01 19:29alert_silentA contributor-reported, apparently unmerged Metal optimization shows a roughly 13% decode gain on one IQ3_XXS MoE workload, but no primary PR contents, merge status, cross-model benchmarks, or indepen
09-01 19:29surface_candidateA contributor-reported, apparently unmerged Metal optimization shows a roughly 13% decode gain on one IQ3_XXS MoE workload, but no primary PR contents, merge status, cross-model benchmarks, or indepen
09-01 19:29alert_routeA contributor-reported, apparently unmerged Metal optimization shows a roughly 13% decode gain on one IQ3_XXS MoE workload, but no primary PR contents, merge status, cross-model benchmarks, or indepen
09-01 19:28groundThis is another implementation-level example of Scott’s “Hardware-aware local inference” position, and the radar already tracks llama.cpp, Apple Silicon, MoE inference, and inference optimization. The
09-01 19:26promote_anchororigin walk conf 0.94
09-01 19:24createA concrete upstream optimization with reported measurements is a distinct episode from the open llama.cpp CPU, NUMA, and CUDA optimization cases.