Artificial Analysis claims its Pocket-Scale Inference benchmark provides useful comparative measurements of local LLM performance on smartphones, giving builders a practical basis for selecting on-device models and hardware.

state: expiredheat: lowuncertainty: highconvergesscott: mediumlocal-inference edge-ai inference-economicsArtificial Analysis

What is this?

Artificial Analysis launched “Benchmarking pocket-scale inference,” comparing small language models and mobile phones using quantized builds that fit within 8 GB of memory and are served with llama.cpp. Artificial Analysis administers intelligence evaluations selected for mobile use, while Liquid AI develops and runs inference tests on real devices; Artificial Analysis says it independently validated Liquid AI’s measurement process. Reported measures include wall-clock time for processing a 1,024-token prompt and generating 256 tokens, although the supplied snippets do not establish how predictive the benchmark is of developers’ applications beyond its chosen evaluations.

Why it matters to Scott

Artificial Analysis’s hardware-and-model comparisons converge with Scott’s vendor-neutral capability-audit and hardware-aware local-inference approach, potentially adding a practical selection input for quantized deployments. It is not yet high relevance because the supplied evidence does not show that its mobile evaluations predict Scott’s representative workloads or measure full application-level utility.

ip:concept.capability-auditip:concept.model-perishabilitydev:concept.hardware-aware-local-inferenceradar:intelligence-per-watt-local-ai-metricradar:concept.mobile-inferenceradar:concept.local-inferenceradar:concept.model-evaluation

queries asked of Scott's wikis

on-device model selection criteria
local inference hardware economics
edge AI latency memory and battery tradeoffs
quantization effects on model capability
benchmark validity for real-world agent workloads
local-first AI product architecture

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	Benchmarking Pocket-Scale Inference	sys42590	84	18
🟧 echo.blog ⭐	Published a dedicated benchmark of pocket-scale local inference on mobile phones.	Artificial Analysis	—	—

Interpretation history

2026-09-02T15:45:09Z
The launch discussion has exhausted itself without representative-device testing, workload validation, or production adoption. The benchmark remains an unvalidated upper-bound comparison, but this episode has faded and can be reopened if substantive validation or implementation appears.
2026-08-31T15:37:42Z
The expanded discussion remains repetitive amplification of known deployment caveats—flagship bias, RAM limits, and unclear NPU utilization—without representative-device results, workload validation, or production adoption. The benchmark still reads as a useful upper-bound comparison rather than a validated guide for shipping mobile applications.
2026-08-30T15:34:44Z
Refreshed discussion reiterates deployment constraints—RAM, flagship bias, and uncertain NPU support—but adds no independent validation, representative-device testing, or production adoption. The benchmark remains a potentially useful upper-bound comparison rather than a proven model-selection tool for shipping mobile applications.
2026-08-30T13:31:42Z
The discussion adds a weak independent report of broadly similar model results, but also surfaces a consequential limitation: flagship-heavy device coverage may make the benchmark a poor guide for production mobile deployment. It now merits watching as a comparative upper-bound artifact, while its workload and market representativeness remain unvalidated.
2026-08-30T04:26:43Z
No new methodology, results, independent validation, or developer adoption emerged; the benchmark remains a promising artifact whose practical predictive value is unproven. The unchanged, discussion-free reobservation cools attention but does not negate the case.
2026-08-30T04:25:22Z
grounded: converges/medium — Artificial Analysis’s hardware-and-model comparisons converge with Scott’s vendor-neutral capability-audit and hardware-aware local-inference approach, potentia
2026-08-30T04:23:27Z
case created — A first-party mobile-inference benchmark is a concrete, reusable artifact in a hot local-inference area and is distinct from existing device-specific cases.

Decision trace

09-03 01:45expireThe launch discussion has exhausted itself without representative-device testing, workload validation, or production adoption. The benchmark remains an unvalidated upper-bound comparison, but this epi
09-03 01:45alert_silentThe only trigger is staleness; no new evidence changes the benchmark’s credibility, coverage, or practical adoption, so there is nothing consequential to surface.
09-03 01:45alert_routeThe only trigger is staleness; no new evidence changes the benchmark’s credibility, coverage, or practical adoption, so there is nothing consequential to surface.
09-01 01:37repriceThe expanded discussion remains repetitive amplification of known deployment caveats—flagship bias, RAM limits, and unclear NPU utilization—without representative-device results, workload validation,
09-01 01:37alert_silentThe delta is additional discussion and engagement, not new evidence about benchmark validity or practical adoption; it can wait until representative hardware testing, application-level validation, or
09-01 01:37alert_routeThe delta is additional discussion and engagement, not new evidence about benchmark validity or practical adoption; it can wait until representative hardware testing, application-level validation, or
08-31 01:34repriceRefreshed discussion reiterates deployment constraints—RAM, flagship bias, and uncertain NPU support—but adds no independent validation, representative-device testing, or production adoption. The benc
08-31 01:34alert_silentThe new comments only elaborate already-known limitations and do not change the benchmark’s credibility or practical status; this can wait for representative device coverage, workload validation, or e
08-31 01:34alert_routeThe new comments only elaborate already-known limitations and do not change the benchmark’s credibility or practical status; this can wait for representative device coverage, workload validation, or e
08-31 01:21sensor_dirtycomment_update
08-30 23:31repriceThe discussion adds a weak independent report of broadly similar model results, but also surfaces a consequential limitation: flagship-heavy device coverage may make the benchmark a poor guide for pro
08-30 23:31alert_silentThe new comments refine the benchmark’s likely scope but provide neither rigorous validation nor a consequential adoption event; this can wait for evidence from representative devices, workloads, or p
08-30 23:31alert_routeThe new comments refine the benchmark’s likely scope but provide neither rigorous validation nor a consequential adoption event; this can wait for evidence from representative devices, workloads, or p
08-30 23:21sensor_dirtycomment_update
08-30 21:21sensor_dirtyengagement_update
08-30 19:21sensor_dirtyengagement_update
08-30 18:21sensor_dirtyengagement_update
08-30 17:21sensor_dirtyengagement_update
08-30 16:21sensor_dirtyengagement_update
08-30 14:26repriceNo new methodology, results, independent validation, or developer adoption emerged; the benchmark remains a promising artifact whose practical predictive value is unproven. The unchanged, discussion-f
08-30 14:26alert_silentThere is no consequential new delta beyond the already-known publication, so the next briefing loses nothing by waiting for validation or use in representative workloads.
08-30 14:26alert_routeThere is no consequential new delta beyond the already-known publication, so the next briefing loses nothing by waiting for validation or use in representative workloads.
08-30 14:25alert_silentArtificial Analysis has established that it published a dedicated mobile local-inference benchmark, but the supplied evidence contains no methodology, device/model coverage, measurements, or validatio
08-30 14:25surface_candidateArtificial Analysis has established that it published a dedicated mobile local-inference benchmark, but the supplied evidence contains no methodology, device/model coverage, measurements, or validatio
08-30 14:25alert_routeArtificial Analysis has established that it published a dedicated mobile local-inference benchmark, but the supplied evidence contains no methodology, device/model coverage, measurements, or validatio
08-30 14:25groundArtificial Analysis’s hardware-and-model comparisons converge with Scott’s vendor-neutral capability-audit and hardware-aware local-inference approach, potentially adding a practical selection input f
08-30 14:23createA first-party mobile-inference benchmark is a concrete, reusable artifact in a hot local-inference area and is distinct from existing device-specific cases.