Edge0 claims its released SSD-streaming MoE framework runs its 35B tier at 14.9–17.7 tokens per second on an M4 Pro with 2.9 GiB peak active MLX memory at short contexts, potentially reducing accelerator-memory requirements for local inference without establishing equivalent total-system memory savings.

What is this?

Why it matters to Scott

Measured heat

How the heat travelled

Evidence (7) — ⭐ canonical anchor

edge0 An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction. English | 中文 edge0 is an open-source streaming MoE inference framework. It
generalizes the production-proven recipe — SSD expert offload +
Recover-LoRA + prerouter routing prediction — into an extensible
framework. The backend is isolated by design: the current MLX backend
runs on Apple Silicon, and additional platforms (CUDA, …) plug into the
same core abstractions. Two model tiers ship with the framework. Each tier is an end-to-end
release: the released checkpoint, the trained LoRA adapters, and the
trained prerouter heads work together as one unit. Tier Released checkpoint Inference profile edge0-35b Edge0/Edge0-35B-A3B-preview 4-bit, 40 layers, 256 experts, prerouter K=4 edge0-8b Edge0/Edge0-8B-A1B-preview 4-bit, 24 layers, 128 experts, prerouter K=8 Both checkpoints are built on open sparse-MoE base models (Qwen3.5-MoE
35B-A3B and the Ling 3.0 bailing hybrid respectively) and ship with the
LoRA and prerouter training done for this framework — the adapter files
are co-located with each checkpoint and load automatically, so edge0 serve <tier> runs the trained pipeline out of the box. Requirements OS / hardware : the MLX backend runs on macOS with Apple Silicon
(M1/M2/M3/M4). The CUDA backend is on the roadmap — no other
platforms are supported yet. Python : 3.10+ (3.12 recommended). Memory : ~2.9 GB peak active memory for edge0-35b , ~1.0 GB for edge0-8b (short contexts; see Benchmark ). Add
headroom for the OS, tokenizer, and long-context KV growth. Disk : the 4-bit checkpoints are ~23 GB ( edge0-35b ) and ~4.2 GB
( edge0-8b ); expert weights are mmapped and read on demand, they are
not loaded into RAM up front. Design transformers-style usage : AutoModel / AutoConfig / AutoEngine resolve the tier from the model name; Backend isolation : all MLX code lives under edge0/backends/mlx/ ;
the core logic (model specs, prerouter, streaming expert pool, server)
depends only on the backend facade ( edge0/backends/base.py ), so a new
backend implements the same facade ( backends/cuda/ is a reserved
slot) with zero changes to core code; Adapters as safetensors : LoRA and prerouter weights are .safetensors files with provenance metadata (source, version, owner
layers), resolved from the model directory or artifacts/ ; Model + adapters in one directory : a model directory holds both
the base checkpoint ( config.json / model*.safetensors / tokenizer)
and that model's adapters; upgrading adapters swaps adapter
files only — the base stays read-only and is never merged. Core mechanisms SSD expert offload : expert weights are streamed from storage on
demand; peak memory is bounded by the active set, not the parameter
count. Prerouter : a trained head predicts expert routing one step
ahead, so expert loads overlap the forward pass instead of stalling
it — up to +59% decode throughput; the gain grows with storage
latency, model size, and routed width K . Recover-LoRA : the int4 base is frozen and LoRA adapters are
trained by distillation from the FP teacher, recovering most of the
quantization loss at 4-bit (see Quality ).  Adapters stay
unmerged: one read-only base serves multiple adapter sets. Quick start 1) Install # Python >= 3.10; the MLX backend requires macOS with Apple Silicon python3.12 -m venv .venv && .venv/bin/pip install -e ' .[dev,fetch] ' 2) Download a model The two tiers are published on Hugging Face — each repo bundles the
base checkpoint and the trained LoRA + prerouter adapters in one
directory , so a single download is a ready-to-run model: Edge0/Edge0-35B-A3B-preview (~23 GB) Edge0/Edge0-8B-A1B-preview (~4.2 GB) # with the repo's helper (defaults to the two repos above): .venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models
.venv/bin/python scripts/fetch_models.py --tier edge0-8b --target-dir models # or directly with the CLI: .venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview     --local-dir models/edge0-35b
.venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview     --local-dir models/edge0-8b Either way you end up with a directory like: models/edge0-35b/
├── config.json, model-*.safetensors, tokenizer files   # base checkpoint
├── lora_edge0_35b.safetensors          # trained LoRA adapters
└── prerouter_edge0_35b.safetensors     # trained prerouter heads 3) Point edge0 at it Tier names resolve to local directories via environment variables
(where you put the download is up to you): export EDGE0_35B_MODEL= $PWD /models/edge0-35b export EDGE0_8B_MODEL= $PWD /models/edge0-8b Or skip the env vars entirely and pass the directory directly — the
tier is auto-detected from the checkpoint's config.json : edge0 demo models/edge0-35b
edge0 serve models/edge0-8b 4) Run # quick demo edge0 demo edge0-35b # serve (OpenAI-compatible /v1/chat/completions) edge0 serve edge0-35b curl http://127.0.0.1:8000/v1/chat/completions \
  -H ' Content-Type: application/json ' \
  -d ' {"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32} ' # 5) One-shot chat (pass --max-new to cap length; add --show-thinking to # print the model's reasoning block too) edge0 chat edge0-35b --prompt " Explain streaming inference in one sentence. " python -m edge0 ... is equivalent to edge0 ... . Python API from edge0 import AutoEngine from edge0 . server . chat import ChatMessage , ChatRequest , ChatSession engine = AutoEngine . from_pretrained ( "/path/to/model" ) # tier auto-detected req = ChatRequest ( model = engine . name , messages = [ ChatMessage ( role = "user" , content = "Hello!" )], max_tokens = 64 ,
) tokens , meta = ChatSession ( engine , req ). run () print ( engine . _tok . decode ( tokens )) engine . close () # release mmaps / expert cache examples/demo.py is the same minimal walkthrough ( edge0 demo runs
this exact path). Models and adapters Checkpoint : the original model directory ( config.json , model*.safetensors , tokenizer). edge0 serve <dir> / AutoEngine.from_pretrained(<dir>) detect the tier from config.json . Adapters (LoRA + prerouter, safetensors) are resolved from either
location automatically: the model directory (recommended): side by side with the base, e.g. lora_edge0_35b.safetensors + prerouter_edge0_35b.safetensors ; artifacts/ (repo root, gitignored): convert once from
training-side npz exports via edge0 convert-adapters --npz-dir ... . The published model repos bundle both the base checkpoint and the
current default adapter release, so scripts/fetch_models.py produces
a ready-to-run model directory.  Check each model's doc page for its
adapter provenance (training data, owner-layer layout). Both adapters are required for the prerouter + LoRA pipeline; if a
file is missing, edge0 fails with a clear message (or pass --no-prerouter / --no-lora to run the plain base model). Quality All benchmarks were run by us with OpenCompass under identical settings and parameters for both the edge0 models (int4 +
trained adapters + prerouter routing) and the original fp16 base models.
The loss of the edge0 pipeline is small: 3.9 points on average for
edge0-35b, 2.8 for edge0-8b (MMLU-Pro is even above the base). Max 100: Benchmark edge0-35b (int4) Qwen3.5-MoE 35B-A3B (fp16) edge0-8b (int4) Ling 3.0 tiny (fp16) AIME 2026 86.6 92.7 63.3 73.3 HumanEval 90.9 95.1 91.5 92.7 GPQA-Diamond 79.8 81.8 70.7 71.2 MMLU-Pro 81.0 84.6 70.1 65.8 IFBench 57.9 61.7 53.9 60.6 Average 79.2 83.2 69.9 72.7 Benchmark Measured with examples/bench.py (3.3k-token prompt prefill → 10 sampled
warmup steps → 200 timed sampled decode tokens, 2 runs per tier): Tier Decode speed Prefill throughput (cold / warm)* Peak active memory** Test machine edge0-35b 14.9–17.7 tok/s 113 / 140 tok/s 2.9 GiB Mac mini M4 Pro, 24 GB edge0-8b 23.9–25.3 tok/s 500 / 1428 tok/s 1.0 GiB Mac mini M4 Pro, 24 GB Cold = first request after process start (expert weights fault in from
SSD); warm = subsequent requests (page cache resident). Prefill numbers
are throughput over a ~3.3k-token prompt ( BENCH_LONG=1 ). * Peak active memory at short contexts (MLX allocator peak; expert weights
stream from SSD via mmap and are not resident). Long contexts add KV
cache: ~3.3 GiB on edge0-8b at 3.3k tokens. Reproduce: python examples/bench.py edge0-35b # via $EDGE0_35B_MODEL python examples/bench.py edge0-8b # via $EDGE0_8B_MODEL Tests pytest # unit tests (no real weights) EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q # real-weight generation; missing tiers are skipped .venv/bin/python scripts/e2e_smoke.py \
  --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b # staged vs exact consistency + generation smoke scripts/generate_example.py # full-pipeline API example examples/demo.py # minimal API walkthrough Documentation Architecture Attention / MoE / SSD streaming / prerouter Adding a model edge0-35b / edge0-8b License Apache-2.0, including vendored third-party code (see NOTICE ).

Interpretation history

• 2026-10-05T11:26:49Z
  An independent M5-mini hands-on run (17.5 tok/s decode, 390 tok/s prefill via carousel buffering) lands inside Edge0's claimed decode band and quantifies the technique's residual cost — the GPU still waits ~27% of decode on expert reads — reframing SSD streaming as a bounded, tunable pattern whose open questions are stall and prefill, not feasibility; Edge0's own figures remain unreproduced, so the case stays corroborated rather than accelerating.
• 2026-10-05T11:25:23Z
  evidence attached: reddit.post.1wy5fma — Independent hands-on SSD-streaming MoE run on Apple Silicon quantifies the expert-streaming decode stall (27% GPU wait) that SSD-streaming claims like Edge0's depend on.
• 2026-09-26T18:50:00Z
  grounded: converges/medium — Multiple independent implementations (Edge0 on MLX, Overspill/FreeToken on CUDA at ~3 tok/s, LayerStoRm at 24.5 tok/s, plus NVMe-paging siblings) have converged
• 2026-09-26T18:41:08Z
  Overspill/FreeToken extends disk-tier expert streaming beyond Apple Silicon to a CUDA implementation (85 GB DeepSeek-V4-Flash on 12 GB VRAM at ~3 tok/s), making the memory-capacity pattern multiply implemented across platforms — the technique itself is now corroborated by independent artifacts, while Edge0's specific 14.9–17.7 tok/s and 2.9 GiB figures remain self-reported and unreproduced, and the same post bounds the technique's throughput floor on slow tiers.
• 2026-09-26T18:25:02Z
  evidence attached: reddit.post.1wqwm3o — Independent disk-tier implementation (Overspill/FreeToken) running an 85GB DeepSeek-V4-Flash MoE on 12GB VRAM corroborates expert-streaming memory reduction, though at ~3 tok/s it also bounds the throughput cost; re-judging the streaming-MoE case should cite it.
• 2026-09-19T12:24:39Z
  The new M5 Pro–RTX 5090 comparison is an unsupported anecdote about a sibling implementation, not evidence of Edge0 performance parity or a new reproduction. Discussion adds no implementation or community expansion this round, leaving Edge0 a plausible but unvalidated local-serving option.
• 2026-09-19T05:27:59Z
  New comments sharpen benchmark-comparability questions but supply neither an Edge0 reproduction nor a credible direct contradiction. The secondhand DRAM comparison concerns a different memory hierarchy, while the prompt-processing caveat remains an unverified warning about a sibling model rather than a changed result.
• 2026-09-18T21:00:54Z
  A separate developer now reports using a 95.5 GiB SSD-streamed model on a 64 GB Mac for local coding and publishing a checkpoint and fork, strengthening the practical capacity argument beyond short benchmark runs. This remains developer-reported evidence for the broader technique, not a reproduction of Edge0’s performance or memory claims.
• 2026-09-18T20:22:41Z
  evidence attached: reddit.post.1wk14il — Independent runnable fork and model artifact support SSD-streamed MoE inference on constrained Apple hardware, materially strengthening the local-memory-capacity hypothesis.
• 2026-09-15T16:34:57Z
  A separate slotstream implementation now reports a 10%+ throughput gain from expert lookahead, adding implementation-level support for overlapping expert loads with inference rather than just enthusiasm for the technique. This modestly strengthens Edge0’s architectural plausibility, but does not independently validate its throughput, memory savings, predictive routing, or quality claims.
• 2026-09-15T16:23:36Z
  evidence attached: reddit.post.1wh3ek8 — An independent implementation reports a measured 10%+ gain from expert lookahead for SSD-streamed Qwen MoE inference, materially strengthening the case's streaming-performance hypothesis.
• 2026-09-11T09:32:53Z
  The refreshed sibling-project comments repeat previously identified testing questions without adding measurements or an Edge0 release change. The macOS release remains established, but its practical advantage for memory-constrained serving still needs independent tests separating active MLX allocations from total-system residency and sustained workload performance.
• 2026-09-11T07:30:58Z
  The refreshed sibling-project discussion remains questions and enthusiasm, not independent Edge0 measurements or a release change. Edge0 is still a testable macOS inference option, with total-system memory savings and sustained workload performance unresolved; the broader streaming pattern does not validate its specific claims.
• 2026-09-11T00:29:22Z
  The refreshed Cherenkov discussion identifies useful tests—routing hit rate, prefill latency and sustained thermals—but supplies questions rather than measurements. It neither validates nor refutes Edge0’s memory-saving claim; the next meaningful evidence remains independent Edge0 testing that accounts for page-cache residency and sustained workloads.
• 2026-09-10T22:38:33Z
  Cherenkov supplies an independent implementation of predictive expert streaming, strengthening the broader placement strategy but not corroborating Edge0’s specific throughput, memory or quality claims. Edge0 remains a concrete macOS experiment rather than a demonstrated reduction in total-system memory requirements; comparative tests should distinguish allocator usage from page-cache residency and sustained performance.
• 2026-09-10T22:22:56Z
  evidence attached: reddit.post.1wcwith — Independent corroboration: Cherenkov reports a similar SSD-streamed expert-working-set approach for running a large MoE on memory-constrained Apple hardware.
• 2026-09-10T16:41:48Z
  The retrieved first-party repository establishes a concrete macOS release worth testing, but the echo adds no independent corroboration of its performance or memory claims. This look adds no substantive evidence: practical savings still depend on total-system residency and cache behavior, and iPhone support remains unsupported.
• 2026-09-10T16:38:35Z
  grounded: converges/medium — Edge0’s claimed SSD expert streaming extends Scott’s Hardware-aware local inference position with a concrete placement strategy worth testing for memory-constra
• 2026-09-10T16:34:03Z
  case created — The repository provides runnable artifacts and reproducible measurements, but explicitly supports macOS rather than the iPhone claimed by the discussion title.

Decision trace

• 10-08 11:36review_screenComments add no reproduction of Edge0's numbers and no contradiction: sdroege_'s expert-prediction and keyboardhack's tiled-matmul ideas restate mechanisms already covered (predictive p
• 10-08 11:35review_screenjev screen borderline (noul=0.67) — luna review
• 10-06 07:22sensor_dirtycomment_update
• 10-05 22:26repriceAn independent M5-mini hands-on run (17.5 tok/s decode, 390 tok/s prefill via carousel buffering) lands inside Edge0's claimed decode band and quantifies the technique's residual cost — the
• 10-05 22:25attachIndependent hands-on SSD-streaming MoE run on Apple Silicon quantifies the expert-streaming decode stall (27% GPU wait) that SSD-streaming claims like Edge0's depend on.
• 10-05 22:24propose_attachIndependent hands-on SSD-streaming MoE run on Apple Silicon quantifies the expert-streaming decode stall (27% GPU wait) that SSD-streaming claims like Edge0's depend on.
• 09-27 11:28review_screenjev screen: no material development (noul=0.10)
• 09-27 11:21sensor_dirtycomment_update
• 09-27 04:50repriceOverspill/FreeToken extends disk-tier expert streaming beyond Apple Silicon to a CUDA implementation (85 GB DeepSeek-V4-Flash on 12 GB VRAM at ~3 tok/s), making the memory-capacity pattern multiply im
• 09-27 04:50groundMultiple independent implementations (Edge0 on MLX, Overspill/FreeToken on CUDA at ~3 tok/s, LayerStoRm at 24.5 tok/s, plus NVMe-paging siblings) have converged on SSD/NVMe expert streaming — a concre
• 09-27 04:25attachIndependent disk-tier implementation (Overspill/FreeToken) running an 85GB DeepSeek-V4-Flash MoE on 12GB VRAM corroborates expert-streaming memory reduction, though at ~3 tok/s it also bounds the thro
• 09-27 04:23propose_attachIndependent disk-tier implementation (Overspill/FreeToken) running an 85GB DeepSeek-V4-Flash MoE on 12GB VRAM corroborates expert-streaming memory reduction, though at ~3 tok/s it also bounds the thro
• 09-19 22:24repriceThe new M5 Pro–RTX 5090 comparison is an unsupported anecdote about a sibling implementation, not evidence of Edge0 performance parity or a new reproduction. Discussion adds no implementation or commu
• 09-19 22:24review_screenA new comment claims M5 Pro performance comparable to an RTX 5090 system, but provides no measurements, workload, or reproducible setup; its significance is unclear. The deleted datacenter comparison
• 09-19 21:20sensor_dirtycomment_update
• 09-19 15:27repriceNew comments sharpen benchmark-comparability questions but supply neither an Edge0 reproduction nor a credible direct contradiction. The secondhand DRAM comparison concerns a different memory hierarch
• 09-19 15:27review_screenThe update adds potentially relevant benchmark caveats and a cited claim that DRAM alternatives outperform SSD offload, but the excerpt is incomplete and does not establish a direct, comparable contra
• 09-19 14:21sensor_dirtycomment_update
• 09-19 07:00repriceA separate developer now reports using a 95.5 GiB SSD-streamed model on a 64 GB Mac for local coding and publishing a checkpoint and fork, strengthening the practical capacity argument beyond short be
• 09-19 06:22attachIndependent runnable fork and model artifact support SSD-streamed MoE inference on constrained Apple hardware, materially strengthening the local-memory-capacity hypothesis.
• 09-19 06:21propose_attachIndependent runnable fork and model artifact support SSD-streamed MoE inference on constrained Apple hardware, materially strengthening the local-memory-capacity hypothesis.
• 09-16 02:34repriceA separate slotstream implementation now reports a 10%+ throughput gain from expert lookahead, adding implementation-level support for overlapping expert loads with inference rather than just enthusia
• 09-16 02:23attachAn independent implementation reports a measured 10%+ gain from expert lookahead for SSD-streamed Qwen MoE inference, materially strengthening the case's streaming-performance hypothesis.
• 09-16 02:21propose_attachAn independent implementation reports a measured 10%+ gain from expert lookahead for SSD-streamed Qwen MoE inference, materially strengthening the case's streaming-performance hypothesis.
• 09-11 19:32repriceThe refreshed sibling-project comments repeat previously identified testing questions without adding measurements or an Edge0 release change. The macOS release remains established, but its practical a
• 09-11 19:32alert_silentBoth releases have already received heads-up routing decisions; this refresh supplies no new result, access change, or actionable development. Repeating the existing opportunity and caveats would spen
• 09-11 19:32alert_routeBoth releases have already received heads-up routing decisions; this refresh supplies no new result, access change, or actionable development. Repeating the existing opportunity and caveats would spen
• 09-11 19:21sensor_dirtycomment_update
• 09-11 17:30repriceThe refreshed sibling-project discussion remains questions and enthusiasm, not independent Edge0 measurements or a release change. Edge0 is still a testable macOS inference option, with total-system m
• 09-11 17:30alert_silentThe releases have already received heads-up routing decisions, and this refresh adds no consequential result beyond the previously identified testing questions. Another notification would repeat exist
• 09-11 17:30alert_routeThe releases have already received heads-up routing decisions, and this refresh adds no consequential result beyond the previously identified testing questions. Another notification would repeat exist
• 09-11 17:21sensor_dirtycomment_update
• 09-11 13:21sensor_dirtyengagement_update
• 09-11 10:29repriceThe refreshed Cherenkov discussion identifies useful tests—routing hit rate, prefill latency and sustained thermals—but supplies questions rather than measurements. It neither validates nor refutes Ed
• 09-11 10:29alert_silentBoth releases have already received heads-up routing decisions. These comments add no measured result or release change, so another notification would repeat the existing opportunity and caveats rathe
• 09-11 10:29alert_routeBoth releases have already received heads-up routing decisions. These comments add no measured result or release change, so another notification would repeat the existing opportunity and caveats rathe
• 09-11 09:21sensor_dirtycomment_update
• 09-11 08:38repriceCherenkov supplies an independent implementation of predictive expert streaming, strengthening the broader placement strategy but not corroborating Edge0’s specific throughput, memory or quality claim
• 09-11 08:38alert_silentCherenkov’s distinct release already received a heads-up routing decision in the supplied recent history. This attachment changes the context for Edge0, not its verified capabilities, and adds no cons
• 09-11 08:38alert_routeCherenkov’s distinct release already received a heads-up routing decision in the supplied recent history. This attachment changes the context for Edge0, not its verified capabilities, and adds no cons