Retrieved article excerpt
Open article · Retrieved 2026-09-20T18:22:56.205722+00:00
Problem
## What had to be solved
Self-hosted DeepSeek-class inference for real chat traffic: long prompts, heavy prefix reuse,
streaming API, TTFT matters more than end-to-end latency.
~80%
Prefix cache hits
~7.9k
Input tokens (P50)
~60
Peak QPS target
~2s
Prod TTFT P50 ballpark
What I tried
## Eight setups, in the order I actually ran them
Chat traffic reuses the same system prompt about 80% of the time. The question was how to stop
recomputing that prefix on every request. I started on the production-class model, burned money,
then dropped to a smaller model and compared three ways of moving the KV cache between GPUs.
Shorthand: **1P2D** means 1 GPU does prefill (read the prompt) and 2 GPUs do decode
(stream tokens). **LMCache MP** is a shared KV store in host RAM that every GPU talks to,
instead of handing cache privately from one GPU to another.
| What I tried | Model and GPUs | How KV moved | What happened |
| --- | --- | --- | --- |
| BF16 baseline | DeepSeek V3.2 on 8× H200 | All 8 GPUs used for tensor parallelism. No split between prefill and decode on one node. | The model ran, but the cluster was too expensive (~$60/hr class) to iterate on daily. |
| Classic LMCache P/D | DeepSeek V2-Lite on A100s | Prefiller pushed KV to the decoder per request, using LMCache’s built-in P/D connector. | Worked in smoke tests. Decoder ran out of memory under real concurrency. |
| NIXL P/D | V2-Lite on 2× A100 1 prefiller + 1 decoder | GPU-to-GPU KV handoff over NVLink/UCX. Each request copies cache privately to its decoder. | First stable split. Fast for a single request, but later chats could not reuse that prefix. |
| NIXL scale-out | V2-Lite on A100s 2 prefillers + 2 decoders | Same GPU-to-GPU handoff, more workers. | Throughput went up. Still no shared prefix cache across requests. |
| Shared L1 | V2-Lite on A100s 1 prefiller + 1 decoder | One central LMCache multiprocess server. Prefiller writes prefix KV once; decoder reads it back. | Prefix reuse started working. This is the architecture that matched the 80% hit rate. |
| Best V2-Lite run | V2-Lite on 3× A100 1 prefiller + 2 decoders | Same shared LMCache store, extra decode GPUs so token streaming can scale independently. | Asked 10 req/s, delivered 8.5. TTFT P50 0.52s vs ~2s in production. The headline result. |
| Production-class model | V3.2 NVFP4 on 8× B200 1 prefiller + 1 decoder | Same shared LMCache store, now on the real model and Blackwell GPUs. | The path works end to end. Sub-second TTFT only held to ~2 req/s, not the V2-Lite curve. |
| Cross-node scale-out | V3.2 NVFP4 on two 8× B200 rentals 1 prefiller + 3 decoders | Same design, split across two machines so decode can grow past one node. | Runbook is done. The provider had no private network between rentals, so the sweep never ran. |
What worked
## Shared L1 via LMCache MP + ZMQ
One **central** LMCache Multiprocess server (ZMQ port 6000) as a shared L1 KV layer in host RAM.
Every prefiller and decoder uses `LMCacheMPConnector` against the *same* store.
Prefiller writes prefix KV once; later requests with the same prefix skip full prefill.
Disagg proxy :9000
Prefiller :8100
Decoders :8200+
LMCache MP · shared L1 KV · ZMQ :6000
Results
## Asked vs delivered
Best V2-Lite stack: 1 prefiller + 2 decoders on 3× A100, shared LMCache store.
Production wanted ~2s time-to-first-token and ~60 req/s. Here is what that stack actually did.
0.52s
TTFT P50 when asked for 10 req/s
vs ~2s production ballpark
8.5 / 10
Req/s delivered vs asked
Sustained at the headline point
~11 / 60
Ceiling vs peak target
Saturates well below 60 QPS
Measured on **DeepSeek V2-Lite** (3× A100), not a production-class model.
On **V3.2 NVFP4** (8× B200) the same architecture held sub-second TTFT only to ~2 req/s.
This proves a *shared L1 KV layer is the right shape* for prefix-heavy chat, not production capacity.
### Could it keep up?
Gold: load we requested. Blue: load the stack actually served. After 10 they split:
served stalls near 11.\* That is this box’s ceiling, not 60.
### How long until the first token?
Wait time in seconds, not speed. Lower is better. Green: typical request (median).
Red: slow tail. Dashed lines: production (~2s / ~3.5s). At 10 requested, median is 0.52s.
\* One 1P2D box (3 GPUs). ~11 req/s is that hardware ceiling, not a failed design.
More QPS means more replicas at about 1:2 to 1:3 prefill:decode, e.g. 3 prefiller + 6 decoder GPUs.
Cost
## Total GPU spend
~$1,205
RunPod + Vast.ai (spot rentals)
Insights
## What I would do again
Small first
V2-Lite on cheap A100 before V3.2 on B200
Move fast
Experiments per week beat perfect plans
Shrink it
One request, one role, one path, then scale up
Technical depth
## Everything else is in the repo
Full write-ups, sweep tables, per-phase runbooks, and the resource bibliography.
- [Deep-dive index ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/README.md)
- [Engineering phases & architecture ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/BUILD.md)
- [Full benchmark tables ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/RESULTS.md)
- [Cost breakdown ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/COSTS.md)
- [Extended lessons ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/INSIGHTS.md)
- [Learning resources ↗](https://github.com/shubhmehta3121/inferpd/blob/master/docs/deep-dives/RESOURCES.md)
- [Repo (runbooks under `models/`) ↗](https://github.com/shubhmehta3121/inferpd)
[Open repo ↗](https://github.com/shubhmehta3121/inferpd)
[LinkedIn ↗](https://www.linkedin.com/in/shubh-mehta-2bb173221/)