2026-11-22 12:39 UTC

FlashStates reports faster agentic inference with hybrid-state checkpoint reuse

state: seedheat: mediumuncertainty: mediumrelevance: highllm-inference agent-systems inference-efficiencyChenxi LiWeili XuTianyi ZhangTogether AIUniversity of Illinois Urbana-Champaign

What is this?

FlashStates is a serving-system project for agentic inference on hybrid language models, whose attention layers use KV caches while recurrent layers maintain a changing state. Its authors list affiliations with Together AI, the University of Illinois Urbana-Champaign and the University of Texas at Austin; the page identifies Weili Xu as project lead and Chenxi Li and Xu as equal contributors. The project describes Arbitrary State Checkpointing, which saves recurrent state at turn boundaries even when they fall inside KV-cache blocks, and manages each checkpoint together with the KV data it needs. Its project page links a paper and code. In November 2026, the project reported performance results on AgentX coding-agent workloads using B200 GPUs; no independent validation is included in the supplied evidence.

Why it matters

Agent turns often resend long contexts, so recomputing earlier tokens can waste inference capacity. FlashStates targets that waste by reusing both attention-cache data and recurrent state at the point a session resumes. If its reported gains generalize, this could reduce agent latency and serving cost, or let a fixed serving budget handle more sessions. The evidence is promising but comes from project-reported tests, not independent benchmarks.

Current assessment

New development: FlashStates presents fine-grained hybrid-state checkpointing and dependency-aware cache management for agentic LLM inference. The project says its eviction policy values checkpoints by prefill time saved per unit of memory freed. On its AgentX coding-agent evaluation, using two B200 GPUs per engine, the page reports up to 3.02× vLLM decode throughput, 3.14× prefill throughput and 88% lower p90 time to first token across GLM-5.3-Flash, Qwen3.8-Flash-Next and Nemotron 3 Super. A separate demo comparison reports 2.8× output tokens per minute, 3.7× lower median end-to-end latency and 16.3× lower median time to first token; these are distinct reported results, not directly interchangeable with the evaluation maxima. The page also reports gains with a Mooncake host-memory cache. Search results link to the preprint and code repository, but provide no independent reproduction or adoption evidence. Results are workload-, model- and hardware-specific; broader performance, operational tradeoffs and availability in commonly used serving stacks remain uncertain.

Measured heat

now 0 pts/hpeak 1 pts/hcomments 0/hpeers p23momentum: steady2 platformsage 335h
points/hour across evidence · reading as of 2026-11-22 23:22:28.401005+11:00 · deterministic, not a model opinion

How the heat travelled

11-08 13:00⭐ origin echo-reconstructedThe earliest primary artifact linked by the project is the authors’ research paper, submitted to alphaXiv on 2026-11-09. Its abstract says:
Chenxi Li, Weili Xu, Tianyi Zhang, Jue Wang, Zhongzhu Zhou, and Chenfeng Xu on paper (echo) · attributed from hn.story.50436074
—
11-10 04:29first on hacker news · published · +39.5hFlashStates Unlocking Flexible Hybrid State Management for Agentic LLM Inference
matt_d
—
11-10 04:29amplified on hacker news 👑hn.story.50436074
matt_d
peak 1 · 0 comments · 104% of case engagement
11-10 05:21our radar first saw it · +40.4hdiscovery anchor: hn.story.50436074—
pace: p12 vs 2006 stories at the 168h mark (now 335h old) — behind adobe-max-ai-agents-creative-tools-mcp (0.5x)

Evidence (2) — ⭐ canonical anchor

sourceobjectauthorscorecomments
🟧 hnFlashStates Unlocking Flexible Hybrid State Management for Agentic LLM Inference
Retrieved article excerpt

Open article · Retrieved 2026-11-10T05:22:44.345291+00:00

# Flash*States*

FlashStates: Unlocking Flexible Hybrid State Management for Agentic LLM Inference

[Chenxi Li](https://www.linkedin.com/in/chen-xi-li/)1,\*, [Weili Xu](https://weilixu.com)2,\*,†, [Tianyi Zhang](https://tonyzhang617.github.io/)1, [Jue Wang](https://juewang.me/about/)1, [Zhongzhu Zhou](https://www.zhongzhuzhou.org)1, [Chenfeng Xu](https://www.chenfengx.com)3

1Together AI2University of Illinois Urbana-Champaign3University of Texas at Austin

\*Equal contribution†Project lead

FlashStates decouples recurrent checkpoint positions from KV-cache block boundaries, achieving fine-grained state reuse without dense checkpoint storage.

[Paper](https://github.com/FutureMLS-Lab/FlashStates/blob/main/docs/FlashStates_Release.pdf)
[Code](https://github.com/FutureMLS-Lab/FlashState)

Demo replay

[![

](media/fs-48-agents-v13-poster.jpg)](media/fs-48-agents-v13.mp4)

0:00 / 0:00

FlashStates vs. vLLM

2.8×

output tokens per minute

22.0k → 61.7k

3.7×

lower P50 end-to-end request latency

46.1 → 12.4 s

16.3×

lower P50 time to first token

3.30 → 0.20 s

## Challenge

In a hybrid model, attention keeps KV for every token, but each recurrent layer keeps one state that it overwrites as it reads. Resuming at a position needs the KV and a checkpoint taken exactly there; we call the pair a hybrid state.

↻ Replay

## Where uncached tokens come from

Each agent turn re-sends its whole context. Ideally, the engine computes only the tokens added since the previous turn ended. Any earlier token it computes again is **redundant prefill**, caused by one of three misses.

Placement
Dependency
Capacity

↻ Replay
[Addressed by Arbitrary State Checkpointing](https://flashstates.ai/#step1)

cached KV
block boundary
checkpoint
evicted
new tokens
redundant prefill
ideal resume
actual resume

### How much each miss costs

Below 40 sessions, placement and dependency misses cause most of the redundant prefill in vLLM and SGLang. Capacity misses grow only from about 40 sessions.

vLLM
SGLang

Concurrent sessions

Token hit rate (a) and redundant prefill tokens per request from each kind of miss (b–d) as concurrency grows. vLLM and SGLang serve the same coding-agent traces with GLM-5.3-Flash NVFP4 and equal cache memory.

SGLang checkpoints every 256 tokens, so it has few placement misses (b). At 32 sessions, however, it keeps 4.2× as many checkpoints per request as vLLM. They compete for a fixed-size state pool, which causes its dependency misses (c).

## Method & Design

FlashStates saves recurrent checkpoints exactly where turns resume, and keeps or evicts each checkpoint together with the KV it depends on.

### Arbitrary State Checkpointing (ASC)

The next turn resumes where the previous request ended, usually inside a KV block. ASC saves the recurrent checkpoint exactly there: at request end it captures the hybrid state before cleanup, and if the position lies inside a prefill chunk, the kernel exports the checkpoint as a side output while the chunk continues. The KV block size and the chunk size do not change, and outputs stay bit-identical.

The checkpoint at *p* falls inside a KV block and inside a prefill chunk. Neither is split.

### Dependency-aware hybrid state

A recurrent checkpoint is useless without the KV before it, and that KV cannot resume a request without the checkpoint. FlashStates therefore keeps or evicts each hybrid state as one unit: its checkpoint and the KV only it needs. KV pages shared with other hybrid states are freed only when the last state that needs them leaves.

Hybrid states A and B extend a shared prefix. Evicting A frees only its checkpoint and the KV pages A alone uses, including its partial last page at full size.



shared KV
KV used by one state
recurrent checkpoint
freed when A is evicted

### Cost-aware eviction

Each hybrid state is priced by the prefill time it saves, measured once per model, per block of memory its eviction would free. Nothing is tuned per workload. Admission uses the same price: when memory is short, a new checkpoint is cached only if it is worth more than the cheapest one already cached. States that no request has resumed from for a while fall behind, so the newest checkpoint of each session, where its next turn resumes, is typically kept longest.

Add memory pressure
Reset

## Evaluation & Results

### Up to 3× throughput, across 3 hybrid models

We serve GLM-5.3-Flash, Qwen3.8-Flash-Next and Nemotron 3 Super using two B200 GPUs per engine. Evaluation on the AgentX coding-agent workload shows that FlashStates achieves up to 3.02× vLLM's decode throughput and 3.14× its prefill throughput, while cutting p90 time to first token (TTFT) by up to 88%.

vLLM
FlashStates

GLM-5.3Nemotron 3Qwen3.8

GLM-5.3-Flash NVFP4, 1M context

Concurrent sessions



### FlashStates also helps with Mooncake

With 512 GiB of host memory for cache and MTP enabled on GLM-5.3-Flash, FlashStates expands the Pareto frontier of the throughput–latency tradeoff, improves peak input throughput by 26%, and cuts p90 TTFT by up to 73% at the same concurrency.

FlashStates
vLLM
SGLang
best vLLM or SGLang

All readings

## BibTeX

```
@article{flashstates2026,
  title   = {FlashStates: Unlocking Flexible Hybrid State Management
             for Agentic LLM Inference},
  author  = {Li, Chenxi and Xu, Weili and Zhang, Tianyi and Wang, Jue and
             Zhou, Zhongzhu and Xu, Chenfeng},
  journal = {alphaXiv preprint},
  year    = {2026},
  url     = {https://www.alphaxiv.org/abs/2611.flashstate}
}
```
matt_d10
🟧 echo.paper ⭐The earliest primary artifact linked by the project is the authors’ research paper, submitted to alphaXiv on 2026-11-09. Its abstract says: Chenxi Li, Weili Xu, Tianyi Zhang, Jue Wang, Zhongzhu Zhou, and Chenfeng Xu——

Interpretation history

Decision trace