2026-11-21 23:03 UTC

Hao AI Lab open-sources agent-led verification workflow and prototype serving engine

state: significantheat: mediumuncertainty: mediumrelevance: highllm-infrastructure formal-verification coding-agentsHao AI LabYuxuan ZhangMinshen ZhangJunda SuYi SunJunda ChenHao Zhang

What is this?

Hao AI Lab at UCSD, with a six-person team led by Yuxuan Zhang, announced vstack and veriserve in a first-party post dated November 10, 2026. Vstack is an open-source five-stage workflow for using coding agents, model checkers and theorem provers to verify software against pinned specifications and production-code behavior. Veriserve is a prototype serving engine for disaggregated LLM inference, built around PdLedger, a versioned KV-page lifecycle design. The lab reports that applying vstack exposed concurrency-bug classes in SGLang and vLLM and led to six proposed upstream pull requests; it also reports a large formal safety proof for PdLedger and low measured overhead. These are the team’s claims and evaluation results; the supplied sources do not establish that the proposed fixes were merged or independently validate the results.

Why it matters

This targets a consequential bottleneck as coding agents take on more systems work: establishing that generated infrastructure behaves safely under rare asynchronous races. If the workflow proves reproducible and maintainable, it could make rigorous verification more accessible to fast-moving open-source projects and help reduce silent data corruption in distributed inference—not just improve code generation. Veriserve’s design illustrates a concrete trade-off: without evidence that hardware transfers have stopped, it quarantines exposed pages rather than risk reusing them, potentially sacrificing capacity for safety. The prototype and reported proofs are promising, but formal guarantees apply to the specified abstract model under stated assumptions, not automatically to the complete engine or hardware stack.

Current assessment

Hao AI Lab announced and open-sourced vstack, a five-stage formal-verification workflow for coding agents, and veriserve, a prototype disaggregated LLM serving engine with a formally modeled KV-page lifecycle. The lab reports that vstack exposed concurrency-bug classes in SGLang and vLLM, with proposed upstream fixes, and that PdLedger’s abstract design passed extensive model checking and a TLAPS proof; it also reports low overhead in six paired GB200 runs. These are consequential first-party claims, not independent validation: the proof does not establish that the full implementation, GPU kernels, or RDMA providers are bug-free, and adoption and upstream fix status remain unresolved. The announcement’s PR accounting is inconsistent: its detailed discussion names vLLM #62065 and #66443, while its summary says six PRs and its final list includes #66443 but not #62065.

Measured heat

now 0 pts/hpeak 15 pts/hcomments 0/hpeers p14momentum: steady2 platformsage 273h
points/hour across evidence · reading as of 2026-11-22 09:27:02.556787+11:00 · deterministic, not a model opinion

How the heat travelled

11-10 13:00⭐ origin echo-reconstructedThe Hao AI Lab X post announces the work and links the blog, veriserve, and vstack: “We introduce vstack, a 5-stage formal verification work
Hao AI Lab on x (echo) · attributed from hn.story.50458579
—
11-11 20:58first on hacker news · published · +32.0hTowards Verifiable LLM Infra: Agent-Driven Proof of Agent-Implemented Inference
Yuxuan_Zhang13
—
11-11 20:58amplified on hacker news 👑hn.story.50458579
Yuxuan_Zhang13
peak 4 · 1 comments · 72% of case engagement
11-12 05:53amplified on hacker newshn.story.50463558
matt_d
peak 2 · 0 comments · 30% of case engagement
11-11 21:20our radar first saw it · +32.4hdiscovery anchor: hn.story.50458579—
pace: p50 vs 2009 stories at the 168h mark (now 273h old) — ahead of 5dive-self-hosted-coding-agents (1.2x), behind ai-screening-for-visa-document-fraud (0.9x)

Evidence (3) — ⭐ canonical anchor

sourceobjectauthorscorecomments
🟧 hnTowards Verifiable LLM Infra: Agent-Driven Proof of Agent-Implemented Inference
Retrieved article excerpt

Open article · Retrieved 2026-11-11T21:22:08.172034+00:00

# Towards Verifiable LLM Infra

November 10, 2026 · 12 minYuxuan Zhang, Minshen Zhang, Junda Su, Yi Sun, Junda Chen, Hao Zhang

[GitHub: vstack](https://github.com/hao-ai-lab/vstack)[GitHub: veriserve](https://github.com/hao-ai-lab/veriserve)[X: @haoailab](https://x.com/haoailab)

[![
](img/poster.png)](img/overview.mp4)

Coding agents are getting remarkably good at writing LLM infrastructure, from GPU kernels and schedulers to custom serving engines. Suppose an AI agent is strong enough to build a fast inference engine from scratch; in this [second half](https://ysymyth.github.io/The-Second-Half/) of the problem, the bottleneck shifts from capability to **verification**: how do we know the engine it wrote is actually correct? Passing latency and throughput benchmarks still leaves rare concurrency bugs hidden. As a single engineer begins orchestrating 100, 1,000, or even 10,000 coding agents in parallel, human code review hits a hard cognitive wall. Having agents review other agents does not solve this either; as [`pstack`](https://github.com/cursor/plugins/tree/main/pstack) author [Lauren Tan noted](https://x.com/poteto/status/2117437607940681849), adversarial reviews are mostly a waste of tokens, and it is far better to spend those tokens on empirical and formal verification.

In this post, we introduce [**`vstack`**](https://github.com/hao-ai-lab/vstack), a 5-stage formal verification workflow for coding agents, and [**`veriserve`**](https://github.com/hao-ai-lab/veriserve), a prototype **verifiable LLM serving engine** with PD disaggregation built around a formally verified KV-page lifecycle design model. Applying `vstack` to production Prefill-Decode serving uncovered three classes of real RDMA and cache concurrency bugs across SGLang and vLLM, resulting in **6 proposed upstream pull requests**. Meanwhile, `veriserve` backs its `PdLedger` design model with a **226-module, 516,973-obligation [`TLAPS`](https://proofs.tlapl.us/) safety proof** while adding only **`0.18%` mean p50 `TPOT` overhead** on NVIDIA GB200. **We believe verifiable serving defines the second half of LLM infra as autonomous coding agents write the deepest layers of production systems.**

---

## Why We Need Verifiable Serving[#](https://haoailab.com/blogs/verifiable-llm-infra/#why-we-need-verifiable-serving)

Why is verifying an LLM serving engine so hard in the first place? Over the past three years, inference engines like [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang) have evolved from single-node executors into asynchronous distributed systems, layering [**paged KV memory**](https://arxiv.org/abs/2309.06180), [**shared prefix caching**](https://proceedings.neurips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html), and [**disaggregated prefill-decode RDMA transfers**](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin) over the same pool of physical GPU KV pages. When a request times out or aborts, its cleanup now races across RDMA network cards, CPU cache trees, and GPU attention kernels simultaneously. Because [GPUDirect RDMA](https://docs.nvidia.com/cuda/gpudirect-rdma/) writes directly into GPU memory without involving the target CPU, a lifecycle mismatch often does not crash the server, but returns `HTTP 200 OK` while silently mixing one user’s KV cache into another user’s response. Recent [LMSYS](https://www.lmsys.org/blog/2026-10-26-cf-security/) and [Cloudflare](https://blog.cloudflare.com/kv-cache-exposure-remediation/) disclosures highlight this risk in production, documenting both cross-node RDMA abort races and a rejected-request KV handoff bug that exposed data across **155 user responses**.

Worse yet, **catching these silent concurrency bugs on real hardware is slow and prohibitively expensive**. Standard CI unit tests run on a single machine and only cover known paths, while subtle distributed races under rare asynchronous interleavings rarely surface in CI. Reproducing an RDMA lifecycle race requires multi-GPU RDMA hardware like H100 or GB200, separate prefill and decode instances, and tight cross-worker timing. Flushing them out with long stress tests on GPU clusters burns scarce compute and still cannot guarantee that all race conditions are gone.

To catch these deep concurrency bugs without burning GPU hours, we turn to **formal verification**: modeling the system as a state machine and combining `TLA+` model checking with [`Z3`](https://github.com/Z3Prover/z3) and [`TLAPS`](https://proofs.tlapl.us/) theorem provers to exhaustively explore millions of asynchronous schedules and prove safety invariants on CPUs. In the past, formal verification was impractical for AI infrastructure because engines like vLLM and SGLang merge new pull requests continuously, while manually writing and updating formal specifications and proofs takes months of human effort. **Today, coding agents give us a practical way to bridge this gap.** Because LLMs can read large systems codebases and generate thousands of lines of formal specifications and proofs far faster than manual proof engineering, we can guide coding agents to specify, model-check, and prove critical subsystems, making **verifiable serving** practical in fast-moving open-source projects.

---

## How We Verify: Formal Verification with `vstack`[#](https://haoailab.com/blogs/verifiable-llm-infra/#how-we-verify-formal-verification-with-vstack)

Turning agent-driven formal verification into a reliable workflow requires solving two immediate challenges. First, the hardest part of formal verification is not running the solver, but writing an accurate **Specification**: a requirement the software must never violate under any asynchronous schedule. Thanks to open-source [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang), we do not invent these requirements in a vacuum, but ground them directly in real user issues, bug-fix PRs, and GPU kernel preconditions.

Second, even with a clear specification, simply asking a coding agent to write a formal model and prove it rarely works out of the box. When a model checker reports a race condition, **a naive coding agent will often reward-hack the verifier**: instead of surfacing the real bug, it weakens the invariant, deletes the timeout branch from the model, or adds an unrealistic `ASSUME` statement so the checker returns `0 errors`.

The vstack 5-stage verification workflow

Figure 1. The 5-stage vstack verification workflow. Common agent reward-hacks such as weakening the rule, deleting the timeout branch, or assuming the race away are blocked by pinned specifications, fidelity checks, and proof gates until a real fix passes.

To prevent reward hacking and make agent-driven verification reliable, we built [**`vstack`**](https://github.com/hao-ai-lab/vstack), an open-source plugin for Claude Code and Cursor that enforces a **5-stage verification workflow**:

1. **`Specification`:** Encode software requirements as formal safety invariants and pin the specification files by hash so unapproved edits to the proof goal are detected and rejected.
2. **`Abstraction`:** Build `TLA+` and `Z3` models where every state transition maps to an exact `file:line@commit` in the production codebase, splitting non-atomic cross-thread and cross-node operations into separate steps.
3. **`Validation`:** Before trusting any proof, verify model fidelity against real code: the historical buggy code must fail on the model, the patched code must pass, removing a guard from the fix must fail again under nonredundant mutation checks, and recorded execution traces must replay cleanly through the model.
4. **`Verification`:** Run [`TLC`](https://docs.tlapl.us/using:tlc:start) breadth-first search to find the shortest counterexample trace within the bounded model, then use [`Apalache`](https://apalache-mc.org/), [`Z3`](https://github.com/Z3Prover/z3), and [`TLAPS`](https://proofs.tlapl.us/) to prove that the modeled state machine and GPU KV-buffer address arithmetic hold under our explicit configuration assumptions.
5. **`Continuous Checking`:** Re-run the specification suite on new pull requests on CPUs, and mirror the proven preconditions as lightweight runtime checks in the engine.

---

## Case Study: Formally Verifying Prefill-Decode Disaggregated Serving[#](https://haoailab.com/blogs/verifiable-llm-infra/#case-study-formally-verifying-prefill-decode-disaggregated-serving)

As a prototype, we applied `vstack` to **Prefill-Decode Disaggregated Serving**, where a Prefill node computes the prompt’s KV cache and transfers it over one-sided RDMA to a Decode node that shares cached prefixes and runs the Decode attention kernel, as shown in **Figure 2**. For this core KV page subsystem, we define **four specifications**:

Silent KV corruption under disaggregated Prefill-Decode RDMA

Figure 2. Silent KV corruption under disaggregated Prefill-Decode RDMA. If Decode reclaims request A’s GPU page on a local timeout and assigns it to request B while Prefill’s one-sided RDMA write is still in flight, A’s late payload overwrites B’s KV page and the server still returns HTTP 200.

1. **`Isolation`:** In-flight RDMA transfers or late control messages from an old request must never read or overwrite a new request’s memory or state.
2. **`Integrity`:** Every KV page read by the Decode kernel or published into the global prefix cache must contain complete, uncorrupted data from the intended request generation.
3. **`Finality`:** Every active request and draining memory page must reach a terminal state in finite steps rather than hanging forever.
4. **`Conservation`:** Every physical page is strictly accounted for: a page exposed to in-flight RDMA returns to `FREE` only after hardware stop evidence arrives, or else stays permanently isolated in `QUARANTINED` accounting.

### Uncovering Real Concurrency Issues in SGLang and vLLM[#](https://haoailab.com/blogs/verifiable-llm-infra/#uncovering-real-concurrency-issues-in-sglang-and-vllm)

Checking the production Prefill-Decode implementations of [SGLang](https://github.com/sgl-project/sglang) and [vLLM](https://github.com/vllm-project/vllm) against these four specifications on CPUs uncovered **three classes of concurrency issues**, for which we reproduced selected RDMA failure paths on NVIDIA GB200 and control-plane deadlocks on CPUs, and submitted **6 proposed upstream pull requests**:

- **Reclaiming Transfer Slots or KV Blocks While RDMA Is Still In Flight:** When a request aborts or times out, one-sided RDMA transfers already queued on the NIC cannot be canceled in hardware on the studied transport paths; yet both engines allowed new requests to reuse the same transfer identity or reallocate the expired GPU page before the hardware transfer stopped, violating **`Isolation`**. On GB200, we reproduced this race in both SGLang and vLLM, where late RDMA accesses silently read or overwrote another request’s KV cache under `HTTP 200 OK`. We proposed upstream fixes in SGLang [`#48883`](https://github.com/sgl-project/sglang/pull/48883), [`#48889`](https://github.com/sgl-project/sglang/pull/48889), [`#48960`](https://github.com/sgl-project/sglang/pull/48960), [`#48979`](https://github.com/sgl-project/sglang/pull/48979), and [`#48982`](https://github.com/sgl-project/sglang/pull/48982), alongside a related vLLM pull-validation PR [`#62065`](https://github.com/vllm-project/vllm/pull/62065) by [`@vadiklyutiy`](https://github.com/vadiklyutiy).
- **Publishing Race-Corrupted KV Pages into the Global Prefix Cache:** When an RDMA transfer finishes, the Decode node can insert the received KV pages into the global prefix cache without checking whether the page’s lifecycle generation was invalidated mid-flight, violating **`Integrity`**. In our vLLM GB200 experiment, race-corrupted KV pages entered the prefix cache and corrupted a subsequent request that reused the c
Yuxuan_Zhang1341
🟧 echo.x ⭐The Hao AI Lab X post announces the work and links the blog, veriserve, and vstack: “We introduce vstack, a 5-stage formal verification workHao AI Lab——
🟧 hnTowards Verifiable LLM Inframatt_d20

Interpretation history

Decision trace