2026-10-11 16:37 UTC

Agent Substrate’s maintainers claim their released Kubernetes runtime can multiplex stateful agent sandboxes at 10-times standard container density with sub-500-millisecond resume and zero-trust isolation, potentially lowering the infrastructure cost of large persistent agent fleets.

state: seedheat: mediumuncertainty: mediumknownscott: mediumagent-orchestration agent-infrastructure sandboxingAgent Substrate

What is this?

Agent Substrate is an open-source, Kubernetes-based runtime for operating large fleets of isolated, stateful agent sandboxes; Google Cloud announced its availability on GKE. It multiplexes mostly idle agent “actors” onto fewer workers, snapshotting suspended environments to local disk or Cloud Storage and restoring them when work arrives. Its maintainers claim 10× the density of standard container runtimes, sub-500 ms resume, more than 500 lifecycle activations per second, and kernel/network isolation using technologies including microVMs and gVisor. These are project and vendor claims, and one supplied source describes the software as experimental, early-stage, and not production-ready.

Why it matters to Scott

The radar already tracks Agent Substrate as the suspend/resume sandbox layer beneath Google’s AX orchestrator on `radar:google-ax-declarative-orchestration`. Its claimed density, resumability, and isolation directly bear on Scott’s long-running-agent and resumable-control-plane designs and could affect infrastructure choices if independently validated, but the supplied evidence remains vendor claims for experimental software.
ip:framework.long-running-agentsip:concept.sandboxed-executiondev:concept.resumable-agent-job-control-planedev:concept.deterministic-agent-control-planedev:project.silo-osradar:google-ax-declarative-orchestrationradar:concept.agent-infrastructureradar:concept.persistent-agentsradar:concept.agent-sandboxesradar:concept.agent-runtime
queries asked of Scott's wikis
  • persistent agent fleet infrastructure economics
  • stateful agent sandbox suspend resume
  • actor-worker multiplexing for agent runtimes
  • zero-trust isolation for agent tool execution
  • Kubernetes orchestration for coding-agent sandboxes
  • warm pools versus snapshot restore latency

Measured heat

now 0 pts/hpeak 0 pts/hcomments 0/hpeers p14momentum: steady2 platformsage 3482h
points/hour across evidence · reading as of 2026-10-12 02:59:37.977291+11:00 · deterministic, not a model opinion

How the heat travelled

05-19 14:00⭐ origin echo-reconstructedGoogle Cloud’s announcement says it is “Introducing Agent Substrate, a new open source project aimed at continuing to push the limits of age
Google Cloud on blog (echo) · attributed from hn.story.49787496
—
09-21 14:08first on hacker news · published · +3000.1hAgent execution runtime engineered to run millions of sandboxes
zorcan1
—
09-21 14:08amplified on hacker news 👑hn.story.49787496
zorcan1
peak 2 · 0 comments · 98% of case engagement
09-21 15:21our radar first saw it · +3001.4hdiscovery anchor: hn.story.49787496—

Evidence (2) — ⭐ canonical anchor

sourceobjectauthorscorecomments
🟧 hnAgent execution runtime engineered to run millions of sandboxes
Retrieved article excerpt

Open article · Retrieved 2026-09-21T15:25:03.349181+00:00

# Agent Substrate

[License](https://opensource.org/licenses/Apache-2.0)

NOTE: This is not an officially supported Google product. This project is not
eligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security).

## What is Agent Substrate?

Agent Substrate is a secure-by-default agent execution runtime engineered to run millions of sandboxes with 10x higher density than standard container runtimes. Purpose-built for the era of autonomous agents, Substrate delivers sub-500ms resume operations at over 500 suspend/resume activations per second with native zero-trust kernel and network isolation. It supports multiple sandbox technologies including microVMs and gVisor, enabling consistent lifecycle operations for all sandbox types.

At its core, Agent Substrate maps a larger set of “actors” (applications such as agents) onto a smaller set of ready “workers”, relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing. It provides functionality to manage an actor’s lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real time, and to route incoming traffic to them.

Agent Substrate is intended to be a low-opinion system. The workloads it manages don't have to be literal AI agents, but those are the best example of the kind of applications it is designed for. It is not an SDK for building agents, but rather a system for running them at scale.

Agent Substrate leverages Kubernetes for the infrastructure provisioning and worker lifecycle management (Kubernetes Pods). It builds on top of Kubernetes features like Pods and Pod autoscaling, while Agent Substrate provides agent-specific scheduling and control to achieve lower latency. Using Kubernetes as the underlying system enables consistent infrastructure management across all workloads types that are required for end to end agentic deployments and allows holistic infrastructure optimizations for RL scenarios that span agentic, inference and training cycles.

## Demo

[Agent Substrate Demo](https://www.youtube.com/watch?v=ZEzkCFJkzjY)

*Watch the Agent Substrate cluster multiplex ~250 stateful actors across just 8 physical pods.*

This demo highlights the core developer experience and "Agentic Infrastructure" capabilities of Substrate:

1. **Actor Teleport:** High-performance suspend and resume of actors onto any available worker in the pool with sub-second activation.
2. **State Persistence:** Persistent working memory (volatile RAM) and filesystem state preserved perfectly across hibernation cycles via full-state snapshots.
3. **Agent Multiplexing:** Demonstrates 30x+ oversubscription by "juggling" a large registry of stateful actors onto a small pool of shared physical pods.

To reproduce this demo in your own cluster, please refer to the detailed walkthrough in the **[Counter Demo](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md)**.

For more videos and walkthroughs, visit our YouTube channel: **[agent-substrate](https://www.youtube.com/channel/UCN9PPqlTtVxlcpbQ-NWpfZQ)**.

## Framework Agnostic & Compatibility

Agent Substrate is designed to be **framework and agent harness agnostic**. Because it manages standard OCI containers at the kernel level (via gVisor), it can host agents built on any stack.

- **Agent Development Kit (ADK):** Support for ADK agents with session state preservation across invocations as actor state. Ideal for all types of agents and stateful tool or subagent calls.
- **LangChain:** Ideal execution environment for LangChain agents and tool calls.
- **Claude Code, CodeX, and Antigravity:** Support for high-density, stateful coding environments that preserve system state and filesystem state across sessions.
- **Model Context Protocol (MCP):** Support for deploying secure, sandboxed MCP servers as Substrate Actors to provide durable tools for any model.

## Ecosystem & Examples

- **[Agent Executor](https://github.com/google/ax):** A distributed agent runtime that demonstrates building a secure, hyper-scalable agent harness on Agent Substrate (see the [announcement blog](https://cloud.google.com/blog/products/ai-machine-learning/agent-executor-googles-distributed-agent-runtime) and [integration guide](https://github.com/google/ax/blob/main/manifests/README.md)).
- **[kagent](https://github.com/kagent-dev/kagent):** A CNCF Sandbox project and Kubernetes-native framework for building, deploying, and managing AI agents that uses Agent Substrate to run sandboxed, stateful agent workloads (see the [announcement blog](https://kagent.dev/blog/the-future-of-kagent)).

## Status and compatibility

Agent Substrate is currently in early development. It is not ready for
production use, and the APIs are almost guaranteed to change. We are not
making any guarantees about backward compatibility at this stage, and
everything in this project may be changed.

### Supported Kubernetes Releases

Currently we aim to support the [latest stable release](https://kubernetes.io/releases/) of Kubernetes, and the previous minor release.

## Community

For announcements, technical discussions, and community support, please join
the **[ate-dev](https://groups.google.com/g/ate-dev)** Google Group.

We host a weekly community meeting every Thursday from 10:00am - 11:00am PST.

- Video call link: <https://meet.google.com/uhq-cxvn-dhy>
- Or dial: (US) +1 253-289-6971 PIN: 787 664 574 59#
- More phone numbers: <https://tel.meet/uhq-cxvn-dhy?pin=9044088223662>
- [Meeting notes](https://docs.google.com/document/d/1obSIvfcafLNniLYTQCcT2eCgxHqa2AQ3Ga7YTsju49s) for the weekly sync meeting
- [Recordings and transcripts](https://drive.google.com/corp/drive/u/0/folders/1rX1S6vPxPrR8dA1lEBuBEXkGKjHtG-mL) of all community meetings

We also have channels in the CNCF slack; [request an invite here](https://slack.cncf.io/)
if you don't have access.

- [#substrate-users](https://cloud-native.slack.com/archives/C0B6RCAJULW) to discuss using substrate.
- [#substrate-dev](https://cloud-native.slack.com/archives/C0B6M3E2J3D) to discuss developing substrate.

## Developing

Please see [CONTRIBUTING.md](https://github.com/agent-substrate/substrate/blob/main/CONTRIBUTING.md) for guidelines on contributing to
the project. We welcome contributions of all kinds, but the project is VERY
young. Our immediate focus is on building out the core system and demos, so we
may not be able to review or merge contributions that don't align with those
goals in the near term.

## Quickstart (Development)

To quickly set up the complete environment:

1. Make sure you have [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [`docker`](https://www.docker.com/) installed and configured on your dev machine. We will automatically manage other dependencies via Go, including [`kind`](https://kind.sigs.k8s.io/).
2. Run the following steps:

```
# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
hack/create-kind-cluster.sh

# install ate, PostgreSQL, rustfs
hack/install-ate-kind.sh --deploy-ate-system

# install counter demo
hack/install-ate-kind.sh --deploy-demo-counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create a counter actor in the demo's atespace (--template names the
# actor template, resolved in the actor's atespace)
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

# port-forward the network router to bind to local port `8000`
kubectl port-forward -n ate-system svc/atenet-router 8000:80
```

3. In a **separate terminal**, send an HTTP request to increment the counter:

```
curl -X POST \
   -H "ate-target-actor: ate-demo-counter/my-counter-1" \
   -i http://localhost:8000/
```

Worker capacity is versioned: the dataplane (the atelet DaemonSet and the
worker pods) schedules only on nodes that carry the
`ate.dev/substrate-version` label, and the install stamps it on every node
that exists when it runs. A node added later hosts no workers until you label
it with the installed version
(`kubectl label node <node> ate.dev/substrate-version=<build version>`).
`kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version`
prints the installed version, off the atelet DaemonSet the install created.

### GKE Quickstart (Development)

1. Create and configure your environment file:

   ```
   cp hack/ate-dev-env.sh.example .ate-dev-env.sh

   # Edit .ate-dev-env.sh to match your project and preferences, then source it:
   source .ate-dev-env.sh
   ```
2. Enable application-default credentials for gcloud:

   ```
   gcloud auth application-default login --project=${PROJECT_ID}
   ```
3. Provision the required GCP resources (GKE cluster, GCS, and IAM bindings):

   ```
   go run ./tools/setup-gcp bootstrap
   ```

   On a fresh project this step also creates the atelet Workload Identity IAM
   grants that snapshots depend on — see
   [what `create iam` actually grants](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md#what-create-iam-actually-grants)
   to audit them or apply them manually. If you bring your own cluster instead,
   note the required Kubernetes beta APIs can only be enabled **at cluster
   creation** — see the [Create Cluster warning](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md#2-create-cluster).
4. Deploy the Agent Substrate system to your cluster:

   ```
   ./hack/install-ate.sh --deploy-ate-system
   ```

   Nodes that GKE adds later (autoscaling, auto-repair, node upgrades) are
   born with the node pool's labels, so the pool needs
   `ate.dev/substrate-version` too; see
   [Node version labels](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md).
5. You can then deploy the sample applications. See [demos/counter/README.md](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md) or [demos/sandbox/README.md](https://github.com/agent-substrate/substrate/blob/main/demos/sandbox/README.md) for detailed walkthroughs.

   ```
   ./hack/install-ate.sh --deploy-demo-counter
   ```

#### Custom Setup and Deployment

You can run individual setup steps to create GCP resources as needed. See `go run ./tools/setup-gcp --help` for available options. For example:

```
go run ./tools/setup-gcp create cluster
go run ./tools/setup-gcp create bucket
```

To run the PostgreSQL store backend on Cloud SQL — with IAM database
authentication and no passwords — see [tools/setup-gcp/cloud-sql.md](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/cloud-sql.md).

Similarly, you can deploy or cleanup specific Agent Substrate components using the installation script. See `./hack/install-ate.sh --help` for all options.

```
# Re-deploy only ate-apiserver of the ATE system
./hack/install-ate.sh --deploy-ate-apiserver

# Delete everything (core system and all demos)
./hack/install-ate.sh --delete-all
```

#### Tearing down resources (GCP)

If you need to delete the resources created by the setup script, you can use the provided script `hack/teardown.sh`. This script will delete resources in the reverse order of creation and handles partial failures gracefully.

```
./hack/teardown.sh --all
```

Or run individual teardown steps as needed (see `./hack/teardown.sh` for available options).

#### Tearing down local `kind` resources

If you need to delete the local `kind` cluster and its registry (if it was created by `hack/create-kind-cluster.sh`):

```
./hack/delete-kind-cluster.sh
```

## Demos

We provide several sample applications demonstrating Agent Substrate's capabilities:

1. **[Counter Demo](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md)**: A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and on-demand actor resumption and routing via the Substrate router.
2. **[Sandbox Demo (Antigravity)](https
zorcan120
🟧 echo.blog ⭐Google Cloud’s announcement says it is “Introducing Agent Substrate, a new open source project aimed at continuing to push the limits of ageGoogle Cloud——

Interpretation history

Decision trace