Retrieved article excerpt
Open article · Retrieved 2026-09-13T13:21:38.428087+00:00
# ⚡ AgentJIT
### Just-In-Time Compiler for AI Agent Trajectories
**Compile flaky, 30-second multi-step AI Agent workflows into 5-millisecond deterministic code.**
[PyPI Version](https://pypi.org/project/agentjit/)
[Python Version](https://pypi.org/project/agentjit/)
[Free-Threaded No-GIL](https://pypi.org/project/agentjit/)
[License](https://github.com/eminsk/agentjit/blob/main/LICENSE)
[CI Test Suite](https://github.com/eminsk/agentjit/actions/workflows/ci.yml)
[Speedup](https://github.com/eminsk/agentjit/blob/main)
[Token Cost](https://github.com/eminsk/agentjit/blob/main)
[PRs Welcome](https://github.com/eminsk/agentjit)
[Open In Colab](https://colab.research.google.com/github/eminsk/agentjit/blob/main/notebooks/AgentJIT_Interactive_Demo.ipynb)
[**Quickstart**](https://github.com/eminsk/agentjit#quickstart) • [**Why AgentJIT?**](https://github.com/eminsk/agentjit#the-problem-in-2026-why-agentjit) • [**Architecture**](https://github.com/eminsk/agentjit#architecture) • [**Benchmarks**](https://github.com/eminsk/agentjit#benchmarks) • [**Speculative Execution**](https://github.com/eminsk/agentjit#speculative-execution--bailouts)
---
## 💥 The Problem in 2026: Why AgentJIT?
In 2026, autonomous AI agents solve real-world workflows across business, DevOps, and data analysis. However, running stochastic LLM loops in production faces four critical barriers:
1. **Massive Latency:** A standard 4-step agent workflow (`think -> tool -> observe -> think`) takes **15 to 45 seconds**.
2. **Exponential Costs:** Running the loop 10,000 times/day costs thousands of dollars in redundant API tokens.
3. **Flakiness & Hallucinations:** Even 98% reliability per step leads to compounding errors across multi-turn trajectories.
4. **Redundant Reasoning:** Most agent invocations execute the *exact same structural trajectory* with slightly different input parameters (e.g. different user IDs or dates).
---
## 🏗️ Architecture & Trajectory JIT Compilation
Just like **V8** compiles hot JavaScript into machine code, and **PyTorch `torch.compile`** traces dynamic tensors into optimized CUDA kernels, **AgentJIT traces dynamic agent trajectories and compiles them into pure, type-safe, ultra-fast Python code.**
```
[Dynamic Agent Task]
│
(1st run / warmup)
▼
┌──────────────────────┐
│ AgentJIT Tracer │ ── (Captures tool calls, data flow, variables)
└──────────────────────┘
│
▼
┌──────────────────────┐
│ DAG Flow Analyzer │ ── (Parameter generalization, dependency graph)
└──────────────────────┘
│
▼
┌──────────────────────┐
│ AST Code Generator │ ── (Synthesizes pure Python pipeline + Guards)
└──────────────────────┘
│
▼
┌────────────────────────────┐
│ Compiled JIT Pipeline │ ──► Subsequent runs: <1ms, $0 tokens!
└────────────────────────────┘
│
(Guard failure? Deopt!)
▼
[Fall back to LLM Agent]
```
---
## ⚡ Key Features
- 🏎️ **Up to 100,000x Speedup:** Hot paths drop from ~20,000 ms to **< 0.1 ms**.
- 💸 **100% Token Savings:** Once compiled, recurring workflows run completely locally with **0 LLM tokens consumed**.
- 🛡️ **Speculative De-Optimization (Bailout):** Automatically generates runtime input guards. If unexpected data formats or divergent branches appear, AgentJIT transparently falls back to the dynamic LLM agent.
- 🔍 **Transparent & Inspectable:** Inspect the exact Python code generated by the JIT with `agent.source_code`.
- 🧵 **Free-Threaded / No-GIL (PEP 703) Ready:** Thread-safe runtime fully tested on Python 3.13t and 3.14t for true multi-core parallel agent execution without GIL contention.
- 🔌 **Framework Agnostic:** Seamlessly wraps LangChain, CrewAI, AutoGen, OpenAI Tool calls, or native Python functions.
---
## 🚀 Quickstart
### Installation
```
pip install agentjit
# or with uv
uv add agentjit
```
### 10-Second Example
Decorate your agent with `@jit` and mark your tools with `@trace_tool`:
```
from agentjit import jit, trace_tool
# 1. Define your tools
@trace_tool()
def search_product(name: str):
return {"name": name, "price": 49.99, "stock": 120}
@trace_tool()
def apply_tax(price: float, tax_rate: float):
return round(price * (1.0 + tax_rate), 2)
# 2. Decorate your agent with @jit
@jit
def checkout_agent(product_name: str, tax_rate: float):
# This dynamic workflow could call an LLM (Claude, GPT, Gemini)
item = search_product(name=product_name)
total = apply_tax(price=item["price"], tax_rate=tax_rate)
return {"item": item["name"], "total": total}
# --- Run 1: Warmup & Tracing (runs dynamic agent, compiles to Python) ---
order1 = checkout_agent("Mechanical Keyboard", 0.19)
# --- Run 2+: Instant compiled execution (ZERO tokens, sub-millisecond!) ---
order2 = checkout_agent("Wireless Mouse", 0.19) # Takes 0.05 ms!
```
---
## 🔎 Inspecting Generated Code
You can view the exact synthesized Python code generated by the JIT at any time:
```
print(checkout_agent.source_code)
```
**Synthesized Output:**
```
def compiled_checkout_agent(product_name, tax_rate):
"""JIT-compiled trajectory pipeline generated by AgentJIT.
Executes deterministically in sub-millisecond time with zero token cost.
"""
# --- Speculative Guards ---
if not (product_name is not None):
raise GuardViolation("Argument 'product_name' must not be None", param="product_name")
if not (isinstance(product_name, str)):
raise GuardViolation("Argument 'product_name' must be of type str", param="product_name")
# --- Execution Steps ---
step_1_out = _tools['search_product'](name=product_name)
step_2_out = _tools['apply_tax'](price=step_1_out['price'], tax_rate=tax_rate)
# --- Return Final Result ---
return {'item': step_1_out['name'], 'total': step_2_out}
```
---
## 📊 Benchmarks
Benchmark comparing a simulated 3-step reasoning agent (15s latency, 2,500 tokens) vs AgentJIT compiled execution over 100 runs:
| Execution Mode | Mean Latency | 99th Percentile | Cost per 1k runs | Token Usage | Determinism |
| --- | --- | --- | --- | --- | --- |
| **Standard LLM Agent** | `14,820 ms` | `22,400 ms` | **$75.00** | 2,500,000 | ~94% |
| **AgentJIT (Warm Path)** | **`0.08 ms`** | **`0.12 ms`** | **$0.00** | **0** | **100%** |
| **Improvement** | **185,000x faster** | **186,000x faster** | **100% savings** | **Zero tokens** | **Rock-solid** |
---
## 🛡️ Speculative Execution & Bailouts
What happens when an input is unusual or triggers an unexpected branch?
AgentJIT uses **Speculative De-Optimization**:
1. Input variables are validated against synthesized guards.
2. If any guard fails (e.g. wrong type, missing required key) or a tool raises an unhandled exception, AgentJIT catches `GuardViolation`.
3. It seamlessly bails out to the dynamic LLM agent to handle the edge case.
4. Telemetry records the bailout for future multi-branch specialization.
```
# Normal input: runs compiled pipeline in 0.08ms
checkout_agent("Monitor", 0.19)
# Divergent input (e.g. invalid type): automatically bails out to dynamic agent
checkout_agent(12345, None) # Transparently de-optimizes, no crash!
```
---
## 🛠️ Telemetry & Observability
Monitor your compiled agents in real time:
```
print(checkout_agent.stats)
# Output:
# {
# "total_calls": 1500,
# "compiled_hits": 1492,
# "bailouts": 8,
# "compiled_hit_rate": 99.47,
# "total_time_saved_ms": 22380000.0,
# "total_tokens_saved": 3730000
# }
```
---
## 🗺️ Roadmap for 2026–2027
- **Core Tracer & DAG Flow Analyzer**
- **AST Code Generation with Speculative Guards**
- **De-optimization / Bailout Runtime**
- **`@jit` Decorator with Auto-Warmup**
- **Multi-Branch Polyhedral JIT:** Merge multiple execution paths into a unified control-flow graph (`if/else` branching synthesis).
- **eBPF-Isolated Micro-Sandbox:** Ultra-fast sub-millisecond process sandbox for compiled shell actions.
- **WebAssembly (Wasm) Export:** Compile agent trajectories into standalone Wasm binaries for browser and edge runtime.
---
## 📄 License
AgentJIT is open-source software licensed under the [Apache 2.0 License](https://github.com/eminsk/agentjit/blob/main/LICENSE).