Ankit Sonthalia and coauthors introduce BOTTLED, a benchmark where LLM agents must convert general capabilities into cheap task-specific artifacts ('bottling'), finding that zero-shot performance doesn't predict bottling success but successful bottling can retain ~82% performance at 657x lower cost.

state: seedheat: lowuncertainty: mediumconvergesscott: highagent-distillation agent-evaluation inference-economics bottling-benchmarkAnkit SonthaliaHaritz PuertoAlexander RubinsteinMartin GubriSeong Joon Oh

What is this?

The case describes a new arXiv paper titled "Agent in a Bottle: Can LLM Agents Turn Their Capabilities into Cheap Artifacts?" submitted October 6, 2026, authored by Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein, Martin Gubri, and Seong Joon Oh. The paper introduces BOTTLED, a benchmark where LLM agents must distill general capabilities into cheap task-specific artifacts ("bottling"), reporting that zero-shot performance doesn't predict bottling success but successful bottling retains ~82% performance at 657x lower cost. No web search results were returned to independently verify these claims; the grounding rests entirely on the case's own summary.

Why it matters to Scott

The BOTTLED benchmark independently validates several load-bearing Scott positions: (1) zero-shot evaluation is the wrong unit — it fails to predict distilled-agent success, directly confirming 'benchmarking the wrong unit'; (2) 'bottling' is a clean instance of context arbitrage — compile general capability once into a cheap task-specific artifact, then run at 657x lower cost with 82% retention; (3) the result strengthens the inference-economics / AI-unit-economics frame with a concrete, extreme cost/performance datapoint. This is not merely an example of his patterns — it provides empirical ammunition that would update his evaluation-driven-development arguments and could serve as a dated-receipts publishing moment.

ip:concept.benchmarking-the-wrong-unitip:concept.context-arbitrageip:concept.ai-unit-economicsip:concept.evaluation-driven-developmentip:concept.transcript-distillationip:concept.demonstration-to-agent-compilationip:concept.model-dividenddev:concept.trace-backed-agent-comparisondev:concept.cheap-model-front-doorradar:concept.agent-benchmarksradar:concept.agent-evaluationradar:concept.model-distillationradar:concept.inference-economicsradar:concept.distillationradar:deepswe-mini-ranking-proxy

queries asked of Scott's wikis

agent distillation benchmark evaluation methodology
inference economics cost-performance tradeoffs agent compression
bottling distillation cheap artifacts from general agents
zero-shot vs distilled agent performance prediction
agent evaluation frameworks beyond zero-shot

Measured heat

now 0 pts/hpeak 6 pts/hcomments 0/hpeers p14momentum: steady1 platformsage 87h

points/hour across evidence · reading as of 2026-10-12 02:59:37.977291+11:00 · deterministic, not a model opinion

How the heat travelled

10-08 01:18	⭐ origin directly observed	Agent in a Bottle: Can LLM Agents Turn Their Capabilities into Cheap Artifacts? simonpure on hacker news	—
10-08 01:18	amplified on hacker news 👑	hn.story.50000881 simonpure	peak 1 · 0 comments · 106% of case engagement
10-08 01:21	our radar first saw it · +0.1h	discovery anchor: hn.story.50000881	—

pace: p10 vs 1243 stories at the 72h mark (now 87h old) — behind 3jsbench-llm-3d-generation-benchmark (0.5x)

Evidence (1) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn ⭐	Agent in a Bottle: Can LLM Agents Turn Their Capabilities into Cheap Artifacts? Retrieved article excerpt Open article · Retrieved 2026-10-08T01:22:51.386259+00:00 # Computer Science > Artificial Intelligence arXiv:2610.08775 (cs) [Submitted on 6 Oct 2026] # Title:Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? Authors:[Ankit Sonthalia](https://arxiv.org/search/cs?searchtype=author&query=Sonthalia,+A), [Haritz Puerto](https://arxiv.org/search/cs?searchtype=author&query=Puerto,+H), [Alexander Rubinstein](https://arxiv.org/search/cs?searchtype=author&query=Rubinstein,+A), [Martin Gubri](https://arxiv.org/search/cs?searchtype=author&query=Gubri,+M), [Seong Joon Oh](https://arxiv.org/search/cs?searchtype=author&query=Oh,+S+J) View a PDF of the paper titled Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?, by Ankit Sonthalia and 4 other authors [View PDF](https://arxiv.org/pdf/2610.08775) [HTML (experimental)](https://arxiv.org/html/2610.08775v1) > Abstract:Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability "bottling": the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance. Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. Bottling is also competitive with Jev, a "system one" model built especially for cheap, repetitive inference: Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. BOTTLED provides a basis for evaluating and improving agents' ability to invest limited resources in reusable solutions for large, repetitive workloads. \| \| \| \| --- \| --- \| \| Subjects: \| Artificial Intelligence (cs.AI) \| \| Cite as: \| [arXiv:2610.08775](https://arxiv.org/abs/2610.08775) [cs.AI] \| \| \| (or [arXiv:2610.08775v1](https://arxiv.org/abs/2610.08775v1) [cs.AI] for this version) \| \| \| <https://doi.org/10.48550/arXiv.2610.08775> Focus to learn more arXiv-issued DOI via DataCite (pending registration) \| ## Submission history From: Ankit Sonthalia [[view email](https://arxiv.org/show-email/c6ba5645/2610.08775)] [v1] Tue, 6 Oct 2026 17:57:19 UTC (165 KB) Full-text links: ## Access Paper: View a PDF of the paper titled Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?, by Ankit Sonthalia and 4 other authors - [View PDF](https://arxiv.org/pdf/2610.08775) - [HTML (experimental)](https://arxiv.org/html/2610.08775v1) - [TeX Source](https://arxiv.org/src/2610.08775) [license icon](http://creativecommons.org/licenses/by/4.0/ "Rights to this article") ### Additional Features - [Audio Summary](https://arxiv.org/audio/2610.08775) ### Current browse context: cs.AI [< prev](https://arxiv.org/prevnext?id=2610.08775&function=prev&context=cs.AI "previous in cs.AI (accesskey p)") \| [next >](https://arxiv.org/prevnext?id=2610.08775&function=next&context=cs.AI "next in cs.AI (accesskey n)") [new](https://arxiv.org/list/cs.AI/new) \| [recent](https://arxiv.org/list/cs.AI/recent) \| [2026-10](https://arxiv.org/list/cs.AI/2026-10) Change to browse by: [cs](https://arxiv.org/abs/2610.08775?context=cs) ### References & Citations - [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2610.08775) - [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2610.08775) - [Semantic Scholar](https://api.semanticscholar.org/arXiv:2610.08775) export BibTeX citation Loading... ## BibTeX formatted citation × loading... Data provided by: ### Bookmark [BibSonomy](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2610.08775&description=Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? "Bookmark on BibSonomy") [Reddit](https://reddit.com/submit?url=https://arxiv.org/abs/2610.08775&title=Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? "Bookmark on Reddit") Bibliographic Tools # Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ([What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer)) Connected Papers Toggle Connected Papers ([What is Connected Papers?](https://www.connectedpapers.com/about)) Litmaps Toggle Litmaps ([What is Litmaps?](https://www.litmaps.co/)) scite.ai Toggle scite Smart Citations ([What are Smart Citations?](https://www.scite.ai/)) Code, Data, Media # Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ([What is alphaXiv?](https://alphaxiv.org/)) Links to Code Toggle CatalyzeX Code Finder for Papers ([What is CatalyzeX?](https://www.catalyzex.com)) DagsHub Toggle DagsHub ([What is DagsHub?](https://dagshub.com/)) GotitPub Toggle Gotit.pub ([What is GotitPub?](http://gotit.pub/faq)) Huggingface Toggle Hugging Face ([What is Huggingface?](https://huggingface.co/huggingface)) ScienceCast Toggle ScienceCast ([What is ScienceCast?](https://sciencecast.org/welcome)) Demos # Demos Replicate Toggle Replicate ([What is Replicate?](https://replicate.com/docs/arxiv/about)) Spaces Toggle Hugging Face Spaces ([What is Spaces?](https://huggingface.co/docs/hub/spaces)) Spaces Toggle TXYZ.AI ([What is TXYZ.AI?](https://txyz.ai)) Related Papers # Recommenders and Search Tools Link to Influence Flower Influence Flower ([What are Influence Flowers?](https://influencemap.cmlab.dev/)) Core recommender toggle CORE Recommender ([What is CORE?](https://core.ac.uk/services/recommender)) - Author - Venue - Institution - Topic About arXivLabs # arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? [Learn more about arXivLabs](https://info.arxiv.org/labs/index.html). [Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2610.08775) \| Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))	simonpure	1	0

source

object

author

score

comments

🟧 hn ⭐

Agent in a Bottle: Can LLM Agents Turn Their Capabilities into Cheap Artifacts?

Retrieved article excerpt

Open article · Retrieved 2026-10-08T01:22:51.386259+00:00

# Computer Science > Artificial Intelligence

**arXiv:2610.08775** (cs)

[Submitted on 6 Oct 2026]

# Title:Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

Authors:[Ankit Sonthalia](https://arxiv.org/search/cs?searchtype=author&query=Sonthalia,+A), [Haritz Puerto](https://arxiv.org/search/cs?searchtype=author&query=Puerto,+H), [Alexander Rubinstein](https://arxiv.org/search/cs?searchtype=author&query=Rubinstein,+A), [Martin Gubri](https://arxiv.org/search/cs?searchtype=author&query=Gubri,+M), [Seong Joon Oh](https://arxiv.org/search/cs?searchtype=author&query=Oh,+S+J)

View a PDF of the paper titled Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?, by Ankit Sonthalia and 4 other authors

[View PDF](https://arxiv.org/pdf/2610.08775)
[HTML (experimental)](https://arxiv.org/html/2610.08775v1)
> Abstract:Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability "bottling": the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance. Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. Bottling is also competitive with Jev, a "system one" model built especially for cheap, repetitive inference: Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. BOTTLED provides a basis for evaluating and improving agents' ability to invest limited resources in reusable solutions for large, repetitive workloads.

|  |  |
| --- | --- |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | [arXiv:2610.08775](https://arxiv.org/abs/2610.08775) [cs.AI] |
|  | (or  [arXiv:2610.08775v1](https://arxiv.org/abs/2610.08775v1) [cs.AI] for this version) |
|  | <https://doi.org/10.48550/arXiv.2610.08775> Focus to learn more  arXiv-issued DOI via DataCite (pending registration) |

## Submission history

From: Ankit Sonthalia [[view email](https://arxiv.org/show-email/c6ba5645/2610.08775)]   
 **[v1]**
Tue, 6 Oct 2026 17:57:19 UTC (165 KB)

Full-text links:

## Access Paper:

View a PDF of the paper titled Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?, by Ankit Sonthalia and 4 other authors

- [View PDF](https://arxiv.org/pdf/2610.08775)
- [HTML (experimental)](https://arxiv.org/html/2610.08775v1)
- [TeX Source](https://arxiv.org/src/2610.08775)

[license icon](http://creativecommons.org/licenses/by/4.0/ "Rights to this article")

### Additional Features

- [Audio Summary](https://arxiv.org/audio/2610.08775)

### Current browse context:

cs.AI

[< prev](https://arxiv.org/prevnext?id=2610.08775&function=prev&context=cs.AI "previous in cs.AI (accesskey p)")
  |   
[next >](https://arxiv.org/prevnext?id=2610.08775&function=next&context=cs.AI "next in cs.AI (accesskey n)")

[new](https://arxiv.org/list/cs.AI/new)
 | 
[recent](https://arxiv.org/list/cs.AI/recent)
 | [2026-10](https://arxiv.org/list/cs.AI/2026-10)

Change to browse by:

[cs](https://arxiv.org/abs/2610.08775?context=cs)

### References & Citations

- [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2610.08775)
- [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2610.08775)
- [Semantic Scholar](https://api.semanticscholar.org/arXiv:2610.08775)

export BibTeX citation
Loading...

## BibTeX formatted citation

×

loading...

Data provided by:

### Bookmark

[BibSonomy](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2610.08775&description=Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? "Bookmark on BibSonomy")
[Reddit](https://reddit.com/submit?url=https://arxiv.org/abs/2610.08775&title=Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? "Bookmark on Reddit")



Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer *([What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))*

Connected Papers Toggle

Connected Papers *([What is Connected Papers?](https://www.connectedpapers.com/about))*

Litmaps Toggle

Litmaps *([What is Litmaps?](https://www.litmaps.co/))*

scite.ai Toggle

scite Smart Citations *([What are Smart Citations?](https://www.scite.ai/))*

Code, Data, Media

# Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv *([What is alphaXiv?](https://alphaxiv.org/))*

Links to Code Toggle

CatalyzeX Code Finder for Papers *([What is CatalyzeX?](https://www.catalyzex.com))*

DagsHub Toggle

DagsHub *([What is DagsHub?](https://dagshub.com/))*

GotitPub Toggle

Gotit.pub *([What is GotitPub?](http://gotit.pub/faq))*

Huggingface Toggle

Hugging Face *([What is Huggingface?](https://huggingface.co/huggingface))*

ScienceCast Toggle

ScienceCast *([What is ScienceCast?](https://sciencecast.org/welcome))*

Demos

# Demos

Replicate Toggle

Replicate *([What is Replicate?](https://replicate.com/docs/arxiv/about))*

Spaces Toggle

Hugging Face Spaces *([What is Spaces?](https://huggingface.co/docs/hub/spaces))*

Spaces Toggle

TXYZ.AI *([What is TXYZ.AI?](https://txyz.ai))*

Related Papers

# Recommenders and Search Tools

Link to Influence Flower

Influence Flower *([What are Influence Flowers?](https://influencemap.cmlab.dev/))*

Core recommender toggle

CORE Recommender *([What is CORE?](https://core.ac.uk/services/recommender))*

- Author
- Venue
- Institution
- Topic


About arXivLabs

# arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2610.08775) |
Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))

simonpure

Interpretation history

2026-10-08T01:35:11Z
grounded: converges/high — The BOTTLED benchmark independently validates several load-bearing Scott positions: (1) zero-shot evaluation is the wrong unit — it fails to predict distilled-a
2026-10-08T01:23:52Z
case created — New arXiv paper (submitted Oct 6, 2026) introducing a novel benchmark for agent distillation with concrete cost/performance tradeoff findings; no community discussion yet.

Decision trace

10-08 13:55attention_communicatedNew arXiv paper (submitted Oct 6) introduces BOTTLED, a benchmark where LLM agents must convert general capabilities into cheap task-specific artifacts ('bottling'). Across 10 models and 3 t
10-08 13:55attention_routeFresh arXiv submission (Oct 6) with empirical findings that directly validate several load-bearing positions: (1) zero-shot evaluation is the wrong unit — it fails to predict distilled-agent success;
10-08 12:40attention_routeFresh arXiv submission (Oct 6) with empirical findings that directly validate several load-bearing Scott positions: (1) zero-shot evaluation is the wrong unit — it fails to predict distilled-agent suc
10-08 12:36attention_candidatecreate
10-08 12:35groundThe BOTTLED benchmark independently validates several load-bearing Scott positions: (1) zero-shot evaluation is the wrong unit — it fails to predict distilled-agent success, directly confirming '
10-08 12:23createNew arXiv paper (submitted Oct 6, 2026) introducing a novel benchmark for agent distillation with concrete cost/performance tradeoff findings; no community discussion yet.