Retrieved article excerpt
Open article · Retrieved 2026-09-10T15:15:31.209433+00:00
As more and more organizations adopt AI agents to put an extra layer of intelligence into the capable hands of their teams, new questions arise: Do we stick to frontier-lab models? When does it make sense to invest in our own GPU infrastructure? Should we switch to the latest and greatest open-weights model? How do we get the most intelligence out of our agents? 01 Intro In our first blog post , we looked into the actual cost and performance of leading AI models deployed on a few reference hardware setups when used to solve real coding tasks. We kept things vanilla when it came to further optimizing these setups, but there are plenty of meaningful optimizations one can do across the stack. We promised a closer look at those, so here’s a first one that deserves an aistack deep-dive: the agent harness . “The harness can often be the distinguishing factor that makes one LLM work better than another.” S. Raschka · Ahead of AI · [1] A harness is the crucial layer that turns an LLM into a smart and capable AI agent. This is what you’d actually install and run, e.g., Claude Code, Codex, or OpenCode. While the model does the thinking, it’s the harness that decides what it thinks about and how, which tools it can reach for, and when to stop. Harness — what you install and run system prompt & context management tool orchestration prompt formatting · output parsing error recovery when to stop decides how and what the model thinks about Task repo · issue · tests Outcome task resolved or not prompts completions calls results Model Qwen3.8-27B (FP8) does the thinking Interfaces terminal · file editor test runner · web search Fig. 01 Where the harness sits Unlike models, which can be benchmarked across many flavors of tasks so you can pick the right one for your needs, harnesses are still mostly chosen on vibes, GitHub stars and X threads. Yet increasingly, your choice of harness seems to matter just as much, if not more than the model choice itself [ 1 ][ 2 ][ 3 ][ 8 ]. Thankfully, efforts to evaluate harnesses are on the rise [ 4 ]. The reported difference when swapping them can “often be the distinguishing factor that makes one LLM work better than another” [ 1 ]. Bold statement, so our interest was piqued and we started running the numbers to find out if that is really the case. 02 Our approach Coding agent harnesses operate by giving the model access to relevant tools (a terminal, file editors, test runners, web search, etc.) and handle context management, prompt formatting, error recovery and output parsing. In ‘ How many devs can you fit on a GPU? ’ we held the harness fixed and varied the hardware. This time we are swapping only the harness, comparing 3 of them across 2 models running on a suitable hardware setup. To figure out whether it’s worth swapping harnesses at all when deciding on your own AI stack, we’re looking at 3 things of note: The token efficiency of the 3 harnesses (this will drive cost, time and concurrency) The prefix cache hit rate (often overlooked, but a good measure of smart or bad context engineering and a big cost driver) And, of course, accuracy (token counts don’t matter if those tokens don’t resolve the task you’re trying to get done) For this deep dive, our aistack team picked two of the latest models at the time of writing: Qwen3.8-27B ( HuggingFace card ) in FP8 precision, a solid smaller model that fits on a single NVIDIA H200. It scores 52 on the Artificial Analysis Intelligence Index [ 6 ] and 51 on their Agentic Index, ahead of models several times its size. The freshly revealed GLM-5.3-Flash in FP8 (known previously as the mysteriously hyped “Ox alpha”) ( HuggingFace card ), running on 4 NVIDIA H200s. As always, realism is key for us. We fully simulate the real SWE development workflow: each developer in our evaluation framework is a separate harness worker with its own environment, resources and access to the web. We reused the same benchmark of 64 coding tasks we curated from SWE-Bench Pro for our first article. While there are many options to pick from, we compared 3 well-known coding harnesses: Claude Code , Codex , and Pi . The selection was based on popularity within agentic coding and what our developers currently use, but you need only look at GitHub to find many more. We excluded harnesses that aren’t coding-specific, but as promised, we’ll come back to other agentic tasks in future posts. We ran each harness with its default settings and kept the inference engine configuration identical across all three. We didn’t apply any customisation to the harnesses and evaluated them as-is. We did our best to get as close to comparing apples-to-apples as it can get, though in our next post you’ll see that our experiment took a slight detour when we zoomed in on what the agents were actually doing. (Yes, they were naughty.) More details on what we observed and how the team handled it will follow in a couple of days. For now, here’s what our harness comparison can tell you about how to optimize your stack. Let’s dig in. 03 Results Resolve rate: surprisingly not where the harness matters All three harnesses land at a 44–53% resolve rate for the coding tasks we threw at them across both models. Surprisingly, the harnesses we selected barely make a difference to accuracy in our tests, with a variation of about 2 to 3 tasks resolved across our runs (see the figure below). If you are picking a stack based on resolve rate alone, save yourself the benchmarking and analysis paralysis – at least for now. Go with what you prefer. Your mileage may vary with the long (long) tail of other harnesses out there, of course. (Let us know if there’s a standout one that’s bound to make a difference here.) 0% 25% 50% 75% 100% 48% 31/64 52% 33/64 Codex 52% 33/64 50% 32/64 Claude Code 44% 28/64 50% 32/64 Pi tasks resolved out of 64 Qwen3.8-27B · 1×H200 GLM-5.3-Flash · 4×H200 Fig. 02 Resolve rate per harness and model, on a 0–100% axis Tokens are where we start seeing the real difference When looking at token efficiency, both input and output tokens show variations worth addressing. While most harness/model combinations generate about 3–4M output tokens on our benchmark, Codex with GLM-5.3-Flash generates only around 2.1M output tokens for about the same accuracy. This is important because, compared to the relatively fast processing of input tokens, generating these output tokens is a considerable part of your total wall time. This means each session will hold its GPU slot longer, so the same hardware will resolve fewer tasks per hour (and thus burn pricey GPU hours, but we’ll get to that). 0M 1.25M 2.5M 3.75M 5M 3.33M 2.1M Codex 3.72M 3.97M Claude Code 3.72M 3.79M Pi output tokens generated per sweep Qwen3.8-27B · 1×H200 ($4.54/hr) GLM-5.3-Flash · 4×H200 ($18.16/hr) Fig. 03 Output tokens per 64-task sweep, per harness and model Input tokens are worth a look as well. On Qwen3.8-27B, Claude Code pushes around 445M input tokens through the model, whereas Codex pushes only 330.5M. That is 1.35× as much context to solve just one more task. On GLM-5.3-Flash, the gap is even bigger (480M vs 192M). Claude Code consistently creates a higher context volume, and in our benchmark, it does not translate into more solved tasks. Talking more doesn’t make you smarter, Claude. 0M 125M 250M 375M 500M 330.5M 192.1M Codex 445.8M 480.2M Claude Code 348.5M 366.4M Pi input tokens pushed through the model per sweep Qwen3.8-27B · 1×H200 ($4.54/hr) GLM-5.3-Flash · 4×H200 ($18.16/hr) Fig. 04 Input tokens per 64-task sweep, per harness and model The lesson here is that throwing more context at the model doesn’t mean it solves more tasks. It just makes each solve more expensive. Clever context engineering can have a significant impact on the cost. From your infrastructure’s perspective, that larger context means a bigger KV cache and therefore more pressure on your GPU’s available memory. That matters when you’re aiming at higher concurrency or higher token throughput. Harness Model Resolved Wall time Agent time Input tokens Output tokens tasks/h resolved/h Claude Code Qwen3.8-27B 33/64 (52%) 2.26 h 16.6 h 445.8 M 3.72 M 27.0 13.9 Codex Qwen3.8-27B 31/64 (48%) 2.07 h 14.5 h 330.5 M 3.33 M 31.1 15.0 Pi Qwen3.8-27B 28/64 (44%) 2.24 h 15.5 h 348.5 M 3.72 M 27.4 12.0 Claude Code GLM-5.3-Flash 32/64 (50%) 2.48 h 15.25 h 480.23 M 3.97 M 25.8 12.9 Codex GLM-5.3-Flash 33/64 (52%) 1.26 h 6.6 h 192.14 M 2.10 M 50.5 26.0 Pi GLM-5.3-Flash 32/64 (50%) 2.48 h 15.9 h 366.4 M 3.79 M 25.9 12.9 ← slide → tasks per hour 0 15 30 45 60 31.1 50.5 Codex 27 25.8 Claude Code 27.4 25.9 Pi resolved tasks per hour 0 7.5 15 22.5 30 15 26 Codex 13.9 12.9 Claude Code 12 12.9 Pi Qwen3.8-27B · 1×H200 GLM-5.3-Flash · 4×H200 Fig. 05 Throughput: tasks per hour and resolved tasks per hour, per harness and model KV cache utilization for Qwen3.8-27B with the three harnesses on 1×H200 (concurrency 8) metric Codex Pi Claude Code Δ Pi vs Codex Δ Claude Code vs Codex KV cache used, mean 22.2% 24.6% 32.4% +10.8% +46.3% KV cache used, p50 22.2% 24.1% 32.6% +8.2% +46.7% KV cache used, p90 34.9% 40.5% 49.9% +16.0% +42.9% KV cache used, max 59.8% 72.1% 78.8% +20.5% +31.6% share of run at ≥90% KV 0.0% 0.0% 0.0% 0.0% 0.0% share of run at ≥98% KV 0.0% 0.0% 0.0% 0.0% 0.0% ← slide → Utilisation as a share of the KV-cache pool; the Δ columns are relative to Codex. 0% 25% 50% 75% 100% 22.2% 34.9% 59.8% Codex 24.6% 40.5% 72.1% Pi 32.4% 49.9% 78.8% Claude Code share of the KV-cache pool in use, Qwen3.8-27B on 1×H200 (concurrency 8) mean p90 max Fig. 06 KV cache in use on Qwen3.8-27B: mean, p90 and peak per harness According to our measurements, with the same inference engine configuration, Codex puts less pressure on the KV cache: peak utilization reaches ~60% of the total KV cache pool, which is noticeably less than Pi (72%) or Claude Code (79%). So while resolve rate stays roughly constant, your choice of harness has a significant effect on the token efficiency of your stack, which matters if you care about total wall time and developer experience. Read on to discuss what that means for your bill. If you need a refresher on input & output tokens and the intricacies of KV cache, see our earlier article “What is your GPU waiting for?” . That wall time is what feeds your invoice When you rent or buy your infrastructure, the GPUs stay running whether the model is reasoning or tool calls are being executed. So that total wall time ends up costing you money either because you’re paying the GPU rent by the hour or because the TCO of those GPUs you bought is heavily dependent on how many useful tasks they resolve for you. On Qwen3.8-27B (1×H200 at $4.54/hr on Modal ), Codex wraps the full 64-task sweep for about $9. Using the Claude Code harness instead costs $10.26. Pi costs $10.16. No major differences, unless we’re talking scale here. But look at GLM 5.3 Flash on those 4×H200s ($18.16/hr on Modal) and the differences get uncomfortable pretty fast. Using Codex you’d pay $22.8, but using Claude Code as a harness that increases to $45. Pi: $45 as well. That’s nearly double as costly for a very similar intelligence, just based on switching your harness . All still considerably cheaper than paying for API pricing of course. Remember from our earlier post that the same workload would cost you around $98 based on API token prices via Anthropic. $0 $12.5 $25 $37.5 $50 $9 $22.8 Codex $10.26 $45 Claude Code $10.16 $45 Pi GPU cost per 64-task sweep (wall time × hourly rate) Qwen3.8-27B · 1×H200 ($4.54/hr) GLM-5.3-Flash · 4×H200 ($18.16/hr) Fig. 07 GPU cost per 64-task sweep, per harness and model So yes, picking the right harness and model mix can have a considerable impact on your total cost, whether you’re renting or buying. Anomalies can get costly Throughout our tests, we bumped into two anomalies that ended up impacting the performance and thus the cost of our runs significantly. Doomlooping – when agents get stuck First up, we noticed a strange recurring anoma