Infernix update adds Qwen-focused RTX 5090 inference tools

state: watchingheat: mediumuncertainty: mediumrelevance: mediumlocal-inference qwen inference-engineWallawalla47

What is this?

Infernix is a C++/CUDA inference engine by GitHub user Wallawalla47, developed from a fork of NInfer, for running Qwen models on a single NVIDIA RTX 5090. The November 1, 2026 repository release and November 2 Reddit post describe native Windows and Linux support, model downloads and serving tools, speculative decoding, and support for recent Qwen models. Its author publishes detailed comparisons against other inference engines, but those results are self-reported; the repository says measured quality comparisons for some configurations are still forthcoming. A Basalt maintainer disputes parts of the release’s characterization of Basalt’s defaults and says the reported 5–8× first-token gap differs from their own measurements. The supplied sources do not independently validate either side’s comparisons.

Why it matters

This is a developer-built tool rather than a new foundation model, but it illustrates how specialized inference software can make large, long-context models more practical on a single high-end consumer GPU. If the author’s results hold, faster prompt processing and a simpler launcher could expand local use for coding agents and other workloads, while keeping inference under the user’s control. The hardware requirement is substantial, and the performance advantage—especially against Basalt—is contested and not independently established.

Current assessment

New: Wallawalla47 announced an Infernix update, with the repository documenting a Windows release dated November 1, 2026 and the Reddit post dated November 2. The engine targets Qwen models on one RTX 5090 and supports Windows and Linux. The update adds model artifacts, DFlash2 speculative decoding and automatic configuration, a beta web launcher for downloading/configuring models and monitoring use, and a lossless mode intended as a reference for quality testing. These are developer-described capabilities and availability, not independently tested findings. The repository’s benchmarks use a single RTX 5090 and compare selected workloads and versions; the author reports faster time to first token and replay completion in some Qwen3.8-Flash-Next NVFP4 comparisons, while Basalt is faster on some decode measures and finishes the IQ3_S replay sooner. Results depend on hardware, settings, precision choices, caching, and model artifact; the author notes some compared engines have since changed and quality measurements for certain comparisons are pending. Basalt’s maintainer disputes the account of its default behavior and says their own measurement did not reproduce the reported 5–8× first-token gap, while agreeing Infernix is a solid option. No independent benchmark or adoption evidence is supplied. Track reproducible comparisons across current versions and quality testing before treating the performance claims as settled. If validated, the combination of host-memory/SSD offloading, speculative decoding, and a GUI could lower software friction for running capable models locally, but it remains aimed at owners of costly RTX 5090-class hardware.

Measured heat

now 0 pts/hpeak 11 pts/hcomments 0/hpeers p0momentum: steady2 platformsage 164h

points/hour across evidence · reading as of 2026-11-08 21:23:01.818014+11:00 · deterministic, not a model opinion

How the heat travelled

11-01 15:30 (minted)	⭐ origin echo-reconstructed	The linked repository is the primary artifact announcing and documenting the release. Its README describes Infernix as “a C++/CUDA inference Wallawalla47 (GitHub user) on github (echo) · attributed from reddit.post.1xl4pf6 · published time unknown	—
11-01 14:40	first on r/LocalLLaMA · published · lag ?	Infernix - New Release! Suspicious-Mark-5450	—
11-01 14:40	amplified on r/LocalLLaMA 👑	reddit.post.1xl4pf6 Suspicious-Mark-5450	peak 22 · 15 comments · 100% of case engagement
11-01 15:24	our radar first saw it · lag ?	discovery anchor: reddit.post.1xl4pf6	—

pace: p64 vs 1821 stories at the 96h mark (now 164h old) — ahead of camthink-small-model-agent-task-evaluation (1.0x), behind apollo-office-multi-agent-github-automation (1.0x)

Evidence (2) — ⭐ canonical anchor

source	object	author	score	comments
🟠 reddit	Infernix - New Release! LocalLLaMA Retrieved article excerpt Open article · Retrieved 2026-11-01T15:27:34.661415+00:00 # Infernix > AI disclaimer: everything added since the fork from NInfer, including most of this README, was > written with AI (mostly Claude Opus 5.5 and Qwen3.8-27B running on Infernix). It is likely to be > neither complete nor entirely accurate. This is hobby development. Infernix is a C++/CUDA inference engine that serves Qwen models on one NVIDIA RTX 5090. It began as a fork of [NInfer](https://github.com/Neroued/ninfer) by [Neroued](https://github.com/Neroued) and reads NInfer's `.ninfer` artifacts as well as its own `.infernix` files. ## Headline results One RTX 5090 on PCIe Gen5 x8, 2026-10-30/31. Every engine ran its latest version at the time, natively on Windows, with its own recommended settings and drafter and the same 84 GiB RAM budget. Context: one request, ~8K to ~250K tokens. Replay: an agentic coding workload, two requests at a time. Full tables and methods: [Benchmarks](https://github.com/Wallawalla47/Infernix#benchmarks). Speed without precision shortcuts. Infernix keeps every activation at the precision the checkpoint specifies (16 bits where the others round to 8), runs every routed expert (no expert tail skipping) and computes the full output head (no 2-bit screen). Basalt, which computes with fewer bits (6-bit dense layers on NVFP4; on IQ3\_S 8-bit activations, an 8-bit copy of the hyper-connection matrices and a 2-bit head screen), has the highest decode tok/s at Infernix's defaults. On NVIDIA's NVFP4 weights Infernix still reaches the first token 5-8× sooner and finishes the agentic workload 36 % sooner, and with its experts at 8 bits as Basalt runs them it matches or passes Basalt's decode too. On the IQ3\_S GGUF Basalt is ahead overall (4.4 against 5.5 min); with the same 8-bit activations Infernix matches its first tokens, but Basalt still decodes faster. Measured quality for every engine follows in a later update. \| Infernix vs \| First token \| Decode \| Replay: mean first token \| Replay: wall time \| \| --- \| --- \| --- \| --- \| --- \| \| Qwen3.8-Flash-Next, NVIDIA's NVFP4 weights \| \| \| \| \| \| [Strata](https://github.com/Niko1221/Strata) v0.2.8 \| 1.4-1.6× sooner \| +10-15 % to 128K, −4 % at 250K \| 3.1 s vs 12.3 s \| 5.7 vs 12.9 min \| \| [strata-nvfp4](https://github.com/sergqwer/strata-nvfp4) 0.2.0-beta-nvfp4.3 \| 1.3× sooner \| level to +3 % \| 3.1 s vs 10.8 s \| 5.7 vs 11.7 min \| \| [Basalt](https://github.com/jesdga95/basalt) `c08284cb` \| 5-8× sooner¹ \| Basalt 7-23 % faster² \| 3.1 s vs 7.2 s \| 5.7 vs 8.9 min \| \| Qwen3.8-Flash-Next, ISTA-DASLab's IQ3\_S GGUF \| \| \| \| \| \| Strata v0.2.8³ \| 15 % sooner at 8K, level from 128K \| +3-12 % \| 3.1 s vs 6.9 s \| 5.5 vs 8.1 min \| \| Basalt `c08284cb` \| Basalt 4 % sooner at 8K, 1.3× from 128K⁴ \| Basalt 1.4-1.7× faster⁴ \| 3.1 s vs 3.7 s \| Basalt sooner: 4.4 vs 5.5 min⁴ \| \| Qwen3.8-27B, the same NVFP4 artifact in both \| \| \| \| \| \| [NInfer](https://github.com/Neroued/ninfer) `1882c6adb` \| 9 % sooner at 8K, 1.6-1.8× from 128K \| +11-17 % (own drafters), +2-3 % without \| 1.9 s vs 2.5 s \| 12.7 vs 16.4 min \| 1. Basalt reads its 50.7 GiB n-gram table from the SSD here: it does not fit in the RAM budget beside its experts. 2. Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. With the experts at 8 bits as Basalt runs them in decode (`--a8-activations`), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner. 3. Strata's default route tail skip (it skips some missed experts) was off: it changes which experts run. 4. All three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (`--a8-activations`) Infernix reaches the first token as soon as Basalt, which still decodes 1.2-1.7× as fast. ### Activation precision Infernix computes each model at the activation precision its checkpoint specifies; options change it: \| Weights \| Default \| Options \| \| --- \| --- \| --- \| \| NVIDIA NVFP4 Flash-Next (Dense8 and the bit-exact conversion) \| Routed experts 4-bit (W4A4 with NVIDIA's calibrated scales), because NVIDIA's checkpoint specifies it; everything else 16-bit \| `--a8-activations`: experts 8-bit. `--a16-activations`: experts 16-bit on the stored 4-bit weights \| \| NVIDIA FP4-Local Flash-Next \| 16-bit, as NVIDIA serves it \| `--expert-a4-activations`: experts 4-bit. `--a8-activations`: experts 8-bit \| \| Uncensored NVFP4 Flash-Next \| 16-bit (weight-only NVFP4 experts) \| `--a8-activations`: experts 8-bit \| \| IQ3\_S GGUF Flash-Next \| 16-bit everywhere \| `--a8-activations`: dense projections and experts 8-bit, as Strata and Basalt \| \| Qwen3.8-27B NVFP4 \| 4-bit at its NVFP4 layers, as the artifact specifies (NInfer the same) \| none \| On NVIDIA's NVFP4 weights Strata, strata-nvfp4 and Basalt run the experts at 8 bits in decode and every other activation at 8 bits. The options on Dense8 (RTX 5090, one request): \| Experts' activations \| First token at ~8K / ~128K / ~250K \| Decode rounds a second \| Agentic replay \| \| --- \| --- \| --- \| --- \| \| 4-bit (default) \| 1.9 / 12.2 / 23.9 s \| \| 5.7 min \| \| 8-bit, `--a8-activations` \| level / 28 % / 28 % later \| 1-7 % more \| decode 6 % faster, uncached prompts 16 % slower: 5.7 min \| \| 16-bit, `--a16-activations` \| level / 38 % / 37 % later \| 2-6 % more \| not run \| Only the 4-bit route reads prompts on the RTX 5090's FP4 tensor cores, which makes it the fastest for agents. Both options change the output; their quality is not measured yet. ## Why Infernix ### Qwen3.8-Flash-Next on one RTX 5090 The 24,576 routed experts (63 GiB at NVFP4) live in pinned RAM, the hot ones cached in VRAM; each layer's misses are split between the CPU and the PCIe link, and the 52 GB n-gram table is read from NVMe. - Soonest first token on NVIDIA's NVFP4 weights: 1.3-1.6× sooner than Strata and strata-nvfp4 from 8K to 250K tokens; 36-56 % less wall time than every other engine in an agentic replay ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-flash-next-nvfp4-infernix-vs-strata-strata-nvfp4-and-basalt)). - Activations as the checkpoint specifies: 16-bit wherever it keeps BF16 (all of them on GGUF), where Strata and Basalt use 8 bits ([activation precision](https://github.com/Wallawalla47/Infernix#activation-precision)). - DFlash2 drafting with Inco AI's drafter ([Qwen3.8-Flash-Next-DFlash2-Infernix](https://huggingface.co/Wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix), 776 MB, any conversion): 8-11 % faster than MTP, 26 % on sampled code ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-flash-next-dflash2-against-mtp)). - Long context: 262K tokens natively, 1M with YaRN; a 250K prompt prefills in about 24 s. - Less RAM than the model? An SSD expert tier reads what does not fit. - Uncensored too: [Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix](https://huggingface.co/Wallawalla47/Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix) keeps orcarouter's abliterated NVFP4 experts and FP8 projections bit-exact (perplexity 4.772). Setup and tuning: the [Qwen3.8-Flash-Next guide](https://github.com/Wallawalla47/Infernix/blob/main/docs/qwen3_8-flash-next.md). ### Qwen3.8-27B: faster prefill and decode Against NInfer on the same artifact ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-27b-infernix-vs-ninfer)): - Hybrid prefix cache for Qwen's linear-attention layers (64-token KV blocks, recurrent-state snapshots, GPU, RAM and file tiers): in an agentic replay the median first token comes 36 % sooner and the work finishes 21 % sooner. - Faster prefill: 1.6-1.8× NInfer's on long prompts. - Faster decode: DFlash2 with n-gram copy drafts, 11-17 % ahead of NInfer's MTP; draft trees (`--draft-tree-nodes`) are an option on top. - Smaller KV: `k4v2` and `vq2` hold 2.7 and 4 times the context of INT8. ### Built for agents - APIs: OpenAI Chat Completions and Responses, Anthropic Messages and streaming, with the options Claude Code, Qwen Code, Codex, Zed and GitHub Copilot send. - Tool calling: the XML forms agents emit, recovery of broken calls (`--tolerant-tool-calls`), and deferred tools (`defer_loading`, regex/BM25 tool search, `tool_reference`) that keep large MCP catalogs out of the prompt. - Constrained decoding: JSON object and Schema, strict tool arguments, GBNF, regex and choice grammars, with every model and drafter. - Thinking control: budgets with a wrap-up message, effort levels and a repetition guard (`--reasoning-loop`). - Concurrency and resilience: one to eight requests with preemption under memory pressure; the prefix cache persists across restarts; a failed Flash-Next expert read fails only the requests running then. - `/v1/decide`: decisions from token probabilities (yes/no, choice, score, number, scalar, point and box). ### And also - Speculative decoding: MTP, DFlash, DFlash2 and n-gram drafting with exact sampling, on by default with the drafter measured fastest for each model (`--spec auto`). - KV formats: `int8`, `bf16`, `fp8`, `nvfp4`, `k8v4`, `vq2` and `k4v2`. - Vision: images and video; the vision encoder can stay in system RAM. - Conversion: Hugging Face, NVIDIA ModelOpt, Quasar and GGUF sources; Q4-Q8, FP8, NVFP4 and mixed recipes. - Windows and Linux: native MSVC and GCC builds, WSL2 included. ## Benchmarks RTX 5090 (32 GB) on PCIe Gen5 x8, Core i9-13900K, 96 GB DDR5, Windows 11; Gen5 x16 reads Flash-Next's prompts up to twice as fast ([measured](https://github.com/Wallawalla47/Infernix#pcie-gen5-x8-against-x16)). TTFT: time to the first token; bold: best in its row. Methods and every engine's settings are at the end of each section and [below](https://github.com/Wallawalla47/Infernix#how-the-benchmarks-were-run). ### Qwen3.8-Flash-Next NVFP4: Infernix vs Strata, strata-nvfp4 and Basalt Infernix runs the [Dense8 conversion](https://github.com/Wallawalla47/Infernix#models) of NVIDIA's checkpoint with Inco AI's DFlash2 drafter; the others run NVIDIA's checkpoint as their own NVFP4 packs with the model's MTP drafter. Context length (one request): \| Context \| TTFT: Infernix \| Strata \| strata-nvfp4 \| Basalt \| Decode tok/s: Infernix \| Strata \| strata-nvfp4 \| Basalt \| \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| \| ~8K \| 1.88 s \| 2.67 s \| 2.48 s \| 9.55 s \| 183 \| 159 \| 177 \| 202 \| \| ~128K \| 12.22 s \| 18.83 s \| 15.34 s \| 92.94 s \| 161 \| 146 \| 161 \| 173 \| \| ~250K \| 23.90 s \| 37.55 s \| 31.78 s \| 125.92 s \| 175 \| 183 \| 172 \| 216 \| Agentic replay (two requests at a time, three seeds): \| Metric \| Strata \| strata-nvfp4 \| Basalt \| Infernix \| Infernix 8-bit experts† \| Infernix vs Strata \| vs strata-nvfp4 \| vs Basalt \| \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| \| Mean TTFT \| 12.3 s (10.8-13.9) \| 10.8 s (9.2-12.0) \| 7.2 s (6.4-7.9) \| 3.1 s (2.9-3.5) \| 3.4 s (3.3-3.6) \| −74 % \| −71 % \| −56 % \| \| Median TTFT \| 9.92 s \| 7.62 s \| 3.81 s \| 2.04 s \| 2.00 s \| −79 % \| −73 % \| −46 % \| \| 90th-percentile TTFT \| 25.9 s \| 24.7 s \| 17.5 s \| 7.1 s \| 8.6 s \| −71 % \| −71 % \| −59 % \| \| Mean TTFT, new long prompts \| 11.8 s \| 10.2 s \| 20.8 s \| 6.1 s \| 7.4 s \| −48 % \| −38 % \| −70 % \| \| Prompt tokens from cache \| 46.1 % \| 53.2 % \| 83.1 % \| 84.5 % \| 84.5 % \| +38.4 pt \| +31.2 pt \| +1.4 pt \| \| Prefill tok/s, no cache hit \| 5,170 \| 5,847 \| 2,349 \| 7,480-7,689 \| 6,216-6,281 \| +45 % \| +30 % \| 3.3× \| \| Output tok/s, one request \| 116 (114-120) \| 110 (108-113) \| 166 (160-170) \| 166	Suspicious-Mark-5450	22	15
🟧 echo.github ⭐	The linked repository is the primary artifact announcing and documenting the release. Its README describes Infernix as “a C++/CUDA inference	Wallawalla47 (GitHub user)	—	—

source

object

author

score

comments

🟠 reddit

Infernix - New Release!

LocalLLaMA

Retrieved article excerpt

Open article · Retrieved 2026-11-01T15:27:34.661415+00:00

# Infernix

> **AI disclaimer:** everything added since the fork from NInfer, including most of this README, was
> written with AI (mostly Claude Opus 5.5 and Qwen3.8-27B running on Infernix). It is likely to be
> neither complete nor entirely accurate. This is hobby development.

Infernix is a C++/CUDA inference engine that serves Qwen models on one NVIDIA RTX 5090. It began as a fork of [NInfer](https://github.com/Neroued/ninfer) by
[Neroued](https://github.com/Neroued) and reads NInfer's `.ninfer` artifacts as well as its own
`.infernix` files.

## Headline results

One RTX 5090 on PCIe Gen5 x8, 2026-10-30/31. Every engine ran its latest version at the time,
natively on Windows, with its own recommended settings and drafter and the same 84 GiB RAM budget.
**Context**: one request, ~8K to ~250K tokens. **Replay**: an agentic coding workload, two requests
at a time. Full tables and methods: [Benchmarks](https://github.com/Wallawalla47/Infernix#benchmarks).

**Speed without precision shortcuts.** Infernix keeps every activation at the precision the
checkpoint specifies (16 bits where the others round to 8), runs every routed expert (no expert tail
skipping) and computes the full output head (no 2-bit screen). Basalt, which computes with fewer
bits (6-bit dense layers on NVFP4; on IQ3\_S 8-bit activations, an 8-bit copy of the hyper-connection
matrices and a 2-bit head screen), has the highest decode tok/s at Infernix's defaults. On NVIDIA's
NVFP4 weights Infernix still reaches the first token 5-8× sooner and finishes the agentic workload
36 % sooner, and with its experts at 8 bits as Basalt runs them it matches or passes Basalt's decode
too. On the IQ3\_S GGUF Basalt is ahead overall (4.4 against 5.5 min); with the same 8-bit
activations Infernix matches its first tokens, but Basalt still decodes faster. Measured quality for
every engine follows in a later update.

| Infernix vs | First token | Decode | Replay: mean first token | Replay: wall time |
| --- | --- | --- | --- | --- |
| **Qwen3.8-Flash-Next, NVIDIA's NVFP4 weights** |  |  |  |  |
| [Strata](https://github.com/Niko1221/Strata) v0.2.8 | 1.4-1.6× sooner | +10-15 % to 128K, −4 % at 250K | 3.1 s vs 12.3 s | 5.7 vs 12.9 min |
| [strata-nvfp4](https://github.com/sergqwer/strata-nvfp4) 0.2.0-beta-nvfp4.3 | 1.3× sooner | level to +3 % | 3.1 s vs 10.8 s | 5.7 vs 11.7 min |
| [Basalt](https://github.com/jesdga95/basalt) `c08284cb` | 5-8× sooner¹ | Basalt 7-23 % faster² | 3.1 s vs 7.2 s | 5.7 vs 8.9 min |
| **Qwen3.8-Flash-Next, ISTA-DASLab's IQ3\_S GGUF** |  |  |  |  |
| Strata v0.2.8³ | 15 % sooner at 8K, level from 128K | +3-12 % | 3.1 s vs 6.9 s | 5.5 vs 8.1 min |
| Basalt `c08284cb` | Basalt 4 % sooner at 8K, 1.3× from 128K⁴ | Basalt 1.4-1.7× faster⁴ | 3.1 s vs 3.7 s | Basalt sooner: 4.4 vs 5.5 min⁴ |
| **Qwen3.8-27B, the same NVFP4 artifact in both** |  |  |  |  |
| [NInfer](https://github.com/Neroued/ninfer) `1882c6adb` | 9 % sooner at 8K, 1.6-1.8× from 128K | +11-17 % (own drafters), +2-3 % without | 1.9 s vs 2.5 s | 12.7 vs 16.4 min |

1. Basalt reads its 50.7 GiB n-gram table from the SSD here: it does not fit in the RAM budget beside
   its experts.
2. Basalt runs NVIDIA's BF16 dense layers at **6 bits** (+10 % KL divergence by its own measure) and
   every activation outside the experts at **8 bits**; Infernix keeps 16. With the experts at 8 bits
   as Basalt runs them in decode (`--a8-activations`), Infernix decodes 199 / 185 / 268 tok/s
   against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the
   replay, and still reaches the first token 4-6× sooner.
3. Strata's default route tail skip (it skips some missed experts) was off: it changes which experts
   run.
4. All three hold the GGUF's weights as published, but Strata and Basalt compute **every activation
   at 8 bits**, and Basalt's decode reads the BF16 hyper-connection matrices from an **8-bit copy**
   and takes the logits outside the top 64 from a **2-bit copy** of the output head; Infernix keeps
   16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (`--a8-activations`) Infernix
   reaches the first token as soon as Basalt, which still decodes 1.2-1.7× as fast.

### Activation precision

Infernix computes each model at the activation precision its checkpoint specifies; options change
it:

| Weights | Default | Options |
| --- | --- | --- |
| NVIDIA NVFP4 Flash-Next (Dense8 and the bit-exact conversion) | Routed experts **4-bit** (W4A4 with NVIDIA's calibrated scales), because NVIDIA's checkpoint specifies it; everything else 16-bit | `--a8-activations`: experts 8-bit. `--a16-activations`: experts 16-bit on the stored 4-bit weights |
| NVIDIA FP4-Local Flash-Next | 16-bit, as NVIDIA serves it | `--expert-a4-activations`: experts 4-bit. `--a8-activations`: experts 8-bit |
| Uncensored NVFP4 Flash-Next | 16-bit (weight-only NVFP4 experts) | `--a8-activations`: experts 8-bit |
| IQ3\_S GGUF Flash-Next | 16-bit everywhere | `--a8-activations`: dense projections and experts 8-bit, as Strata and Basalt |
| Qwen3.8-27B NVFP4 | 4-bit at its NVFP4 layers, as the artifact specifies (NInfer the same) | none |

On NVIDIA's NVFP4 weights Strata, strata-nvfp4 and Basalt run the experts at 8 bits in decode and
every other activation at 8 bits. The options on Dense8 (RTX 5090, one request):

| Experts' activations | First token at ~8K / ~128K / ~250K | Decode rounds a second | Agentic replay |
| --- | --- | --- | --- |
| 4-bit (default) | 1.9 / 12.2 / 23.9 s |  | 5.7 min |
| 8-bit, `--a8-activations` | level / 28 % / 28 % later | 1-7 % more | decode 6 % faster, uncached prompts 16 % slower: 5.7 min |
| 16-bit, `--a16-activations` | level / 38 % / 37 % later | 2-6 % more | not run |

Only the 4-bit route reads prompts on the RTX 5090's FP4 tensor cores, which makes it the fastest
for agents. Both options change the output; their quality is not measured yet.

## Why Infernix

### Qwen3.8-Flash-Next on one RTX 5090

The 24,576 routed experts (63 GiB at NVFP4) live in pinned RAM, the hot ones cached in VRAM; each
layer's misses are split between the CPU and the PCIe link, and the 52 GB n-gram table is read from
NVMe.

- **Soonest first token** on NVIDIA's NVFP4 weights: 1.3-1.6× sooner than Strata and strata-nvfp4
  from 8K to 250K tokens; 36-56 % less wall time than every other engine in an agentic replay
  ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-flash-next-nvfp4-infernix-vs-strata-strata-nvfp4-and-basalt)).
- **Activations as the checkpoint specifies**: 16-bit wherever it keeps BF16 (all of them on GGUF),
  where Strata and Basalt use 8 bits ([activation precision](https://github.com/Wallawalla47/Infernix#activation-precision)).
- **DFlash2 drafting** with Inco AI's drafter
  ([Qwen3.8-Flash-Next-DFlash2-Infernix](https://huggingface.co/Wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix),
  776 MB, any conversion): 8-11 % faster than MTP, 26 % on sampled code
  ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-flash-next-dflash2-against-mtp)).
- **Long context**: 262K tokens natively, 1M with YaRN; a 250K prompt prefills in about 24 s.
- **Less RAM than the model?** An SSD expert tier reads what does not fit.
- **Uncensored too**:
  [Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix](https://huggingface.co/Wallawalla47/Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix)
  keeps orcarouter's abliterated NVFP4 experts and FP8 projections bit-exact (perplexity 4.772).

Setup and tuning: the [Qwen3.8-Flash-Next guide](https://github.com/Wallawalla47/Infernix/blob/main/docs/qwen3_8-flash-next.md).

### Qwen3.8-27B: faster prefill and decode

Against NInfer on the same artifact ([benchmarks](https://github.com/Wallawalla47/Infernix#qwen38-27b-infernix-vs-ninfer)):

- **Hybrid prefix cache** for Qwen's linear-attention layers (64-token KV blocks, recurrent-state
  snapshots, GPU, RAM and file tiers): in an agentic replay the median first token comes 36 %
  sooner and the work finishes 21 % sooner.
- **Faster prefill**: 1.6-1.8× NInfer's on long prompts.
- **Faster decode**: DFlash2 with n-gram copy drafts, 11-17 % ahead of NInfer's MTP; draft trees
  (`--draft-tree-nodes`) are an option on top.
- **Smaller KV**: `k4v2` and `vq2` hold 2.7 and 4 times the context of INT8.

### Built for agents

- **APIs**: OpenAI Chat Completions and Responses, Anthropic Messages and streaming, with the options
  Claude Code, Qwen Code, Codex, Zed and GitHub Copilot send.
- **Tool calling**: the XML forms agents emit, recovery of broken calls (`--tolerant-tool-calls`),
  and deferred tools (`defer_loading`, regex/BM25 tool search, `tool_reference`) that keep large MCP
  catalogs out of the prompt.
- **Constrained decoding**: JSON object and Schema, strict tool arguments, GBNF, regex and choice
  grammars, with every model and drafter.
- **Thinking control**: budgets with a wrap-up message, effort levels and a repetition guard
  (`--reasoning-loop`).
- **Concurrency and resilience**: one to eight requests with preemption under memory pressure; the
  prefix cache persists across restarts; a failed Flash-Next expert read fails only the requests
  running then.
- **`/v1/decide`**: decisions from token probabilities (yes/no, choice, score, number, scalar, point
  and box).

### And also

- **Speculative decoding**: MTP, DFlash, DFlash2 and n-gram drafting with exact sampling, on by
  default with the drafter measured fastest for each model (`--spec auto`).
- **KV formats**: `int8`, `bf16`, `fp8`, `nvfp4`, `k8v4`, `vq2` and `k4v2`.
- **Vision**: images and video; the vision encoder can stay in system RAM.
- **Conversion**: Hugging Face, NVIDIA ModelOpt, Quasar and GGUF sources; Q4-Q8, FP8, NVFP4 and
  mixed recipes.
- **Windows and Linux**: native MSVC and GCC builds, WSL2 included.

## Benchmarks

RTX 5090 (32 GB) on **PCIe Gen5 x8**, Core i9-13900K, 96 GB DDR5, Windows 11; Gen5 x16 reads
Flash-Next's prompts up to twice as fast ([measured](https://github.com/Wallawalla47/Infernix#pcie-gen5-x8-against-x16)). TTFT: time to the
first token; bold: best in its row. Methods and every engine's settings are at the end of each
section and [below](https://github.com/Wallawalla47/Infernix#how-the-benchmarks-were-run).

### Qwen3.8-Flash-Next NVFP4: Infernix vs Strata, strata-nvfp4 and Basalt

Infernix runs the [Dense8 conversion](https://github.com/Wallawalla47/Infernix#models) of NVIDIA's checkpoint with Inco AI's DFlash2 drafter;
the others run NVIDIA's checkpoint as their own NVFP4 packs with the model's MTP drafter.

**Context length** (one request):

| Context | TTFT: Infernix | Strata | strata-nvfp4 | Basalt | Decode tok/s: Infernix | Strata | strata-nvfp4 | Basalt |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ~8K | **1.88 s** | 2.67 s | 2.48 s | 9.55 s | 183 | 159 | 177 | **202** |
| ~128K | **12.22 s** | 18.83 s | 15.34 s | 92.94 s | 161 | 146 | 161 | **173** |
| ~250K | **23.90 s** | 37.55 s | 31.78 s | 125.92 s | 175 | 183 | 172 | **216** |

**Agentic replay** (two requests at a time, three seeds):

| Metric | Strata | strata-nvfp4 | Basalt | Infernix | Infernix 8-bit experts† | Infernix vs Strata | vs strata-nvfp4 | vs Basalt |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mean TTFT | 12.3 s (10.8-13.9) | 10.8 s (9.2-12.0) | 7.2 s (6.4-7.9) | **3.1 s** (2.9-3.5) | 3.4 s (3.3-3.6) | −74 % | −71 % | −56 % |
| Median TTFT | 9.92 s | 7.62 s | 3.81 s | **2.04 s** | 2.00 s | −79 % | −73 % | −46 % |
| 90th-percentile TTFT | 25.9 s | 24.7 s | 17.5 s | **7.1 s** | 8.6 s | −71 % | −71 % | −59 % |
| Mean TTFT, new long prompts | 11.8 s | 10.2 s | 20.8 s | **6.1 s** | 7.4 s | −48 % | −38 % | −70 % |
| Prompt tokens from cache | 46.1 % | 53.2 % | 83.1 % | **84.5 %** | 84.5 % | +38.4 pt | +31.2 pt | +1.4 pt |
| Prefill tok/s, no cache hit | 5,170 | 5,847 | 2,349 | **7,480-7,689** | 6,216-6,281 | +45 % | +30 % | 3.3× |
| Output tok/s, one request | 116 (114-120) | 110 (108-113) | **166** (160-170) | **166**

Suspicious-Mark-5450

🟧 echo.github ⭐

The linked repository is the primary artifact announcing and documenting the release. Its README describes Infernix as “a C++/CUDA inference

Wallawalla47 (GitHub user)

—

Interpretation history

2026-11-05T09:23:28Z
audience grounding: medium — This is a developer-built tool rather than a new foundation model, but it illustrates how specialized inference software can make large, long-context models more practical on a single high-end consumer GPU. If the author’s results hold, faster prompt processing and a simpler launcher could expand local use for coding agents and other workloads, while keeping inference under the user’s control. The hardware requirement is substantial, and the performance advantage—especially against Basalt—is contested and not independently established.
2026-11-05T09:23:10Z
A Basalt maintainer disputes a material premise in Infernix’s comparison: the cited expert-skipping and related optimizations are opt-in, not enabled by default. A user’s limited comparison also reports faster Basalt decoding and an Infernix improvement after a team-provided pin, reinforcing that results depend on configuration rather than establishing a general performance ranking.
2026-11-01T15:28:23Z
audience grounding: medium — This is a useful example of how specialized inference software can make large, long-context models more practical on a single consumer GPU, including by placing model experts in system RAM or on SSD and using speculative decoding. The beta launcher and Windows support may lower setup barriers for local experimentation and agent workflows. The results are promising but narrow: they come from the developer’s own tests on one RTX 5090, and some speed comparisons trade off arithmetic precision or model quality. It is a meaningful community tool and engineering effort, not yet evidence of broad adoption or a general shift in inference economics.

Decision trace

11-05 20:23attention_candidatematerial_reprice
11-05 20:23repriceA Basalt maintainer disputes a material premise in Infernix’s comparison: the cited expert-skipping and related optimizations are opt-in, not enabled by default. A user’s limited comparison also repor
11-05 20:23groundThis is a developer-built tool rather than a new foundation model, but it illustrates how specialized inference software can make large, long-context models more practical on a single high-end consume
11-05 20:22review_screenThe Basalt maintainer directly contradicts the assessment’s characterization of expert tail skipping and related optimizations as enabled by default, saying they are opt-in and warned at startup. A us
11-05 20:22review_screenjev screen borderline (noul=0.58) — luna review
11-02 08:21sensor_dirtycomment_update
11-02 03:25sensor_dirtycomment_update
11-02 02:30attention_candidatecreate
11-02 02:30promote_anchororigin walk conf 0.83
11-02 02:28groundThis is a useful example of how specialized inference software can make large, long-context models more practical on a single consumer GPU, including by placing model experts in system RAM or on SSD a
11-02 02:28createThe released engine and downloadable builds are a concrete local-inference option, while its extensive performance comparisons come from the project itself.