ggeorgovassilis releases llm-gauze, an OpenAI-compatible HTTP gateway that detects and remediates open-weight LLM quirks (malformed tags, empty responses, stuck loops, context overflows) before clients see them, with logging, metrics, and Docker deployment.

Open article · Retrieved 2026-10-08T17:49:33.819540+00:00

# llm-gauze

[CI](https://github.com/ggeorgovassilis/llm-gauze/actions/workflows/ci.yml)

llm-gauze is an HTTP gateway that sits in front of a local LLM (served via an
OpenAI-compatible API) and works around its shortcomings: transient errors
without retries, silently hung or looping models, empty or sloppy responses,
context-window overflows, runaway reasoning, and malformed tool calls. It
logs every exchange and remediates what it can before the client ever sees
it.

## Quick start

llm-gauze ships as a published container image and runs with Docker Compose.

1. Create your configuration:

   ```
   cp .env.example .env
   ```
2. Point `LLM_BASE_URL` in `.env` at your local LLM's OpenAI-compatible
   endpoint (default `http://host.docker.internal:14434`, which reaches a
   host-side LLM on port 14434 from inside the container).
3. Start the gateway:

   ```
   docker compose up
   ```

   This runs the published image `ghcr.io/ggeorgovassilis/llm-gauze:latest`
   (defined in `docker-compose.yml`). To pin a specific release, override the
   tag — for example `ghcr.io/ggeorgovassilis/llm-gauze:7`.

The gateway listens on `http://localhost:9317` and exposes an
OpenAI-compatible API (e.g. `POST /v1/chat/completions`), forwarding to the
`LLM_BASE_URL` in `.env`.

### File ownership

The container runs as your host user (`UID`/`GID`, default `1000`) so the
files it writes to the bind-mounted `data/` directory are owned by you, not
root. If a `data/` directory was created earlier as root, fix ownership once
before the non-root container can write to it:

```
sudo chown -R "$(id -u):$(id -g)" data
```

## Health check

```
curl http://localhost:9317/health
```

Returns `{"status": "ok", "upstream": "<LLM_BASE_URL>"}` when the gateway is up.
Operational metrics are exposed at `/metrics` (Prometheus text format, or JSON
with `Accept: application/json`).

## Configuration

Every setting is documented in [`docs/configuration.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/configuration.md).

## How it works

llm-gauze sits between your client and the local LLM, recording every exchange:

```
flowchart LR
    Client[Your client] -->|OpenAI-compatible API| Gauze[llm-gauze]
    Gauze -->|forwards| LLM[Local LLM]
    Gauze -.->|logs every exchange| Store[(data/*.jsonl)]
```

 Loading

When the model misbehaves, llm-gauze detects it and remediates what it can before
you ever see it — retrying transient failures, nudging empty replies, cleaning
leaked thinking tags, breaking loops with varied sampling:

```
sequenceDiagram
    participant C as Client
    participant B as llm-gauze
    participant L as Local LLM

    C->>B: POST /v1/chat/completions
    B->>L: forward
    L-->>B: error, hang, loop, or sloppy reply
    B->>B: detect & classify
    B->>L: remediate (retry / nudge / repair)
    L-->>B: clean completion
    B-->>C: chat.completion
```

 Loading

See [`docs/architecture.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/architecture.md) for the full design and
[`docs/configuration.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/configuration.md) for every setting.

## Developing

See [`DEVELOPING.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/DEVELOPING.md) for local development, running the tests,
and adding a detector or remediation.