nagaozen/bonsai-ternary-8b repository overview

⁠Ternary Bonsai 8B — Docker Image

CPU-only inference server for Ternary Bonsai 8B⁠, a 1.58-bit quantized language model (Qwen3-8B base) by PrismML. Serves an OpenAI-compatible API via llama-server from the PrismML fork of llama.cpp⁠.

⁠Build

Requires Docker with BuildKit enabled. The build compiles llama.cpp from source (~10 min) and downloads the model weights (~2.2 GB). Subsequent rebuilds use cached layers.

DOCKER_BUILDKIT=1 docker build -t bonsai-ternary-8b:local .

If a previous build was cancelled mid-download, clear the cache first to avoid a corrupt model file:

docker builder prune --filter type=exec.cachemount
DOCKER_BUILDKIT=1 docker build -t bonsai-ternary-8b:local .

⁠Run

docker run --rm -p 8080:8080 \
  -e LLAMA_THREADS=8 \
  -e LLAMA_THREADS_BATCH=16 \
  -e LLAMA_CTX_SIZE=16384 \
  bonsai-ternary-8b:local

⁠Environment Variables

⁠Context Size vs. Memory

⁠API

The server exposes an OpenAI-compatible REST API on port 8080.

⁠Health check

curl http://localhost:8080/health

⁠Chat completion

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ternary-bonsai-8b",
    "messages": [
      {"role": "user", "content": "Explain ternary quantization in two sentences."}
    ]
  }'

⁠Streaming

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ternary-bonsai-8b",
    "stream": true,
    "messages": [
      {"role": "user", "content": "Write a haiku about compressed models."}
    ]
  }'

⁠Generation Parameters (defaults)

These are set in the entrypoint and reflect PrismML's recommended values for the model.

To override any parameter at runtime, append flags after the image name:

docker run --rm -p 8080:8080 bonsai-ternary-8b:local --temp 0.7 --top-p 0.95

⁠Thinking Mode

Thinking mode is enabled by default. The model will produce chain-of-thought reasoning inside <think>...</think> tags before its final answer. This improves accuracy on complex tasks but adds latency.

To disable it:

docker run --rm -p 8080:8080 bonsai-ternary-8b:local \
  --reasoning off \
  --reasoning-budget 0 \
  --reasoning-format none

⁠CPU Tuning Tips

Decode vs. prefill are different bottlenecks — tuning them separately is the biggest lever:

• LLAMA_THREADS (decode) — set to physical core count, not logical. Hyperthreading does not help and can hurt due to cache contention. On a 16-core/32-thread host, try 8–12.
• LLAMA_THREADS_BATCH (prefill) — set to all physical cores available to the container. Prefill is a large matrix multiply that scales well across cores.
• LLAMA_BATCH_SIZE — default 4096 is a good balance. If you have spare RAM, try 8192 for heavier prompt workloads.

Other tips:

• On NUMA systems, pin to a single node to avoid cross-socket memory latency: docker run --cpuset-cpus="0-7" --cpuset-mems="0" ...
• Add --no-mmap if the model is being evicted from OS page cache under memory pressure — forces an upfront load into RAM.
• Throughput on x86-64 with AVX2: approximately 30–35 tok/s decode at 8–10 threads (comparable to PrismML's published 32 tok/s on 10-thread NEON).

⁠Notes on the llama.cpp Fork

The Q2_0 ternary quantization type (GGML type 42) is not yet merged into upstream llama.cpp. This image builds from PrismML's prism branch. Once an upstream PR lands or PrismML publishes a tagged Ternary-specific release, the --branch prism line in the Dockerfile should be replaced with a pinned tag for reproducible builds.