Docker setup

LlamaNexus ships only as prebuilt images on Docker Hub - there's no source build to run yourself. It runs in Docker alongside llama-server (GPU-enabled) and Open WebUI.

Image tags

Tags follow <gpu-backend>[-<backend-version>]-<llamanexus-version|latest>:

Pick the tag matching your GPU driver/toolkit, or pin an exact version (e.g. cuda-13.0.0-0.1.21) instead of latest for reproducible upgrades.

Server

services:
  llamanexus:
    image: makershop/llamanexus:cuda-13.0.0-latest
    network_mode: host   # required for --discovery (see Distributed inference)
    command: serve --discovery --llamaport 8080 --port 11434 -- --jinja --context-shift # llama-server specific arguments after double dash
    environment:
      - HF_TOKEN=${HF_TOKEN:-}
      - LLAMANEXUS_CHAT_TIMEOUT_SECONDS=900
    volumes:
      - ./data/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Worker

services:
  llamanexus-worker:
    image: makershop/llamanexus:cuda-13.0.0-latest
    network_mode: host
    command: worker --discovery
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Using a rocm-* or vulkan-latest tag instead? Drop the Nvidia deploy.resources block above and pass through the right device nodes instead - --device=/dev/kfd --device=/dev/dri for ROCm, --device=/dev/dri for Vulkan.

See Distributed inference for --discovery/--rpc details and the network_mode: host requirement.

Volumes

Mount a persistent volume at ~/.cache/huggingface inside the container - it holds the Hugging Face cache (downloaded GGUF files) and router.preset.ini (per-model context-size overrides).