How to Run a Local LLM on Your Homelab: Ollama + Open-WebUI with Docker Compose

Running a local LLM self-hosted on your homelab used to mean wrestling with CUDA versions, Python venvs, and model conversion scripts at 1am. Ollama killed that pain. One binary, one API, GGUF models on tap. Pair it with Open-WebUI behind Traefik and you have a self-hosted ChatGPT alternative running in 15 minutes. This guide is the ollama docker compose setup I wish r/homelab had pinned — GPU passthrough for Nvidia and AMD, reverse proxy labels, and which quantised models are actually usable on consumer hardware.

1. Install the Nvidia Container Toolkit (or ROCm runtime for AMD) on the Docker host so the GPU is visible inside containers.
2. Drop the Ollama + Open-WebUI docker-compose.yml into a stack directory and bring it up with docker compose up -d.
3. Pull a quantised model that fits your VRAM with docker exec ollama ollama pull llama3.1:8b-instruct-q4_K_M.
4. Add Traefik labels to expose Open-WebUI on a subdomain with automatic Let's Encrypt TLS.
5. Lock down the Ollama API container so only Open-WebUI (and your LAN) can hit it.
6. Test inference, then create your first admin user in Open-WebUI before anyone else can grab it.

What hardware do you actually need to run a local LLM at home?

The minimum viable rig is 16GB of VRAM or 32GB of unified memory. Anything less and you're stuck with 7B-or-smaller models at heavy quantisation, which is fine for autocomplete but painful for conversation.

The sweet spot in 2026 is a used RTX 3090 (24GB VRAM, ~£600 on eBay) or an Apple Silicon Mac mini M4 Pro with 48GB unified memory. Both run Llama 3.3 70B at Q4 quantisation at usable token rates. A refurb Beelink SER8 with 64GB DDR5 will run mid-size models on CPU at around 4-6 tokens/sec — slow but free of GPU power draw.

Skip anything with less than 12GB VRAM unless you only care about embeddings or tiny coding models. The 3060 12GB is the genuine entry point; the 4060 Ti 16GB is the better buy if you can stretch.

The Docker Compose file that actually works

This compose file runs Ollama with Nvidia GPU passthrough and Open-WebUI on the same internal network. No port-forwarding, no host networking hacks, no envsubst gymnastics.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ./ollama-data:/root/.ollama
    networks:
      - llm
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_HOST=0.0.0.0

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    volumes:
      - ./webui-data:/app/backend/data
    networks:
      - llm
      - proxy
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
    labels:
      - traefik.enable=true
      - traefik.http.routers.webui.rule=Host(`chat.yourdomain.tld`)
      - traefik.http.routers.webui.entrypoints=websecure
      - traefik.http.routers.webui.tls.certresolver=letsencrypt
      - traefik.http.services.webui.loadbalancer.server.port=8080
      - traefik.docker.network=proxy

networks:
  llm:
    driver: bridge
  proxy:
    external: true

The OLLAMA_KEEP_ALIVE=24h bit is non-obvious — by default Ollama unloads models after 5 minutes of idle, which means a cold inference every time you fire up a chat. Twenty-four hours is overkill for a public service but perfect for a homelab where the model stays warm in VRAM all day. If you've already got Traefik running from the Traefik 3 + Docker Compose setup, the proxy network already exists.

How do you pass through an Nvidia or AMD GPU to Docker?

For Nvidia, install nvidia-container-toolkit on the host and add the deploy block shown above. For AMD, use the rocm/dev-ubuntu image variant and pass /dev/kfd and /dev/dri devices instead.

Nvidia setup on Ubuntu 24.04:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify with docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi. If that shows your card, the compose stack will pick it up automatically.

AMD is messier. Use image: ollama/ollama:rocm instead of ollama/ollama:latest, drop the deploy block, and add:

    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video

RDNA3 cards (7900 XTX, 7900 XT) work well. RDNA2 (6000-series) needs HSA_OVERRIDE_GFX_VERSION=10.3.0 as an env var. The official ROCm support matrix is the source of truth — the Ollama docs are out of date.

Which quantised models are actually usable on consumer hardware?

Stick to Q4_K_M for the best size-to-quality ratio. Q5_K_M is worth it if your VRAM can hold it; Q3 and below noticeably degrade output quality.

What fits where:

• 8GB VRAM (3060/4060): Llama 3.2 3B, Phi-3.5 Mini, Qwen 2.5 7B Q4. Fine for autocomplete and quick lookups.
• 12GB VRAM (3060 12GB): Llama 3.1 8B Q4_K_M, Mistral Nemo 12B Q4. The genuine entry tier for conversational use.
• 16GB VRAM (4060 Ti): Qwen 2.5 14B Q4, Codestral 22B Q4. Coding gets usable here.
• 24GB VRAM (3090/4090): Qwen 2.5 32B Q4, Llama 3.3 70B Q2_K (tight), Mistral Small 24B Q5. This is where local LLMs stop feeling like a toy.
• 48GB+ (dual GPU or Mac): Llama 3.3 70B Q4_K_M. Genuinely competitive with GPT-4o-mini for most tasks.

Pull them with docker exec ollama ollama pull qwen2.5:14b-instruct-q4_K_M. Don't bother with the unquantised :latest tags — they're rarely what you want and they're enormous.

How do you expose Open-WebUI safely without opening it to the internet?

Don't port-forward it. Put Open-WebUI behind Traefik on a Tailscale or WireGuard network, or restrict access via Traefik's IP whitelist middleware.

The compose above exposes it on a subdomain. If that subdomain resolves to a public IP, anyone on the internet can hit your login page. Three reasonable patterns:

• Tailscale-only: bind Traefik to the Tailscale interface. Only devices on your tailnet can reach it. Free, zero config rot.
• IP whitelist middleware: add traefik.http.middlewares.lan-only.ipallowlist.sourcerange=192.168.0.0/16,10.0.0.0/8 to a router. Public DNS, private access.
• Authelia / Authentik in front: add MFA before Open-WebUI's own auth screen. Worth it if family or housemates use it.

Whatever you do, set WEBUI_AUTH=true and create the admin account before exposing the service — the first signup becomes admin, full stop. Backblaze B2 is also worth wiring in for off-site backups of the webui-data volume; you'll accumulate chat history and custom prompts faster than you expect.

Should you actually self-host an LLM, or just use the API?

Self-host if you want privacy, offline use, or are running it constantly. Use APIs if you spend less than £20/month on inference and don't care who sees your prompts.

Power draw matters. A 3090 at full tilt pulls 350W. Run it 8 hours a day and you're looking at ~£15/month in UK electricity. That's the same as Claude Pro. The justification has to be privacy, integration depth (Home Assistant, n8n, code agents pointed at http://ollama:11434), or the fact that you already own the hardware.

For development workflows specifically, local LLMs shine for autocomplete via VS Code extensions like Continue — no rate limits, no latency to a US datacentre, no leaking proprietary code.

Final verdict

Ollama + Open-WebUI is the boring, correct choice for a self-hosted ChatGPT alternative. The compose file above is what I run on my own homelab. Pick a model that fits your VRAM at Q4_K_M, put it behind Traefik or Tailscale, and stop paying for inference you can do yourself.

FAQs

Is Ollama better than LM Studio for a homelab setup?

For headless servers, yes — Ollama is API-first, runs cleanly in Docker, and has the larger integration ecosystem (Open-WebUI, Continue, n8n). LM Studio is a GUI desktop app and not designed for always-on server use.

Can I run Ollama without a GPU?

Yes, but only models up to about 13B parameters are usable on CPU, and even then you're looking at 4-8 tokens per second on modern Ryzen or Intel chips. For anything bigger or faster, you need a GPU with at least 12GB of VRAM.

How do I update Ollama and Open-WebUI without losing my chat history?

Run docker compose pull && docker compose up -d. Chat history, users, and model files live in the named volumes (ollama-data, webui-data), so the containers can be recreated freely. Just back up those two directories.

Does Open-WebUI support multiple users on the same Ollama instance?

Yes. Open-WebUI has built-in multi-user auth with role-based access, per-user chat history, and admin controls over which models each user can access. The first registered account becomes admin automatically.

What's the fastest quantised model for code completion on a 12GB GPU?

Qwen2.5-Coder 7B at Q4_K_M is the current sweet spot — fast enough for inline completion (30+ tokens/sec on a 3060) and noticeably better than DeepSeek Coder V2 Lite at the same size. Pull it with ollama pull qwen2.5-coder:7b-instruct-q4_K_M.