How do we deploy an LLM on-premise with Docker?

Question

Nanobase AI Engineering · Accepted Answer

Deploying an LLM on-premise with Docker means packaging the inference engine, model weights and any supporting services into containers so the whole stack can be started, updated and moved consistently across servers. In practice this starts with an inference engine image, typically vLLM's official container or NVIDIA's NIM container, run with the NVIDIA Container Toolkit so the container can access the host's GPUs, and the model weights either baked into the image or mounted from local storage. A docker-compose file usually ties together the inference container, a vector database like Qdrant for retrieval, a chat front end such as Open WebUI, and a reverse proxy handling TLS and authentication, so the entire stack comes up with a single command. This approach works well for a single server or a small number of GPUs and gives a team a working private LLM environment in hours rather than weeks. It becomes limiting once an organization needs multi-node scaling, automatic failover, or fine-grained resource scheduling across many GPUs, at which point most teams migrate the same containers onto Kubernetes with the GPU Operator. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds these Docker-based deployments for initial rollouts and migrates them to Kubernetes when scale requires it.

Docker's job is consistency, not performance

Docker does not make an LLM run faster; GPU throughput is identical whether the inference engine runs in a container or directly on the host. What Docker provides is a consistent, repeatable way to deploy, update, and move the inference engine, chat interface, and supporting services across servers without each one having a hand-tuned, drift-prone installation. The value of Docker for an on-premise LLM is operational: the same container image that worked in testing behaves identically in production and on a replacement server, which removes an entire class of "it worked on my machine" failures.

Prerequisites before the first container runs

Skipping any one of these three prerequisites is the single most common reason a first GPU container deployment fails before ever reaching the application itself.

Install the NVIDIA drivers matching the GPU on the host operating system, since the container shares the host's GPU driver rather than bundling its own.
Install the NVIDIA Container Toolkit, which lets Docker containers access the host's GPU; without it, a container cannot see the GPU at all regardless of image configuration.
Verify GPU visibility from inside a test container before deploying anything else, since a driver or toolkit mismatch is easier to diagnose in isolation than inside a full application stack.

A minimal working example

The example below is intentionally minimal, and every gap it leaves open is listed explicitly in the next section rather than left as a silent assumption. The following illustrates the shape of a docker-compose setup pairing vLLM with a chat interface; exact image tags and configuration should be checked against current documentation before use.

services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - ./models:/models
    command: --model /models/llama-3.1-70b --quantization fp8
    ports:
      - "8000:8000"

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
    ports:
      - "3000:3000"
    depends_on:
      - vllm

This pairs a vLLM container serving an OpenAI-compatible API with Open WebUI pointed at it, the same pattern used in most single-server pilots regardless of which specific inference engine or chat interface is chosen.

What this simple setup does not cover

Gap in the minimal example	What production needs instead
No persistent volume for chat history	Mounted volume or external database for Open WebUI data
No TLS	Reverse proxy (nginx, Traefik) terminating TLS in front of both services
No SSO	OIDC configuration added to Open WebUI, identity provider registration
No health checks or restart policy	`restart: unless-stopped` and defined health checks for both services
No multi-GPU or multi-node support	Kubernetes, covered under Kubernetes vs Docker Compose for on-prem LLM

A working docker-compose file is a legitimate way to reach a functioning pilot within a day, but it should be treated explicitly as a pilot configuration, with a known list of gaps to close before real users and sensitive documents depend on it.

Frequently asked questions

Does the model need to be inside the Docker image?

No, and generally should not be; model weights are typically mounted as a volume from local storage rather than baked into the image, since weights are large and change independently of the inference engine's software version.

Can this same Docker setup work in an air-gapped environment?

Yes, with adjustment: all images need to be pulled and transferred to an internal registry ahead of time rather than pulled from a public registry at deploy time, following the same transfer process used in air-gapped LLM deployment.

How do updates work with this setup?

Updating means pulling a new image tag and recreating the container, which Docker Compose handles with a single command; the key operational practice is pinning specific version tags rather than latest in production, so an update is a deliberate, tested action rather than something that happens on a routine restart.

How Nanobase AI helps

Nanobase AI builds production-grade Docker and container deployments for on-premise LLMs, closing the gaps a minimal example leaves open: persistent storage, TLS, SSO, health checks, and a controlled update process. The team also advises on when a Docker Compose setup has reached its limits and a move to Kubernetes makes sense.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.