How to Deploy an LLM with vLLM and Docker as an OpenAI-Compatible API

Deploying an LLM with vLLM and Docker means packaging a high-performance inference server inside a reproducible container, giving it controlled access to GPU resources, loading a model, and exposing that model through an HTTP interface compatible with commonly used OpenAI API request patterns. Instead of every application learning a model-specific serving interface, clients can communicate with a familiar API while vLLM manages scheduling, model execution, batching, and GPU-side inference work. 

That architectural boundary matters in production. Running a model is only one part of serving it reliably: teams also have to manage model artifacts, GPU memory, concurrent requests, container versions, authentication, latency, observability, upgrades, failures, and rollback. Docker supplies a repeatable runtime boundary; vLLM supplies the inference engine and API server; the surrounding platform supplies production controls. 🏗️

📑 In This Post

1. Foundations: what vLLM and Docker actually do
2. How an inference request works
3. A practical deployment example
4. Production implementation and tuning
5. Enterprise rollout
6. Common mistakes
7. FAQ
8. References & Further Reading
9. Summary

🔀 Quick Comparison

Layer	Main responsibility	What it does not solve alone
Docker	Packages the inference runtime and dependencies and provides a repeatable container execution boundary.	LLM scheduling, token generation, model quality, or fleet-level orchestration.
vLLM	Loads models, manages inference scheduling and KV-cache resources, executes generation, and exposes serving APIs.	Enterprise identity, perimeter security, organizational governance, or complete infrastructure lifecycle management.
NVIDIA runtime	Makes selected NVIDIA GPU resources and required driver capabilities available to containers.	Model serving policy or inference scheduling.
API gateway	Can centralize authentication, TLS, rate controls, routing, and policy enforcement.	GPU execution and model inference.

1. Foundations: vLLM, Docker and the Serving Boundary

🧒 Sticky-note analogy: Think of a restaurant. Docker is the standardized kitchen space and equipment setup; vLLM is the kitchen team organizing incoming orders so the expensive oven—the GPU—keeps doing useful work instead of waiting between customers.

Technically, these components operate at different layers. Docker isolates and packages userspace software. GPU access is supplied through the host GPU driver and container runtime integration. Inside that environment, vLLM loads the model and operates the inference engine. The HTTP server turns external requests into inference work and returns generated tokens to clients.

The official vLLM project publishes the vllm/vllm-openai container image for running its OpenAI-compatible server. Its Docker documentation demonstrates GPU exposure, Hugging Face cache mounting, port publication, and shared-memory configuration. Those pieces should be understood rather than copied blindly because production requirements differ by model, GPU, host configuration, and security boundary.

Why Docker? LLM inference stacks combine Python packages, compiled kernels, CUDA-facing libraries, model-specific code, and serving components. A container image makes that userspace environment portable and testable. It does not virtualize the physical GPU itself; the host driver and runtime still matter.

Why vLLM? Interactive LLM serving is not equivalent to calling a conventional stateless function. Requests have different prompt lengths, generation lengths, arrival times, and memory footprints. vLLM is designed around high-throughput model serving and includes mechanisms such as continuous batching, KV-cache management, prefix caching capabilities, and distributed inference strategies.

💡 Important distinction: OpenAI-compatible describes an API compatibility surface; it does not mean a locally served model becomes an OpenAI-hosted model or that every behavior, parameter, extension, tokenizer, model capability, or output will be identical.

🎯 Use this when... you want applications to consume self-hosted models through a familiar HTTP contract while retaining control of the inference infrastructure.

2. Mechanics: What Happens to One Request?

🧒 Sticky-note analogy: A good dispatcher does not send every taxi back to the garage after one passenger. It continuously coordinates new passengers with available capacity. LLM serving similarly benefits from coordinating concurrent work rather than treating every request as an isolated GPU job.

Suppose an application sends a chat-completion request. The simplified production path is:

1. The client sends an HTTP request to the serving endpoint, usually through a gateway or internal load balancer in a production design.
2. The API layer validates and converts the request into model-serving inputs.
3. The tokenizer converts text into model tokens.
4. The scheduler admits work according to available execution and cache capacity.
5. Prompt processing creates the internal attention state required for subsequent generation.
6. Generated tokens are produced iteratively by the model on the GPU.
7. If streaming is enabled, token output can be returned incrementally instead of waiting for the complete response.
8. Metrics and logs provide operational signals about latency, request activity, token processing, cache utilization, and failures.

The KV cache is central to serving capacity. During autoregressive generation, attention key/value state from already processed tokens can be retained so it does not have to be recomputed from scratch for each next token. That memory competes with model weights and other runtime allocations for accelerator memory.

This explains a production behavior that surprises newcomers: a model fitting into GPU memory does not automatically mean the service has enough memory for the desired concurrency and context lengths. Serving capacity also depends on the memory available for active sequence state.

vLLM exposes information about its GPU KV-cache capacity during startup, and its production metrics include KV-cache utilization. Capacity planning should therefore consider the workload distribution—prompt lengths, generated lengths and concurrency—not just model weight size.

Continuous batching changes the mental model. Requests arrive independently, but an inference engine can continuously schedule compatible work as sequences progress. This improves accelerator utilization compared with a simplistic design that waits for a fixed batch to finish before admitting more work.

Prefix caching targets another form of waste. vLLM's automatic prefix caching can reuse cached KV state when new requests share previously computed prefixes. This is especially relevant when workloads repeatedly contain large common prefixes, such as stable system instructions or shared document context. It does not remove the cost of generating new output tokens.

🎯 Use this when... you need to reason about why concurrency, prompt length, generation length and cache pressure directly affect an inference service.

3. Practical Example: One GPU, One Model, One API

🧒 Sticky-note analogy: Before opening a factory, first prove that one production line can receive material, run correctly, and ship a finished item. Then add capacity.

Example type: hypothetical implementation based on documented vLLM and NVIDIA container mechanisms. We will expose a Hugging Face-hosted model through vLLM on a Linux GPU host. The exact model must be chosen according to its license, architecture support, GPU memory requirements, and intended workload.

Step 1 — prove the GPU works on the host. Container troubleshooting should start below Docker. If the host driver cannot operate the GPU correctly, changing vLLM arguments will not repair that layer.

Step 2 — configure container GPU access. NVIDIA's current Container Toolkit documentation instructs Docker users to configure the runtime with nvidia-ctk runtime configure --runtime=docker and restart Docker. Treat the exact installation procedure as platform-specific and follow NVIDIA's current documentation for the host distribution.

Step 3 — launch the vLLM server. The following is intentionally an original minimal example rather than a copy of a documentation command:

docker run --rm \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN \
  vllm/vllm-openai:<tested-version> \
  --model <model-id>

Example only. Replace the image version and model identifier with versions validated for your environment. For production, prefer an approved immutable image reference rather than an unqualified moving tag.

The cache mount matters operationally. Hugging Face documents local model repository caching beneath its cache hierarchy, and a persistent cache can avoid treating every container recreation as a completely fresh model download. In controlled environments, pre-staging a pinned model revision can provide stronger deployment reproducibility.

✅ Worked example: If a deployment passes host GPU checks but the container reports no GPU, investigate the Docker-to-NVIDIA runtime boundary before tuning vLLM. If the container sees the GPU but vLLM fails while allocating the model, investigate model/runtime compatibility and accelerator memory next. Troubleshooting layer by layer prevents unrelated configuration changes.

Step 4 — call the API.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id>",
    "messages": [
      {"role": "user", "content": "Explain KV cache in two sentences."}
    ],
    "max_tokens": 80
  }'

vLLM documents support for OpenAI-compatible APIs including completion and chat-completion interfaces. Applications already structured around a compatible client can often point the client toward the vLLM base URL, subject to the endpoints and request features supported by the deployed vLLM/model combination.

🎯 Use this when... you are proving the complete path from host GPU to container to model server to HTTP client before adding production infrastructure.

4. Production Implementation: Memory, Parallelism, Security and Observability

🧒 Sticky-note analogy: A bridge is not production-ready merely because one car crossed it. Engineers need to know how much traffic it can carry, how failures are detected, and what happens when demand exceeds capacity.

Start with workload measurements, not a fashionable model size. Build a representative test set containing realistic prompt lengths, output lengths, concurrency patterns, streaming behavior and important request types. A synthetic benchmark with tiny prompts can hide the memory and latency behavior that dominates the real application.

Measure latency as a distribution. For interactive generation, total response time alone is insufficient. vLLM exposes Prometheus-compatible metrics including time-to-first-token and inter-token latency, as well as prompt and generation token counts, running requests and KV-cache utilization. These signals help separate queueing, prompt-processing pressure and generation behavior.

Single GPU first. vLLM's scaling guidance recommends avoiding distributed inference when the model fits appropriately on one GPU and distributed execution is unnecessary. This keeps the topology simpler and avoids communication overhead.

Tensor parallelism when a replica needs multiple GPUs. vLLM supports splitting model execution across GPUs with tensor parallelism. This can make a model fit and alter the memory available per accelerator, but it introduces communication between participating devices. Hardware topology therefore matters.

vllm serve <model-path> \
  --tensor-parallel-size 4

Pipeline parallelism is another distribution mechanism. vLLM documents combining tensor and pipeline parallelism for models spanning larger topologies, including multi-node deployments. Do not assume that adding GPUs linearly improves application performance; communication, memory layout, request shape and model architecture affect the result.

Scale replicas for aggregate traffic when appropriate. Once one model replica has a sound configuration, additional independent serving capacity can be placed behind a load-balancing layer. vLLM also documents data-parallel deployment mechanisms. The right design depends on whether the objective is fitting one model, increasing independent request capacity, or both.

Shared memory deserves attention. vLLM's Docker documentation explains that shared memory is used by underlying multiprocessing behavior and documents either host IPC or an explicit shared-memory allocation. In hardened environments, choose the narrowest configuration that satisfies the tested workload instead of granting broad host sharing automatically.

Do not expose the raw server to an untrusted network simply because an API key flag exists. Current vLLM documentation explicitly warns that its API-key authentication does not protect every endpoint on the HTTP server and recommends stronger perimeter hardening such as a reverse proxy. In an enterprise architecture, place authentication, TLS, authorization, network policy and request controls at an appropriate trusted boundary.

Pin artifacts. Docker documents that image digests identify immutable image content. A production release should record the tested container digest, model identity/revision, serving configuration and associated evaluation result. That gives rollback a concrete target instead of a vague instruction to redeploy “the previous image.”

Protect model credentials. Hugging Face supports HF_TOKEN for authentication. In production, provide credentials through the platform's secret mechanism rather than embedding tokens in Dockerfiles, source repositories, image layers, or copied shell scripts.

Separate readiness from mere process existence. An LLM server can have a running process while still loading weights or otherwise being unable to satisfy useful inference. Upstream routing should therefore represent actual serving readiness and remove unhealthy replicas from traffic.

💡 Capacity trade-off: Maximum model context is not the same as the context length you should permit for every production request. Large contexts consume serving resources and can affect queueing and concurrency. Set application limits from measured requirements and protect the service against unexpectedly expensive requests.

Benchmark the system you will actually operate. Test warm and cold startup behavior, representative concurrent traffic, long prompts, streaming, request cancellation, overload, model-cache misses, GPU pressure, container restarts and dependency failures. Evaluate quality separately from infrastructure performance: a fast endpoint returning unacceptable answers is not a successful inference deployment.

🎯 Use this when... the prototype works and you now need predictable latency, controlled concurrency, reproducible releases and actionable telemetry.

5. Enterprise Rollout: From Working Container to Managed Service

🧒 Sticky-note analogy: A school does not change every student's textbook because one teacher liked a new edition. The new edition is reviewed, approved, introduced carefully, and replaceable if problems appear.

Treat a model-serving release as a versioned system, not merely a container. A useful release record ties together the container digest, model revision, tokenizer/configuration, inference arguments, evaluation suite, deployment policy and change approval.

1. Define ownership. Identify owners for the model, inference platform, application behavior, security controls, evaluation suite and incident response.
2. Version the model and evaluation data. A changed model or tokenizer can alter behavior even when the API contract remains unchanged.
3. Build CI gates. Scan container artifacts, validate configuration, run API-contract tests, perform representative inference checks and run model-quality regressions before promotion.
4. Control artifact access. Limit who can publish production images, change model artifacts, modify deployment arguments or retrieve protected model credentials.
5. Protect production-derived evaluation data. Prompts and responses can contain confidential or personal information. Define collection, minimization, access, retention and redaction policies before using production traffic as an evaluation corpus.
6. Establish budget controls. Track GPU allocation, replica count, utilization and workload volume. Capacity changes should have explicit operational and financial ownership.
7. Build dashboards and alerts. Watch availability, errors, queue pressure, time-to-first-token, inter-token latency, token throughput, KV-cache pressure, GPU telemetry and application-level quality indicators.
8. Use controlled rollout. Send a bounded portion of eligible traffic to the candidate release and compare it against explicit acceptance criteria before expanding exposure.
9. Predefine rollback criteria. Roll back on material quality regressions, elevated failures, unacceptable latency, resource instability, security issues or other agreed service-level violations.

Offline evaluation should test important capabilities against version-controlled representative datasets before deployment. For a RAG application, evaluate retrieval and generation separately where practical: otherwise a retrieval failure can be mistaken for a model-generation failure.

Online evaluation answers different questions. Once deployed, monitor whether the system behaves correctly under actual workload distributions. Production signals can reveal shifts in prompt length, language, retrieval behavior, traffic mix or latency that a fixed laboratory dataset did not represent.

Human review remains useful for subjective dimensions. Automated evaluators—including LLM-based judges—can accelerate regression testing but should be calibrated against human-reviewed examples for the task. Avoid letting a single automated score become the sole release gate when its biases or failure modes are poorly understood.

Canary deployment needs a rollback unit. If a candidate fails, the organization should know exactly which image digest, model revision and configuration constitute the last approved release. Immutability and release metadata turn rollback from reconstruction into redeployment.

🎯 Use this when... multiple teams or business-critical applications depend on the inference endpoint and model changes must become auditable operational releases.

6. Common Mistakes and Why They Hurt Production

🧒 Sticky-note analogy: If a race car overheats, randomly changing the tires, fuel and steering together makes diagnosis harder. Production troubleshooting works better when each system layer is tested independently.

Mistake 1: assuming “the model fits” means “the service is sized.” Model weights are only part of accelerator memory demand. Active request state, KV cache and runtime allocations matter. The production impact is usually poor concurrency, preemption pressure, allocation failure, or latency instability under realistic traffic.

Mistake 2: using a moving image tag as the production release identity. A mutable tag can resolve to different content over time. This weakens reproducibility and makes incident reconstruction harder. Record and deploy an approved immutable digest, while maintaining an explicit process for adopting security updates.

Mistake 3: exposing vLLM directly because an API key was configured. vLLM currently documents authentication limitations for endpoints outside protected path prefixes. Treat application-server authentication as one control, not as a complete Internet-facing security architecture.

Mistake 4: tuning from average latency. Averages can conceal queue spikes and slow requests. Examine distributions and component-level signals such as time-to-first-token, inter-token latency, active/waiting requests and cache utilization.

Mistake 5: benchmarking unrealistic prompts. A test consisting entirely of short requests may tell you very little about a production workload containing long context, bursty concurrency or large generations. Build representative request distributions and preserve them as regression workloads.

Mistake 6: changing model, image and runtime configuration simultaneously. When quality or performance moves, you lose attribution. Version components separately and promote controlled changes so regression analysis has a meaningful baseline.

Mistake 7: treating startup downloads as harmless. Pulling model artifacts during every replacement can make recovery dependent on external bandwidth, credentials and repository availability. Persistent caches or pre-staged approved artifacts can make startup behavior more deterministic.

Mistake 8: adding GPUs without understanding parallelism. Tensor parallelism, pipeline parallelism and independent replicas solve different problems. Cross-device communication can become significant. Select topology based on whether you need model-fit capacity, aggregate throughput, or both.

Mistake 9: monitoring only GPU utilization. A busy GPU does not prove users receive good service. Couple infrastructure metrics with request latency, queue behavior, errors, token throughput and application-quality signals.

🎯 Use this when... a deployment works in a demo but becomes unstable, slow, difficult to reproduce, or difficult to diagnose under production load.

7. ❓ FAQ

1. Does OpenAI-compatible mean every OpenAI API feature works identically?

No. Compatibility applies to supported API surfaces and request/response conventions. Available endpoints, parameters and model behaviors depend on the vLLM version and the model being served. Validate the exact client workflow you need.

2. Do I need multiple GPUs to use vLLM?

No. A model that fits appropriately on one supported GPU can be served without distributed inference. Multiple-GPU strategies become relevant when model fit, memory requirements or the chosen serving architecture require them.

3. Why persist the Hugging Face model cache?

A persistent cache can allow downloaded model artifacts to survive container replacement, reducing unnecessary repeated transfers. Production releases should still identify the approved model revision rather than treating whatever happens to be cached as the release definition.

4. What metrics should I watch first?

Start with request success/failure, request concurrency or queue pressure, time-to-first-token, inter-token latency, prompt and generation token volume, KV-cache utilization, and relevant GPU telemetry. Add application-quality and cost indicators so infrastructure health is not mistaken for product quality.

5. Is vLLM's API key enough for a public production endpoint?

No. Current vLLM documentation states that its API-key mechanism does not authenticate every endpoint on the HTTP server. Use an appropriate trusted security boundary such as a hardened proxy or gateway and apply network, TLS, identity, authorization and request-control policies there.

8. 🔗 References & Further Reading

• vLLM Documentation — Using Docker — official container image, GPU execution, model-cache mounting, shared memory and non-root deployment guidance.
• vLLM Documentation — OpenAI-Compatible Server — supported serving APIs and current API-key security limitations.
• vLLM Documentation — Parallelism and Scaling — single-GPU, tensor-parallel, pipeline-parallel and multi-node deployment guidance.
• vLLM Documentation — Metrics — Prometheus-compatible inference and latency metrics.
• vLLM Documentation — Automatic Prefix Caching — reuse of KV-cache state for matching prefixes.
• NVIDIA Documentation — Container Toolkit Installation Guide — official Docker runtime configuration for NVIDIA GPU access.
• Hugging Face Documentation — Hub Environment Variables — HF_TOKEN and cache configuration.
• Hugging Face Documentation — Hub Local Cache — model repository cache organization and location.
• Docker Documentation — Pulling Images by Digest — immutable image selection and digest pinning.
• Docker Documentation — Resource Constraints — host resource controls for containers.

Attribution notice: Docker is a trademark of Docker, Inc.; NVIDIA and CUDA are trademarks of NVIDIA Corporation; Hugging Face is associated with Hugging Face, Inc.; OpenAI is a trademark of OpenAI. Other names belong to their respective owners.

9. 📝 Summary

• Foundations: Docker provides the repeatable runtime boundary while vLLM provides the model-serving engine and compatible HTTP interface.
• Mechanics: Production inference depends on scheduling, token processing, KV-cache capacity, batching and GPU execution—not simply loading model weights.
• Practical deployment: Validate the host GPU, container GPU access, model server and API independently before adding complexity.
• Production implementation: Size from representative traffic, observe latency distributions and cache pressure, secure the perimeter, and pin deployable artifacts.
• Enterprise rollout: Version models, images, configuration and evaluation data together; introduce changes through controlled gates, canaries and explicit rollback criteria.
• Common mistakes: Avoid moving release identities, unrealistic benchmarks, single-metric monitoring and parallelism choices made without understanding their purpose.
• FAQ: OpenAI compatibility simplifies client integration but does not remove the need to validate supported features, capacity, security and operational behavior.

Build the smallest measurable serving path first, make every release reproducible, and scale only after the bottleneck is understood. Happy building! 🚀