Running vLLM on a Small Home Lab with Docker Compose

Introduction

Most vLLM guides assume you have a real accelerator behind the box: an NVIDIA card, a larger RAM budget, and enough headroom to talk about aggressive batching and big instruct models. That is not this machine. The target for this article is a much smaller home-lab node with an Intel Core i5-10400, Intel UHD 630 integrated graphics, and 16 GB of RAM.

That hardware changes the whole deployment strategy. Intel UHD 630 is not the Intel ARC or Intel Data Center GPU path that current vLLM XPU guidance targets, so this host should be treated as CPU-only. That rules out the usual GPU playbook. It also means the right question is no longer “how do I squeeze the biggest model into the node?” but “how do I run a small model cleanly, predictably, and in a way that still teaches good production habits?”

This guide uses Qwen/Qwen3-0.6B as the primary model because it is small enough to be realistic on this box, it is a natural fit for a CPU-only deployment, and it avoids the friction that comes with gated model access. We will package it with Docker Compose, expose the OpenAI-compatible API that vLLM already provides, and keep the setup honest about what this hardware can and cannot do well.

The shape of the article is borrowed from stronger production-oriented vLLM guides: start with what vLLM is good at, explain why Docker matters, walk through a concrete deployment, then spend time on health, security, metrics, and tuning. The difference here is that every recommendation is adapted to a small CPU-only home lab instead of a GPU server.

The model page for the exact build used in this article is on Hugging Face at https://huggingface.co/Qwen/Qwen3-0.6B. That is the canonical source for the model card, file list, license, and upstream usage notes.

What vLLM Gives You

The official vLLM docs split usage into a few broad paths: inference, serving, and deployment. For this article we only care about the serving and deployment path. That matters because the value of vLLM is not “run any model anywhere.” The value is a serving engine with a stable API surface and operational features you can actually build around.

Even on a small machine, three parts are still useful:

• an OpenAI-compatible API, so your client code can point at a self-hosted endpoint without a custom protocol
• better request handling than a toy local runtime when you have more than one caller
• a deployment shape that looks like a service, not a single interactive command

That is why vLLM still makes sense here even though the hardware is small.

Why vLLM Still Makes Sense on a Small Box

If your only goal is the easiest possible local model runtime, tools like Ollama or llama.cpp are simpler. That is still true. They are often a better fit for one user on one laptop. vLLM becomes interesting when the thing you care about is not only the model, but the service around the model.

On a Core i5 home-lab host, you are not chasing extreme throughput. You are learning how an inference service behaves, wiring clients against the OpenAI API format, understanding how health and metrics work, and building a path from “small lab box” to “larger inference node” later.

That difference matters. Once your application talks to an OpenAI-compatible endpoint behind Docker Compose, moving later to a better server or a GPU-backed host becomes mostly an infrastructure change instead of an application rewrite.

Why Docker Still Matters Here

The reference articles all spend time on Docker for a reason. Even if the host is small, containerizing vLLM still solves real problems:

• dependency isolation, so you do not hand-assemble Python, system packages, and runtime flags on the host
• repeatability, so the same Compose file can move from WSL2 to a real Linux box later
• persistent model caching, so restarts do not re-download everything
• cleaner operational boundaries around restart policy, health checks, port binding, and metrics

That is the right mental model for this setup. We are not using Docker because it looks more serious. We are using it because it gives us a cleaner service boundary.

Hardware Reality Check

Before you pull any image, set expectations correctly.

For this box, these are the realistic bands:

• Qwen/Qwen3-0.6B: recommended default
• Qwen/Qwen2.5-0.5B-Instruct: safe fallback if you want an even lighter model
• 1B to 1.1B instruct models: possible, but slower and tighter on memory
• 3B and above: no longer a good default for this hardware
• 7B class models: not recommended for the setup in this article

The sample and tuning guidance below are built around the first case. That is not because larger models are theoretically impossible in every circumstance. It is because this article is trying to give you a setup that stays usable after the first successful boot.

When This Setup Is a Good Fit

Use this setup when you want one or more of these:

• a private OpenAI-compatible endpoint for experiments or internal tools
• a realistic self-hosted service for learning vLLM operations
• a CPU-only fallback environment before you buy stronger hardware
• a small internal assistant where modest latency is acceptable

Do not use this setup when you need one or more of these:

• large instruct models
• high request concurrency
• low latency under sustained load
• public internet exposure without a proper gateway and rate limiting

That boundary is important. A small home lab can still be useful, but only if you stay inside the envelope.

Architecture Overview

The deployment is intentionally small.

flowchart LR
    Client[OpenAI SDK or curl] --> API[Docker Compose vLLM API]
    API --> Model[Qwen3-0.6B]
    API --> Cache[Persistent Hugging Face Cache]
    Prom[Prometheus] --> API

The stack has two services:

• vllm: serves Qwen/Qwen3-0.6B through the OpenAI-compatible API on port 8000
• prometheus: scrapes /metrics so you can watch queue depth, latency, and cache behavior

The API is bound to 127.0.0.1 in the Compose file. That is deliberate. The stronger production guides make the same point in a different context: the inference engine should sit behind a gateway, not out on the open network. On a small home-lab node, localhost binding is the simplest way to enforce that boundary.

Objectives

This article is trying to do four things at once:

1. Run vLLM on hardware that is far below the usual GPU-oriented examples.
2. Keep the deployment shape production-minded even though the box is small.
3. Expose a stable OpenAI-compatible API that local clients can use immediately.
4. Leave you with a tuning path instead of a one-shot demo command.

Prerequisites

You need a Linux environment. On Windows, that usually means WSL2 with a recent Ubuntu distribution.

Minimum host setup:

• Docker Engine with Compose support
• Enough free disk space for the container image and model cache
• Network access to Hugging Face for the first model pull
• curl on the host for validation commands

Recommended quick checks:

docker --version
docker compose version
uname -a
lscpu | grep -E 'Model name|CPU\(s\)|Flags'
free -h

The CPU docs call out avx512f as recommended and avx2 as limited-feature support. An i5-10400 is an avx2-class part, which is one more reason to stay in the very small model range.

You should also be realistic about disk behavior. The first run is slow because you pay for image pull, model download, and model initialization. The second run is where persistent cache starts paying you back.

Hugging Face Model Source

The exact model ID used throughout this guide is Qwen/Qwen3-0.6B.

• Hugging Face model page: https://huggingface.co/Qwen/Qwen3-0.6B
• Hugging Face files view: https://huggingface.co/Qwen/Qwen3-0.6B/tree/main

That page is worth checking directly before you pin a revision or change runtime settings. It shows the current files, model card, license, and upstream notes about thinking mode and sampling.

Installing the Hugging Face CLI

Yes, you can pull the model with the Hugging Face CLI first and then use it from vLLM afterward.

You do not have to do that, though. vLLM can also download the model directly on first startup when you pass Qwen/Qwen3-0.6B as the model ID and provide a writable Hugging Face cache mount. In other words, the CLI pre-download path is optional, not required.

According to the Hugging Face CLI docs, the recommended installer on Linux and macOS is:

curl -LsSf https://hf.co/cli/install.sh | bash

After installation, verify it:

hf --help

If you prefer Python packaging instead of the standalone installer, the docs also support:

pip install -U "huggingface_hub"

If you need access to gated or private repositories later, log in with:

hf auth login

For this specific model, login is usually not required because Qwen/Qwen3-0.6B is public. It still does no harm to have the CLI ready.

Pre-Downloading the Model with the Hugging Face CLI

There are two practical ways to use the CLI before starting vLLM.

Before going deeper into those options, it is worth being explicit about the default path: if you simply start the current Compose sample as-is, vLLM itself will pull Qwen/Qwen3-0.6B from Hugging Face on the first run and store it in ./data/huggingface through the mounted cache volume.

That means you have three valid workflows:

1. Let vLLM download the model directly on first startup.
2. Pre-fill the Hugging Face cache with the hf CLI.
3. Download into a regular local folder and point vLLM at that mounted path.

Option 1: Pre-fill the same Hugging Face cache used by the container

This is the cleanest fit for the current sample because the Compose file already mounts ./data/huggingface into the container cache path.

mkdir -p ./data/huggingface
HF_HOME=./data/huggingface hf download Qwen/Qwen3-0.6B

That command downloads the model into the standard Hugging Face cache structure under ./data/huggingface. After that, you can keep the Compose file exactly as it is and start vLLM normally:

docker compose up -d

In that mode, vLLM still uses the model ID Qwen/Qwen3-0.6B, but most or all of the files are already present in the mounted cache.

Option 2: Download to a regular local folder and point vLLM at that path

If you want a more explicit local model directory, the Hugging Face docs also support downloading to a chosen folder with --local-dir:

hf download Qwen/Qwen3-0.6B --local-dir ./models/Qwen3-0.6B

That creates a normal local folder with model files in it. You can then mount that folder into the container and point vLLM at the mounted path instead of the Hub model ID.

For example, the Compose service would need a volume like:

volumes:
  - ./models/Qwen3-0.6B:/models/Qwen3-0.6B:ro

And the model argument would become:

command:
  - --model
  - /models/Qwen3-0.6B

That approach is useful if you want a more explicit offline-style setup or if you want to inspect the downloaded snapshot directly.

Which one fits this article best?

For the current sample, Option 1 is better because it does not require changing the Compose file. You pre-download into the same cache directory and still run vLLM with the normal Qwen/Qwen3-0.6B model ID.

Use Option 2 when you deliberately want the model as a visible local folder and are willing to point vLLM at a mounted path.

If you do not care about pre-downloading at all, the simplest path is even shorter: start the Compose stack and let vLLM fetch the model itself.

Sample Files

All files used in this article live under the sample folder below:

• Sample folder
• docker-compose.yml
• docker-compose.production.yml
• docker-compose.observability.yml
• Nginx production config
• Token gateway app
• Token gateway Dockerfile
• prometheus.yml
• alerts.yml
• Grafana datasource provisioning
• client.py
• README.md

Implementation

Start by copying the sample folder and editing the Compose file directly. The sample is intentionally small, but each piece has a job.

Why the Compose stack looks this way

The Compose stack uses the official CPU image path, vllm/vllm-openai-cpu, and keeps the concrete defaults in the file itself. For a one-node home-lab sample, that is easier to read and easier to run than splitting the defaults across Compose and an env file. If you want to hard-pin a different CPU image tag, change that one line directly.

The baseline service does four important things:

• persists the Hugging Face cache so the model is not downloaded every restart
• adds a health check with a long enough startup window for model loading
• enables CPU-related container permissions that help vLLM's NUMA and scheduling behavior
• keeps the API on localhost instead of exposing it everywhere

The sample also enables an API key. vLLM does not ship with a complete auth system, but --api-key is still better than leaving a local endpoint totally unauthenticated if other processes on the machine can reach it.

Compose walkthrough

The easiest way to understand the sample is to read it in blocks.

Image and cache

The vllm service uses a concrete CPU image and mounts ./data/huggingface into /root/.cache/huggingface. That follows the same operational advice you see in other vLLM Docker guides: model weights should live on persistent storage. Otherwise every restart becomes part deployment and part download job.

If you use hf download with HF_HOME=./data/huggingface before startup, this same mount also doubles as your pre-populated local cache.

Local-only port binding

The port mapping uses 127.0.0.1:8000:8000. This is one of the most important lines in the file. It means the service is reachable from the local host only. If you later want LAN access, put Nginx, Traefik, Caddy, or another deliberate gateway in front of it and move auth and rate limiting there.

CPU-specific container flags

The sample uses:

• cap_add: SYS_NICE
• security_opt: seccomp=unconfined
• shm_size: 4g

These come from the CPU deployment guidance around scheduling and shared-memory behavior. On small hosts you should avoid the temptation to jump straight to privileged: true. The current sample keeps the scope narrower.

Health check and startup window

The health check calls /health from inside the container and gives the service a start_period of 180s. This is the same idea you see in larger deployment guides that use even longer probe delays for big GPU models. A model server does not look like a normal stateless web service at startup. You need to give it time to initialize before Docker decides it is dead.

Request logging

The sample uses --disable-log-requests. That follows the same pattern as other production-flavored guides: request-level logging has I/O cost and can leak prompt content into logs. On a small box, avoiding extra overhead is sensible.

Prefix caching

The sample enables --enable-prefix-caching. For repeated system prompts or repeated templates, that can reduce repeated work and improve time to first token. On a small CPU host it will not turn the node into a fast server, but it is one of the few low-risk optimizations worth keeping enabled.

In-Compose Settings and Secrets

This sample keeps its defaults directly in docker-compose.yml instead of an .env file. That makes the stack more obvious to read on first pass and avoids the extra “copy this template, then open a second file” step.

There are three groups worth thinking about separately:

Image and model selection

• image tag
• model name
• dtype

Pin the image to a real CPU release tag when you start using this seriously. Leave the model at Qwen/Qwen3-0.6B until the rest of the stack is stable.

Resource controls

• --max-model-len
• --max-num-seqs
• --max-num-batched-tokens
• VLLM_CPU_KVCACHE_SPACE
• VLLM_CPU_NUM_OF_RESERVED_CPU
• VLLM_CPU_OMP_THREADS_BIND

These are your real tuning knobs. If performance or stability is off, most of the work happens here.

Secrets and access

• HF_TOKEN
• --api-key

Qwen/Qwen3-0.6B should not require a gated-model token, but leaving HF_TOKEN in the Compose file keeps the sample reusable. The built-in API key is not a complete security model, but it is still a worthwhile baseline.

If you later move this to a shared host, that is the point where you should externalize secrets again.

Compose Configuration That Fits This Host

The default tuning in the sample is intentionally conservative.

These are not maximum-throughput numbers. They are “this box should stay up and remain usable” numbers. That distinction is important because many public vLLM tuning examples are trying to maximize throughput on accelerators. This article is doing almost the opposite: we are right-sizing the service so it behaves consistently on a constrained CPU host.

Why These Knobs Matter

The external Docker guides tend to focus on GPU knobs such as --gpu-memory-utilization, tensor parallelism, quantization format, and multi-GPU layout. None of that is relevant here. For this machine, the main performance and stability levers are different.

--max-model-len

This is the first lever to pull because longer context windows increase KV cache pressure and total work per request. If your internal tool only needs short prompts, there is no reason to carry a larger context budget.

--max-num-seqs

This controls how many sequences vLLM tries to handle concurrently. Higher values improve aggregate throughput when the hardware has room. On this host, high values are more likely to increase queueing and latency.

--max-num-batched-tokens

This is the second big throughput-versus-latency tradeoff. A larger batch budget can improve efficiency, but it also increases pressure on memory and response time. On a small CPU host, conservative values are safer.

VLLM_CPU_KVCACHE_SPACE

This defines how much memory is reserved for the CPU KV cache. Too high and the node becomes memory-tight. Too low and the runtime gets less room to work with. On 16 GB RAM, this is worth watching closely.

VLLM_CPU_OMP_THREADS_BIND

This is the most host-specific setting in the sample. auto is the safest default. A custom physical-core list can help once you know the exact CPU topology inside Linux or WSL2.

Performance Tuning for a Core i5-10400

There are two kinds of tuning on this machine: safe tuning and aggressive tuning. On better hardware you might frame that as “baseline” and “optimized.” On this box, aggressive still needs to mean “careful.”

Safe starting point

Use the sample defaults first.

That means:

• VLLM_MAX_MODEL_LEN=2048
• VLLM_MAX_NUM_SEQS=4
• VLLM_MAX_NUM_BATCHED_TOKENS=512
• VLLM_CPU_NUM_OF_RESERVED_CPU=1
• VLLM_CPU_OMP_THREADS_BIND=auto

This is the right place to begin if you care about stability more than squeezing out a few more tokens per second.

What this profile optimizes for:

• predictable startup
• lower risk of memory pressure
• acceptable single-user or light shared usage
• a cleaner baseline for later tuning

Tuned profile for this exact host

Once the baseline works, test this host-specific profile:

VLLM_MAX_MODEL_LEN=1024
VLLM_MAX_NUM_SEQS=6
VLLM_MAX_NUM_BATCHED_TOKENS=768
VLLM_CPU_KVCACHE_SPACE=3
VLLM_CPU_NUM_OF_RESERVED_CPU=1
VLLM_CPU_OMP_THREADS_BIND=0-5

This profile trades some context length for more predictable latency and slightly better CPU locality. It assumes you have confirmed that 0-5 maps to one logical thread per physical core on your Linux or WSL guest. Do not hard-code that blindly on every machine. Check with lscpu -e first.

What this profile is trying to do:

• shrink the context budget to lower memory pressure
• allow a little more concurrency without letting requests pile up too deeply
• keep batch size restrained enough that the node does not thrash
• bind work more deliberately to the physical cores that matter most

What to tune first

If performance is poor, tune in this order:

1. Reduce VLLM_MAX_MODEL_LEN
2. Reduce VLLM_MAX_NUM_SEQS
3. Reduce VLLM_MAX_NUM_BATCHED_TOKENS
4. Lower VLLM_CPU_KVCACHE_SPACE if RAM pressure is visible
5. Replace auto thread binding with a confirmed physical-core list

On a small CPU host, shorter context and lower concurrency usually help more than clever tricks.

A Practical Tuning Loop

Use a short loop instead of random edits.

1. Start with the safe baseline.
2. Send five to ten repeatable test requests.
3. Watch /metrics, docker stats, and system memory.
4. Change one variable.
5. Repeat the same request pattern.
6. Keep the change only if latency or stability actually improves.

That sounds slow, but it is faster than changing four settings at once and not knowing which one mattered.

OpenAI-Compatible Client Integration

One of the better reasons to choose vLLM is that you do not need a custom client protocol. The included client.py uses the standard OpenAI Python client and only changes two things:

• the base_url
• the model

That means an internal tool written against OpenAI can usually move to this server with minimal change.

from openai import OpenAI

client = OpenAI(
  base_url="http://127.0.0.1:8000/v1",
  api_key="changeme-local",
)

response = client.chat.completions.create(
  model="Qwen/Qwen3-0.6B",
  messages=[{"role": "user", "content": "Explain vLLM in one paragraph."}],
)

That is a small detail, but it is what makes this setup useful beyond a demo.

Validation

Bring the stack up:

docker compose up -d

Check that the container becomes healthy:

docker compose ps
docker compose logs -f vllm

Validate the health endpoint:

curl http://127.0.0.1:8000/health

Then send one small chat request:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer changeme-local" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Explain what vLLM does in two sentences."}
    ],
    "max_tokens": 120,
    "temperature": 0.2
  }'

Metrics should also be reachable:

curl http://127.0.0.1:8000/metrics | head

If those three checks pass, the deployment is working.

You can add one more check when you start tuning: send the same prompt several times and compare both latency and output behavior. That will tell you more than a single successful request.

If you want dashboards too, start the optional overlay:

docker compose -f docker-compose.yml -f docker-compose.observability.yml up -d

Observability

Prometheus is included because /metrics is one of the easiest ways to understand whether the runtime is actually healthy. This is where the production-oriented articles are useful even if they focus on GPUs. The lesson transfers cleanly: a model server can be technically alive while still being operationally unhealthy.

For this class of machine, these metrics matter most:

• request queue growth
• time to first token
• end-to-end latency
• cache pressure

In practice, if queue depth keeps rising while the API is still “up,” your tuning is too aggressive for the box. That is the normal failure mode on small CPU hosts.

Use these operational readings as a guide:

• rising queue depth with flat CPU availability usually means concurrency is too high
• bad time-to-first-token usually means prompt processing cost is too high for the chosen context or batch settings
• long end-to-end latency with stable health usually means the service is overloaded, not broken
• cache pressure combined with host memory pressure usually means VLLM_CPU_KVCACHE_SPACE or context settings are too ambitious

If you want dashboards, the sample includes an optional Grafana overlay and datasource provisioning. I still would not run it by default on this same 16 GB node unless you actually need the UI. Prometheus alone is the better tradeoff for the base stack.

Security and Exposure

The external production guides make a point that is still true on a home lab: do not treat the model server as the internet-facing edge.

For this sample, that translates into a few concrete rules:

• keep the vLLM API bound to 127.0.0.1
• use --api-key even on local-only deployments
• put a reverse proxy in front if you later need LAN or remote access
• add rate limiting and real authentication at the proxy layer, not inside the model server
• if you need per-user token budgets, enforce them in a gateway layer backed by persistent state such as Redis

For a small internal deployment, --api-key plus localhost binding is enough to start. For anything broader, add an actual gateway.

If You Move to Production on NVIDIA A100

First, one naming detail: an NVIDIA A100 is a datacenter GPU, not a GeForce card. That distinction matters because the production path for vLLM on A100 is very different from the CPU-only home-lab path in this article.

The good news is that the overall service shape still holds. You can keep Docker, keep the OpenAI-compatible API, keep Prometheus, keep your reverse proxy pattern, and keep most of your client code. What changes is the runtime image, the container runtime, the tuning model, and the operational expectations.

What changes immediately

If you want a runnable starting point for that future GPU path, the sample set now includes a separate docker-compose.production.yml file alongside the CPU home-lab Compose file.

That production sample is no longer just “vLLM with a public port.” It uses a layered edge:

• Nginx for connection and request-rate limiting
• a small token-budget gateway for per-user token accounting
• Redis for storing budget counters
• vLLM behind those internal services

For an A100-based production deployment, these parts of the current sample should be replaced:

• replace vllm/vllm-openai-cpu with the CUDA image vllm/vllm-openai
• replace CPU-only scheduling knobs such as VLLM_CPU_KVCACHE_SPACE, VLLM_CPU_NUM_OF_RESERVED_CPU, and VLLM_CPU_OMP_THREADS_BIND
• replace the current conservative CPU batch limits with GPU-oriented tuning
• replace localhost-only direct usage with a proper production edge such as Nginx, Envoy, Traefik, or an API gateway
• add a stateful policy layer if you want per-user token budgets instead of IP-only rate limits

These parts should stay:

• persistent Hugging Face cache
• health checks
• metrics scraping
• API contract
• request logging policy
• prefix caching where it helps your workload

Image and runtime changes

The base image becomes the NVIDIA deployment image described in the vLLM Docker docs:

vllm/vllm-openai:latest

And the container runtime must expose GPUs to the container. In Docker terms, that means using NVIDIA runtime support with GPU access enabled.

At a high level, the current CPU service changes like this:

image: vllm/vllm-openai:latest

And instead of CPU-only assumptions, the deployment needs GPU access. In Docker examples from the vLLM docs, that shows up as --runtime nvidia --gpus all for docker run. In Compose, you would model the same intent with GPU device reservations or the equivalent runtime configuration used by your environment.

Shared memory changes

On the current CPU sample, shm_size: 4g is enough for a tiny model. On A100 production, shared memory becomes more important, especially when you use tensor parallelism or larger models.

The vLLM Docker docs call out two valid approaches:

• ipc: host
• a larger shm_size

For A100 production, ipc: host is the more common choice in examples because PyTorch uses shared memory under the hood, especially for multi-process and tensor-parallel inference.

What replaces the CPU knobs

In the home-lab sample, the important knobs are CPU-specific. On A100, the center of gravity shifts to GPU-specific engine arguments.

These are the most important replacements:

• replace VLLM_CPU_KVCACHE_SPACE with --gpu-memory-utilization
• replace most CPU thread-binding concerns with GPU parallelism choices such as --tensor-parallel-size
• keep --max-model-len, but now treat it as a GPU KV-cache and throughput tradeoff rather than a RAM-survival setting
• keep --max-num-seqs and --max-num-batched-tokens, but tune them for throughput versus latency on GPU instead of “do not overload the CPU”

The production-style GPU flags you would likely introduce include:

• --gpu-memory-utilization
• --tensor-parallel-size
• --max-model-len
• --max-num-seqs
• --max-num-batched-tokens
• --enable-prefix-caching

If you later run very large models or quantized variants, you may also add:

• --quantization
• pipeline parallel or other distributed-serving settings

A100 production profile: practical direction

If you eventually move beyond Qwen/Qwen3-0.6B, an A100 changes your model envelope completely. At that point, the system is no longer constrained by “can this tiny CPU box stay alive?” but by production goals such as:

• target latency
• target throughput
• concurrency under load
• cost per request
• failure isolation

That means your first production tuning loop on A100 should focus on:

1. choosing the real production model size
2. setting --gpu-memory-utilization conservatively at first
3. deciding whether one GPU is enough or whether you need --tensor-parallel-size
4. increasing --max-num-seqs and --max-num-batched-tokens based on real traffic
5. watching p95 and p99 latency, queue depth, and cache usage before pushing throughput harder

What a production replacement looks like conceptually

Here is the important mental model for migration:

• current CPU config optimizes for stability on weak hardware
• A100 config should optimize for measured latency, throughput, and operational safety

So in practice, the production replacement of the current service looks something like this at a high level:

nginx -> token-gateway -> vllm
|
-> redis

And the production Compose file in the sample folder reflects that shape:

• Nginx owns the published port and applies per-IP request and connection limits
• the token gateway uses bearer tokens as user keys
• Redis stores daily token-budget counters
• the gateway forwards accepted requests to vLLM with an internal API key

That gives you two control layers instead of one:

• edge protection for bursty or abusive traffic
• per-user token accounting for quota-style control

What else you should add for real production

Moving to A100 is not only a hardware swap. It is the point where the surrounding platform should become stricter too.

Add these around the current sample shape:

1. a reverse proxy or API gateway in front of vLLM
2. real authentication and rate limiting at the edge
3. TLS termination
4. pinned image tags instead of latest
5. non-root container execution where practical
6. quota enforcement if you need per-user or per-tenant token budgets
7. dashboarding and alerting that track latency, queue depth, GPU cache usage, rate-limit hits, and restart events
8. separate environments for staging and production
9. capacity tests with real prompt distributions before rollout

If you run multiple A100s or larger models, add:

1. tensor parallel planning
2. stronger shared-memory configuration
3. more careful node sizing for CPU, RAM, disk, and PCIe or NVLink topology
4. orchestration beyond single-node Compose if availability matters

What not to carry over unchanged

Do not carry these assumptions from the current sample into A100 production:

• tiny batch sizes by default
• CPU tuning variables
• localhost-only access as your final production edge pattern
• IP-only controls if your real requirement is per-user quota enforcement
• single-node thinking if availability or scale actually matters
• latest tags for long-lived environments

Best migration path from this article

If you use this article as the starting point, the cleanest upgrade path is:

1. keep the API contract and client code
2. replace the CPU image with vllm/vllm-openai
3. add NVIDIA runtime and GPU access
4. switch from CPU knobs to GPU knobs
5. put a real gateway in front of the service
6. add a token-budget layer if you need per-user quotas
7. tune against real production traffic instead of lab prompts

What the sample production gateway now does

The current production sample adds two concrete controls on top of the GPU-ready vLLM service.

First, Nginx rate-limits requests at the edge. In the sample, that means:

• 5 requests per second per client IP
• burst 20
• maximum 10 concurrent connections per client IP

Second, the token gateway enforces a Redis-backed daily budget per bearer token. In the sample, that means:

• the caller sends Authorization: Bearer user-key
• the gateway estimates prompt tokens with the model tokenizer
• it adds requested output tokens such as max_tokens
• it reserves that amount against the user's daily quota
• it returns 429 if the budget is exhausted

The default example limit in the sample is:

• DAILY_TOKEN_LIMIT=200000

The sample also includes protected admin endpoints for checking or resetting a user's daily budget entry through the gateway. Those calls require X-Admin-Key and are useful for operational support or staged rollouts, but they should stay behind trusted network boundaries.

That is why the current home-lab setup is still worth doing carefully. The hardware changes later, but the service boundary and operational habits are still useful.

Why This Sample Does Not Add More Services

You will see richer examples online with Grafana, reverse proxies, load balancers, autoscaling, and multi-node layouts. Those are good references, but they are not the right default for this exact box.

This sample intentionally stops at:

• one inference service
• one metrics scraper
• one small client example

Everything else is optional. That is why Grafana lives in a separate overlay instead of the base stack.

That keeps the node focused on the thing it can realistically do. The moment you add too much supporting infrastructure to a 16 GB machine, you stop measuring the model server and start measuring contention between support services.

Troubleshooting

Native Windows problems

Use Linux or WSL2. Current vLLM guidance is centered on Linux.

Slow or unstable startup

That is usually one of these:

• first model download is still in progress
• model context is too large for the available RAM budget
• batch settings are too high for the host

Start with the conservative sample values and only tune after the first successful run.

If startup repeats the download on every launch, the cache mount is wrong or the cache path is not persisting.

get_mempolicy: Operation not permitted

The CPU docs call out this exact issue in Docker. The sample includes cap_add: SYS_NICE and security_opt: seccomp=unconfined to reduce that problem while avoiding the much broader privileged setting.

Bad latency under load

Lower one or more of these:

• VLLM_MAX_NUM_SEQS
• VLLM_MAX_NUM_BATCHED_TOKENS
• VLLM_MAX_MODEL_LEN

On this machine, lower settings are usually the right answer.

Model works but quality feels weak

That is a model-size limitation, not a container problem. Qwen/Qwen3-0.6B is a pragmatic fit for this host, not a substitute for a larger model on stronger hardware.

Health endpoint works but the service feels overloaded

That usually means the server is alive but over-tuned for the hardware. Watch queue depth and reduce:

• VLLM_MAX_NUM_SEQS
• VLLM_MAX_NUM_BATCHED_TOKENS
• VLLM_MAX_MODEL_LEN

Do not treat a healthy container as proof that the current tuning is good.

Where To Go Next

Once this setup is stable, there are three sensible next steps:

1. Put a reverse proxy in front of the localhost-bound service and add real auth.
2. Move the exact Compose shape to a stronger Linux node with a supported accelerator.
3. Keep the client code and operational flow, but swap the model or image version under controlled tests.

That is the real benefit of doing the small version carefully. You keep the deployment shape even as the hardware changes.

Conclusion

You can run vLLM on a small home-lab machine, but the right way to think about it is not “cheap production LLM serving.” It is “a disciplined CPU-only lab deployment that uses the same API shape and operational patterns you would keep on better hardware later.”

For an Intel Core i5-10400 with 16 GB RAM, Qwen/Qwen3-0.6B is a sensible target. The CPU image, persistent model cache, private port binding, health checks, Prometheus metrics, and conservative tuning values give you a setup that is realistic, teachable, and stable enough to extend.

When you outgrow it, the migration path stays clean: keep the Compose shape, keep the API contract, upgrade the hardware. That is the real win here. The hardware is temporary. The service shape is what lasts.