Serving an LLM, End to End — Ruthvik Garlapati

Stage 1 · Hugging Face Hub · lives on the internet

The model is a folder of files.

A model on the Hugging Face Hub is a Git repository. The large files are stored outside Git history and fetched on demand, so you can download one file without cloning everything. For a typical instruction-tuned model the folder holds:

config.jsonThe architecture: number of layers, attention heads, KV heads, hidden size, vocabulary size and maximum position. The serving engine reads this first to build an empty model of the right shape.

model-00001-of-00004.safetensorsThe weights, split into shards of a few GB each. Safetensors is a flat format of raw tensors plus a small header, so it loads fast with memory mapping and cannot run code when opened, unlike older pickle files.

model.safetensors.index.jsonA map from every tensor name to the shard that holds it.

tokenizer.json tokenizer_config.jsonThe vocabulary and rules that turn text into token IDs. tokenizer_config.json usually carries the chat template, which formats a list of messages into the exact prompt the model was trained on.

generation_config.jsonDefault sampling settings and the end-of-sequence token IDs that tell the server when to stop.

Gated models

Some repos require you to accept a license first. Downloads then need an access token, which in a cluster belongs in a Kubernetes Secret, never in an image or a values file.

Pin a revision

Repos change. Record the commit hash you tested and download that exact revision, so a re-download next month gives you the same weights and the same chat template.

Stage 2 · Sizing · happens on paper

Do the memory math before you pick a GPU.

Two things compete for GPU memory (VRAM): the weights, which are a fixed size, and the KV cache, which grows with every token of every active conversation.

Weights

parameters × bytes per parameter

8B params × 2 bytes (BF16) ≈ 16 GB. FP8 halves that; 4-bit formats such as AWQ or GPTQ bring it near a quarter, plus a little for scale factors.

KV cache per token

2 × layers × KV heads × head dim × bytes

Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB per token. An 8K-token conversation holds 1 GiB.

The 2 is for one key and one value vector per layer. Models with grouped-query attention share each KV head across several query heads, which is why Llama 3.1 8B stores 8 KV heads instead of 32. If the weights don't fit on one GPU, you split them across several with tensor parallelism (stage 6). Try the numbers in the memory calculator.

Stage 3 · Fetch · Hub → cluster storage

Download once, mount everywhere.

Weights for a 70B model are about 140 GB. You don't want every pod downloading that from the internet on every restart, so the usual pattern is to download once into storage the cluster can mount.

Download Job into a volume

A one-off Kubernetes Job runs the Hugging Face CLI and writes into a PersistentVolumeClaim. With a ReadWriteMany volume (NFS, EFS, Filestore, Lustre) every replica mounts the same copy read-only.

Mirror to object storage

Copy the repo to your own S3 or GCS bucket. Pods or an init container pull from there, inside your network, with no dependency on the Hub at serve time.

Init container per pod

Each pod downloads into its own disk before the server starts. Simple, but slow to scale and every new replica pays the full download.

Bake weights into the image

Works for small models, but a 100 GB image is slow to push, slow to pull, and ties every code change to a weight re-upload.

# Kubernetes Job: download a pinned revision into the shared volume
apiVersion: batch/v1
kind: Job
metadata:
  name: fetch-llama-3-1-8b
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: fetch
        image: python:3.12-slim
        command: ["sh", "-c"]
        args:
        - pip install -U huggingface_hub &&
          hf download meta-llama/Llama-3.1-8B-Instruct
            --revision $REVISION
            --local-dir /models/llama-3.1-8b-instruct
        env:
        - name: REVISION
          value: "<commit-sha-you-tested>"
        - name: HF_TOKEN            # gated model: token from a Secret
          valueFrom:
            secretKeyRef: { name: hf-token, key: token }
        volumeMounts:
        - { name: models, mountPath: /models }
      volumes:
      - name: models
        persistentVolumeClaim: { claimName: models-rwx }

Older tooling uses huggingface-cli download, which takes the same arguments. Set HF_HUB_OFFLINE=1 in the serving pod so it never tries to reach the Hub at startup.

Stage 4 · Package · container registry

The image holds the engine, not the model.

The container image packages the serving engine (vLLM, SGLang, TensorRT-LLM or Text Generation Inference), Python, PyTorch and the CUDA user-space libraries. The weights stay on the volume from stage 3.

Serving engineAPI server, scheduler, KV-cache manager, model codeimage
PyTorch + CUDA librariescuBLAS, cuDNN, NCCL, attention kernelsimage
NVIDIA Container ToolkitInjects the GPU device files and driver libraries into the containernode
NVIDIA kernel driverTalks to the physical GPU. Must be new enough for the CUDA version in the imagenode

The split matters: the image brings its own CUDA runtime, but the driver comes from the host. An image built for a newer CUDA than the node's driver supports fails at startup with a driver-version error. Most teams start from the engine's official image, pin the exact tag, scan it, and push it to a private registry close to the cluster. Engine images are several GB, so pre-pulling them onto GPU nodes cuts startup time.

Stage 5 · Deploy · Kubernetes control plane

What the cluster needs before a pod can use a GPU.

GPU nodes

Worker machines with GPUs, usually in their own node pool with a taint so only pods that tolerate it land there.

NVIDIA GPU Operator

Installs and manages the driver, container toolkit, device plugin, DCGM exporter and feature discovery on every GPU node.

Device plugin

Advertises each GPU to the kubelet as a countable resource, nvidia.com/gpu. The scheduler only places a pod where enough are free.

DCGM exporter

Publishes GPU utilization, memory, temperature, power and error counters as Prometheus metrics.

Deployment or StatefulSet

Runs the serving pods. Multi-node models use a LeaderWorkerSet or similar so one leader and its workers start and fail together.

Service + Ingress or Gateway

The Service gives the pods one stable address. The Ingress or Gateway terminates TLS, checks auth and routes outside traffic in.

# The parts of the serving Deployment that matter
spec:
  template:
    spec:
      tolerations:
      - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
      containers:
      - name: vllm
        image: registry.example.com/vllm-openai:<pinned-tag>
        args: ["--model", "/models/llama-3.1-8b-instruct",
               "--served-model-name", "llama-3.1-8b",
               "--max-model-len", "8192",
               "--gpu-memory-utilization", "0.90"]
        env: [{ name: HF_HUB_OFFLINE, value: "1" }]
        ports: [{ containerPort: 8000 }]
        resources:
          limits: { nvidia.com/gpu: 1 }
        startupProbe:                  # loading can take minutes
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 60
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
        volumeMounts:
        - { name: models, mountPath: /models, readOnly: true }
        - { name: shm, mountPath: /dev/shm }   # shared memory for multi-GPU workers
      volumes:
      - name: models
        persistentVolumeClaim: { claimName: models-rwx }
      - name: shm
        emptyDir: { medium: Memory, sizeLimit: 8Gi }

Helm, or not?

Helm turns these manifests into a template with a values file: model path, GPU count, replicas, max context. It pays off once you run several models or environments. For one model, plain manifests with Kustomize overlays are easier to read. Either way, an ArgoCD or Flux setup can sync them from Git.

Higher-level options

KServe, Ray Serve and the vLLM production stack wrap the same pieces with routing, autoscaling and model registries built in. They save work but add components you have to operate.

Stage 6 · Load · GPU node → VRAM

Startup: from files on disk to a warm GPU.

Schedule. The scheduler finds a node with a free nvidia.com/gpu that tolerates the taint.
Pull and mount. The kubelet pulls the image (skipped if cached) and mounts the model volume read-only.
Build the model. The engine reads config.json and creates every layer on the GPU with empty memory.
Stream the weights. Each safetensors shard is memory-mapped from disk, passes through host RAM and is copied over PCIe into VRAM. With tensor parallelism each GPU loads only its slice of every matrix.
Profile. The engine runs a dummy batch at the maximum size to measure peak activation memory.
Allocate the KV cache. Whatever is left under the memory limit (for example 90% of VRAM) becomes a pool of fixed-size KV blocks. The engine logs how many tokens fit.
Capture CUDA graphs. Common batch shapes are recorded once so later decode steps launch with less CPU overhead.
Go ready. /health starts returning 200, the readiness probe passes, and the Service starts sending traffic.

Load time is usually limited by storage read speed, not the GPU. A network volume that reads at 500 MB/s needs over four minutes for 140 GB, which is why fast shared filesystems or local NVMe caches matter for scaling.

Stage 7 · Serve · one request, end to end

Follow a request through the cluster.

The request travels left to right, the reply streams back token by token. Select any component to see what it does.

Request inTokens streaming out

The call

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role": "user", "content": "What is a KV cache?"}],
    "max_tokens": 200,
    "stream": true
  }'

The reply, as server-sent events

data: {"object":"chat.completion.chunk",
       "choices":[{"delta":{"role":"assistant"}}]}

data: {"object":"chat.completion.chunk",
       "choices":[{"delta":{"content":"A"}}]}

data: {"object":"chat.completion.chunk",
       "choices":[{"delta":{"content":" KV"}}]}
…
data: {"choices":[{"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Prefill

Read the whole prompt at once

All prompt tokens go through the model in one parallel pass. This fills the KV cache for every prompt token and produces the first output token. It is compute-heavy, and its length sets the time to first token (TTFT).

Decode

One token per step

Each step feeds only the newest token, reads every weight and the whole KV cache, and produces one more token. It is limited by memory bandwidth, and its speed sets the inter-token latency. Batching many conversations into the same step is what makes serving affordable.

Inside stage 7 · KV cache

Why the GPU remembers every token.

Attention compares the newest token with every earlier token. The keys and values of earlier tokens never change, so the engine stores them instead of recomputing them each step. vLLM's PagedAttention keeps them in fixed blocks, here 16 tokens each, so memory is handed out a block at a time, the way an operating system hands out memory pages.

Conversation A Conversation B Conversation C Shared prefix Free block

Step: 0
Tokens cached: 0
Blocks in use: 0 / 48
Memory (Llama 3.1 8B): 0 MiB

Press play to start.

Prefix caching

Conversations that start with the same system prompt can point at the same blocks. B reuses A's first two blocks instead of computing them again.

When the pool runs out

The scheduler pauses the newest conversation and frees its blocks, then recomputes them later. Users see this as a latency spike, and num_requests_waiting climbs.

Making it smaller

An FP8 KV cache halves memory per token. Grouped-query attention, already built into most recent models, divides it by the number of query heads per KV head.

Memory · VRAM, host RAM, disk

Where every byte lives.

GPU memory (VRAM / HBM)24 to 141 GB per GPU · hundreds of GB/s to several TB/s

Holds weights, KV cache, activations and CUDA graphs. Decode speed depends on how fast weights stream out of here every step.

Host RAMhundreds of GB per node · tens of GB/s to the GPU

Staging area while loading weights, page cache for memory-mapped shards, and optional overflow space for paused KV blocks.

/dev/shmRAM-backed folder in the pod

Lets worker processes on the same node pass tensors to each other. Docker's 64 MB default is too small for multi-GPU serving.

Disk and network volumeshundreds of MB/s to several GB/s

Holds the safetensors files. Only read at startup, but its speed decides how fast a new replica comes up.

Will it fit? estimate

WeightsRuntime overheadKV cache neededSpare

Per GPU, with the engine allowed 90% of VRAM and a flat 2 GB allowance for activations and CUDA graphs. Real engines measure this at startup and log the exact KV capacity, so treat these numbers as a first pass.

When one GPU is not enough · NVLink, RDMA

How GPUs talk to each other.

With tensor parallelism every layer is split across GPUs, and they exchange partial results (an all-reduce, run by the NCCL library) twice per layer, on every token. That traffic has to be fast.

RDMA

Remote direct memory access lets a network card read or write another machine's memory without either CPU copying the data. InfiniBand does it natively; RoCE does it over Ethernet.

In Kubernetes

The NVIDIA Network Operator installs NIC drivers and exposes RDMA devices to pods, often as a second network attached with Multus. NCCL then picks the fastest path it can find.

Rule of thumb

Keep tensor parallelism inside one NVLink-connected node. Split across nodes with pipeline parallelism, which sends far less data per token, or run more replicas instead.

Stage 8 · Test · before and after traffic

Tests that catch real serving failures.

Hardware and cluster

GPU health: nvidia-smi shows every GPU; a DCGM diagnostic run passes.
Interconnect: nccl-tests all-reduce reaches expected bandwidth over NVLink and RDMA.
Scheduling: a pod requesting nvidia.com/gpu lands on a GPU node and nowhere else.

Smoke

/health returns 200 and /v1/models lists the served name.
One short chat completion returns text and a finish_reason.
Streaming ends with data: [DONE].

Correctness

The chat template renders a known conversation into the exact expected prompt.
Golden prompts with greedy decoding match a reference run.
After quantizing, an eval suite such as lm-evaluation-harness scores within your tolerance.
A prompt at full max-model-len succeeds; one token over is rejected cleanly.

Performance

Sweep concurrency and record TTFT, inter-token latency (p50, p95, p99) and tokens per second.
Use realistic prompt and output lengths, not one fixed prompt.
Find the concurrency where latency breaks your target. That is a replica's real capacity.

Resilience

Kill a pod mid-stream: the client gets an error, the replica comes back, nothing else fails.
Drain a GPU node: a PodDisruptionBudget keeps enough replicas serving.
Soak at steady load for hours to catch memory growth and slowdowns.

Rollout

Send a small share of traffic to the new version (canary) and compare error rates and latency.
Roll back automatically when they regress.

Stage 9 · Operate · Prometheus, Grafana, autoscaler

Watch the queue, not the CPU.

CPU usage says little about an LLM server. Scale and alert on the signals that track user experience:

vllm:num_requests_waiting

Requests queued for a slot. The best single autoscaling signal, used through KEDA or a custom-metrics HPA.

vllm:num_requests_running

Conversations in the current batch.

KV-cache usage

Near 100% means new requests wait or get paused.

vllm:time_to_first_token_seconds

A histogram. Alert on its p95 against your target.

DCGM GPU metrics

Memory, utilization, temperature, power and XID errors that point at hardware faults.

A new replica takes minutes to become ready (image pull, weight load, warm-up), so scale out early, keep a minimum replica count, and treat scale-to-zero as a trade of cost for a slow first request.

Go deeper

Commands and flags follow vLLM and Hugging Face conventions at the time of writing. Check the docs for the versions you run.

From a model repo to a streaming token.