Drivers & runtime
The driver exposes the GPU to the host. A compute platform such as CUDA provides APIs and libraries; kernels are the programs that actually run on the device.
LITTLE BY LITTLE / THE HARDWARE EDITION
Meet the parts.
Follow the token.
A GPU is one member of a much bigger team. Explore the silicon, memory, connections, and supporting equipment that turn a language model into a running service.
26 illustrated components4 layers1 connected picture
01 / FOLLOW A REQUEST
Where does the work happen?
02 / OPEN THE SYSTEM
From the chip to the room.
Choose a layer, then a component. These are conceptual groups, not a wiring diagram or a universal parts list. NVIDIA terms illustrate common accelerator concepts; details vary by vendor and generation.
03 / CONNECT CAUSE AND EFFECT
What changes when…
These are places to investigate, not diagnoses from a single symptom. Benchmark the model, precision, prompt length, output length, concurrency, and latency target together.
04 / THE SOFTWARE BRIDGE
Hardware needs an organizer.
Serving engine
An engine such as vLLM schedules requests, batches work, and manages KV-cache memory. More efficient scheduling can make better use of the same hardware.
Communication & monitoring
Libraries such as NCCL coordinate supported GPU collectives. Metrics for memory, queueing, latency, errors, power, and temperature help explain where work is waiting.
05 / YOUR TURN
Which component would you inspect?
Longer conversations cause GPU out-of-memory errors, while the model weights stay the same size.
Keep exploring.
This guide covers the main parts relevant to LLM serving, rather than every circuit or vendor accessory. Always use the exact platform’s supported parts and topology documentation.