Creating a model from scratch is a time-consuming process, and we don’t do that every single day. Even if we did, what really changes across models are two things — the data it’s trained on, and the underlying architecture and its functionality, like tool calling, image generation, or whether it’s plain text-to-text or fully multimodal.
What we do often work with as part of our daily tasks is inferencing. Inferencing as a topic is pretty vast, but lately I’ve been working on projects where I have to scale up inferencing for LLMs, and it’s forced me to actually sit down and understand what’s happening under the hood.
To better understand inferencing in LLMs, there are two key terms that need to be understood: Prefill and Decode. These two stages are also essential to understanding the KV cache, or the memory requirement, of a model during generation.
Disclaimer -> This is specific to decoder-based architectures (autoregressive models).
A quick detour: Sampling Parameters
A brief understanding of sampling parameters will also help with the overall picture. For example:
- Temperature -> increases the overall spread of the output distribution and randomness, commonly (and loosely) called “creativity.”
- Top-k -> restricts sampling to only the k most likely next tokens.
- Top-p (nucleus sampling) -> restricts sampling to the smallest set of tokens whose cumulative probability crosses p.
- max_tokens -> caps how many tokens the model is allowed to generate.
Why do we even need this? -> To avoid pure greedy search (always picking the single most probable next token, which makes outputs repetitive and robotic), to give the model some personality/variation in tone, and to stay within safety guardrails.
A bit off-topic, but useful context. Let’s get back to the two stages.
What really happens when we send an input prompt to an LLM
Tokenization -> Each model comes with its own unique tokenizer, which converts the input text into token IDs.
Once tokenized, the request is placed in a waiting queue. There are two types of requests that can exist in this queue:
- Prefill request (waiting request)
- Decode request (running request)
What is Prefill?
The model processes all the prompt tokens together, in one parallel pass. It builds the KV cache and produces the first output token.
The below diagram shows an example ->
The metric to keep in mind here is TTFT (Time To First Token) — this could be something like 42ms, for example, depending on prompt length and hardware.
The KV cache is essentially a set of context vectors stored from each transformer layer, computed during this pass. It exists so the next token generation step can reuse this context instead of recomputing everything the model already worked out once before.
The KV cache is stored in GPU VRAM (in memory regions often referred to as “blocks”).
What is Decode, then?
During decode, the model generates only one new token per pass. That token is fed back into the model as input to generate the next one — this feedback loop is the “autoregressive” part.
The below diagram shows an example ->
The metric to keep in mind here is TPOT (Time Per Output Token) — this could be something like 9ms per token, for example.
.
Put together: total latency for a generation roughly works out to TTFT + (number of output tokens × TPOT). So for a 42ms TTFT and 4 tokens generated at ~9ms each, you’re looking at ~78ms end to end — a good chunk of which comes from decode once you’re generating longer outputs, even though decode is the “lighter” step per token.
Inference Engines
To improve inference speed, or to better optimize it for the GPU, there are different inference engines and serving frameworks out there. Examples: vLLM, SGLang, TensorRT-LLM, Triton Inference Server, etc.
Maybe that’s a topic for another post.
Hope to deep-dive into inference engines like vLLM and SGLang soon. I’ll also share a detailed post on Transformers and the attention mechanism soon.