What we do often work with as part of our daily tasks is inferencing. Inferencing as a topic is pretty vast, but lately I’ve been working on projects where I have to scale up inferencing for LLMs, and it’s forced me to actually sit down and understand what’s happening under the hood.

To better understand inferencing in LLMs, there are two key terms that need to be understood: Prefill and Decode. These two stages are also essential to understanding the KV cache, or the memory requirement, of a model during generation.

Disclaimer -> This is specific to decoder-based architectures (autoregressive models).

A quick detour: Sampling Parameters

A brief understanding of sampling parameters will also help with the overall picture. For example:

• Temperature -> increases the overall spread of the output distribution and randomness, commonly (and loosely) called “creativity.”
• Top-k -> restricts sampling to only the k most likely next tokens.
• Top-p (nucleus sampling) -> restricts sampling to the smallest set of tokens whose cumulative probability crosses p.
• max_tokens -> caps how many tokens the model is allowed to generate.

Why do we even need this? -> To avoid pure greedy search (always picking the single most probable next token, which makes outputs repetitive and robotic), to give the model some personality/variation in tone, and to stay within safety guardrails.

A bit off-topic, but useful context. Let’s get back to the two stages.

What really happens when we send an input prompt to an LLM

Tokenization -> Each model comes with its own unique tokenizer, which converts the input text into token IDs.

Once tokenized, the request is placed in a waiting queue. There are two types of requests that can exist in this queue:

• Prefill request (waiting request)
• Decode request (running request)

What is Prefill?

The model processes all the prompt tokens together, in one parallel pass. It builds the KV cache and produces the first output token.

The below diagram shows an example ->

The metric to keep in mind here is TTFT (Time To First Token) — this could be something like 42ms, for example, depending on prompt length and hardware.

The KV cache is essentially a set of context vectors stored from each transformer layer, computed during this pass. It exists so the next token generation step can reuse this context instead of recomputing everything the model already worked out once before.

The KV cache is stored in GPU VRAM (in memory regions often referred to as “blocks”).

What is Decode, then?

During decode, the model generates only one new token per pass. That token is fed back into the model as input to generate the next one — this feedback loop is the “autoregressive” part.

The below diagram shows an example ->

The metric to keep in mind here is TPOT (Time Per Output Token) — this could be something like 9ms per token, for example. . Put together: total latency for a generation roughly works out to TTFT + (number of output tokens × TPOT). So for a 42ms TTFT and 4 tokens generated at ~9ms each, you’re looking at ~78ms end to end — a good chunk of which comes from decode once you’re generating longer outputs, even though decode is the “lighter” step per token.

Inference Engines

To improve inference speed, or to better optimize it for the GPU, there are different inference engines and serving frameworks out there. Examples: vLLM, SGLang, TensorRT-LLM, Triton Inference Server, etc.

Maybe that’s a topic for another post.

Hope to deep-dive into inference engines like vLLM and SGLang soon. I’ll also share a detailed post on Transformers and the attention mechanism soon.

But inside companies, the conversation is shifting from “Can we build it?” to “Who’s going to own it when it goes wrong?”

Because deploying AI at scale is not like shipping a feature. When AI fails, it does not just return an error—it can break your SLA, your compliance posture, or your customer’s trust. And regulators are now asking companies to prove how they govern AI systems, not just whether they work on demo day.

A Real Problem: Processing 3 Lakh Videos a Day

Let’s make this concrete.

You need to process 300,000 videos per day. That is about 3.5 videos per second (300k ÷ 86,400 seconds). Sounds manageable—until you realize this isn’t a “write some Python” problem. It’s a Big Data Architecture problem.

The first question everyone asks: “Which AI model should I use?”

But the real questions are:

• Streaming or batch?
• Who decides the trade-offs?
• What happens when the system is wrong?
• How much does the company lose?

CCTV vs. Upload — Two Different Architectures

Here’s where design judgment matters.

Scenario 1: Live CCTV Monitoring (Security, Traffic, Proctoring)

You need real-time answers. A person fell—alert security now.

→ Use streaming: RTSP/WebRTC → Media Server → Kafka → AI Workers

Scenario 2: User Uploads (Insurance Claims, Medical Records)

Videos arrive in bursts. Users don’t expect instant answers.

→ Use batch processing with presigned URLs: Client uploads → S3 → SQS trigger → GPU workers pull from queue

The queue is your shock absorber. When 10,000 uploads land at once, your system bends instead of breaking.

Can AI Decide This for You?

No.

AI can’t tell you whether to use streaming or batch. It can’t weigh your latency SLA against your budget. It can’t know whether your compliance team will accept a 2% false positive rate.

AI can write the code. It cannot design the system.

Architecture decisions require judgment about trade-offs: cost vs. speed, explainability vs. accuracy, risk vs. scale. These are business and operational decisions, not just technical ones. And when things go wrong—when the model drifts, when a critical edge case fails—someone human has to own the consequences.

This is why system design remains a core competency, even as AI automates more of the coding.

Two Techniques That Change the Economics

If you try to run AI on every pixel of 300,000 videos, you’ll burn your GPU budget and still miss your SLA.

The secret is filtration, not brute force.

1. Keyframe Extraction

Process only the frames that change (e.g., scene cuts, motion events).

• A 1-minute video has ~1,800 frames
• Extract just 5 keyframes → 99% reduction in compute cost

2. Cascade Filtering

Don’t send everything to your expensive model. Filter in stages:

But which technique should you use?

AI can’t answer that. You need to know:

• What are you detecting? (anomalies, people, weapons, text?)
• What’s the cost of a false negative? (safety, money, reputation?)
• What’s your latency budget?

That decision is architecture judgment, not prompt engineering.

The “AI Goes Wrong” Question (The One Nobody Wants to Own)

This is where the real heat shows up. When your AI makes a mistake at scale:

• Who decides whether to trust the output: engineering, product, compliance, or the customer?
• What’s the rollback plan when the model drifts?
• What’s the cost of false positives vs. false negatives—in rupees, time, brand, and legal exposure?

My Takeaway

New models (from Anthropic and others) are making it easier to start. But the hard problems are still human problems:

• Who decides?
• What if it goes wrong?
• How much do we lose?

AI can automate the coding. It can suggest optimizations. But it can’t make the architectural trade-offs or carry the accountability.

Designing these systems will remain a core competency—maybe not the coding, but the judgment.