In today's newsletter:

• Markov decision processes and value functions in RL.
• How LLM inference works internally.

TODAY'S ISSUE

AI engineering

​Markov decision processes and value functions in RL​

Last week, we launched a hands-on course series on reinforcement learning.

​Part 2 is now available, and you can read it here →​

Part 1 gave you the RL interaction loop and the exploration-exploitation tradeoff through bandits, and this one gives you the formal language that every RL algorithm is built on.

It covers:

• the Markov property and why it makes RL tractable
• the MDP as a 5-tuple (states, actions, transitions, rewards, discount factor)
• episodic vs. continuing tasks
• returns and discounting with concrete numerical examples
• the reward hypothesis and its limits (including reward hacking),
• deterministic and stochastic policies, state-value functions
• and a complete hands-on implementation of Monte Carlo policy evaluation on a 4×4 gridworld.

Everything is covered from scratch, so no RL background is required.

​You can read Part 2 of the course here →​

Why care?

Every frontier LLM released in the past two years uses RL in its post-training pipeline.

• ChatGPT was shaped by RLHF.
• DeepSeek-R1 used GRPO to develop reasoning capabilities.
• Claude uses constitutional AI with RL.

The pattern is consistent: pre-training gives the model knowledge, but RL is what gives it behavior.

And it is not just LLMs.

Agentic AI systems that take actions, interact with tools, and operate in multi-step environments are fundamentally RL problems. Robotics, recommendation systems, game-playing, autonomous driving, drug discovery: RL is the common thread.

Google Trends for “reinforcement learning” was nearly flat from 2004 to 2023. In the past year, it hit an all-time high.

The field is having its moment, and the demand for engineers who understand it deeply is growing fast.

This series builds that understanding from the ground up, concept by concept, with math where it matters and hands-on code you can run. No prior RL background needed.

This series is structured the same way as our MLOps/LLMOps course: concept by concept, with clear explanations, diagrams, math where it matters, and hands-on implementations you can run.

And no prior RL background is needed.

If you haven’t read Part 1, start there first. It covers the agent-environment loop, exploration vs. exploitation, and multi-armed bandits.

Over to you: What topics would you like us to cover in this RL series?

LLms

​How LLM inference works internally​

Every generate() call to an LLM runs two distinct computational phases on the same GPU:

• prefill (processing the prompt) is compute-bound
• while decode (generating tokens one at a time) is memory-bound.

Most inference optimizations target one phase or the other, and diagnosing which phase is the bottleneck is the first step in making a deployment faster.

Today, let's walk through the full pipeline, from tokenized input to streamed output, and look at where the time goes in each phase.

Tokenization and embedding

Tokenizers like Byte Pair Encoding (BPE) convert raw text into integer IDs from a vocabulary of roughly 50,000 tokens.

Each ID maps to a row in the embedding table, a learned matrix of shape [vocab_size, hidden_dim]. For a model with a hidden dimension of 4,096, each token becomes a 4,096-dimensional vector.

Position information gets injected at this stage.

Most modern architectures use Rotary Position Embeddings (RoPE), which encode position by rotating the embedding vectors rather than adding a separate positional vector.

Transformer layers

The embedded sequence passes through a stack of transformer layers (typically 32 to 80+, depending on model size).

Each layer applies two operations in sequence:

1) Self-attention computes three projections per token (query Q, key K, value V) via learned weight matrices.

Each token's query is scored against every other token's key, and those scores (after scaling and softmax) determine how much of each token's value gets mixed in.

2) Feed-forward network (FFN) processes each token's vector independently through a two-layer MLP. Attention moves information between positions. The FFN transforms it.

After the final layer, the model projects the last token's hidden state back to vocabulary size ([hidden_dim, vocab_size]), applies softmax, and samples from the resulting distribution to produce the first output token.

Prefill: the compute-bound phase

Processing the input prompt is the first phase. All tokens are processed in parallel: Q, K, and V are computed for every token simultaneously, and attention runs as a large matrix-matrix multiplication.

This is compute-bound work. The GPU's arithmetic throughput is the bottleneck, and utilization is high. The metric that captures this phase is Time to First Token (TTFT), the latency before the first output token appears.

During prefill, the model also populates the KV cache: the K and V tensors for every layer get stored in GPU memory for reuse.

Decode: the memory-bound phase

Once the first token is generated, the model switches to generating one token at a time. For each new token, it only computes Q, K, and V for that single token. The K and V from all previous tokens are already in the cache.

The arithmetic per step is tiny (one query vector against the cached key matrix instead of a full matrix-matrix multiply). But the GPU still loads every weight matrix and the entire cached K/V from memory for that small computation. The bottleneck flips from compute to memory bandwidth.

The metric for this phase is Inter-Token Latency (ITL): the time between consecutive output tokens. Low ITL is what makes a model feel responsive.

The KV cache

Without caching, generating a 1,000-token response would require recomputing attention over the entire growing sequence at every step, giving quadratic complexity.

The KV cache stores each layer's K and V tensors once and appends new entries incrementally.

The video below depicts LLM inference speed with vs. without KV caching:

The speedup is roughly 5x or more for long generations.

The cost is that the cache grows linearly with sequence length and exists per-layer. For a 13B-parameter model, the cache consumes roughly 1 MB per token. A 4K-token context burns through 4 GB of VRAM on the cache alone.

This is why long contexts get expensive. The cache competes directly with batch size for GPU memory, i.e., more cache per request means fewer concurrent requests per GPU.

Standard mitigations include quantizing the cache to INT8 or INT4, sliding window attention (dropping tokens outside a fixed window), grouped-query attention (GQA, sharing K/V across attention heads to reduce the number of cached tensors), and PagedAttention (the memory management trick behind vLLM that pages the cache like an OS pages virtual memory, eliminating fragmentation).

Frontier: redesigning attention around the cache

Quantization and paging treat the KV cache as a fixed cost to manage. DeepSeek's V4 series (released April 2025) takes a different approach: redesign attention so the cache is structurally smaller from the start.

V4 uses a hybrid of two compressed attention mechanisms.

Compressed Sparse Attention (CSA) compresses KV entries by 4x using softmax-gated pooling, then applies sparse attention over the compressed tokens.

Heavily Compressed Attention (HCA) is more aggressive. It consolidates KV entries across 128 tokens into a single compressed entry and applies dense attention over those representations.

At a 1M-token context, V4-Pro requires 27% of the single-token inference FLOPs and 10% of the KV cache compared to DeepSeek-V3.2.

In absolute terms, that's 9.62 GiB of KV cache per sequence at 1M context in bf16, compared to an estimated 83.9 GiB for a V3.2-style architecture. With fp4/fp8 quantization on top, the cache shrinks by another 2x.

The KV cache has become the constraint the field is optimizing the model architecture around. When attention itself gets redesigned to minimize cache footprint, the bottleneck has shifted from "how to serve the model" to "how to design the model for serving."

​Quantization​

Training uses FP32 or BF16 for gradient stability. Inference doesn't need that precision. The memory savings from reducing bit width are linear:

• 7B parameters at FP32: 28 GB
• 7B parameters at FP16/BF16: 14 GB
• 7B parameters at INT8: 7 GB
• 7B parameters at INT4: 3.5 GB

INT4 is why 7B models run on laptop GPUs with 4-6 GB of VRAM. Methods like GPTQ and AWQ use per-channel scaling factors to minimize quality degradation from the lossy compression.

Done well, INT4 lands within 1-2 percentage points of the full-precision model on standard benchmarks.

Going from FP16 to INT8 often cuts inference latency in half with negligible quality loss, making quantization the single highest-leverage optimization for most deployments.

Serving infrastructure

Modern inference servers wrap the prefill-decode loop with several optimizations:

• ​Continuous batching interleaves tokens from multiple requests on the same GPU step, keeping utilization high even during memory-bound decode phases.
• ​Speculative decoding uses a small draft model to propose multiple tokens, then the large model verifies them in a single forward pass. When the draft model's acceptance rate is high, this effectively converts multiple sequential decode steps into one parallel verification.
• ​PagedAttention (vLLM) manages KV cache memory in fixed-size blocks, eliminating fragmentation and enabling more concurrent requests per GPU.

Frameworks like vLLM, TensorRT-LLM, and Text Generation Inference (TGI) combine these techniques. A single GPU can serve dozens of concurrent users because decode leaves most of the arithmetic capacity idle, and continuous batching fills that idle capacity with other requests.

Putting it together

The full inference path:

1. Tokenize: Text becomes integer IDs via BPE.
2. Embed: IDs become vectors. RoPE encodes position.
3. Prefill: All input tokens processed in parallel through every layer. Compute-bound. KV cache populated. First token emitted.
4. Decode loop: One token per step: project Q for the new token, attend over cached K/V, run FFN, sample. Append new K/V to cache. Memory-bound.
5. Detokenize: Token IDs mapped back to text and streamed.

Some practical implications:

• long prompts are expensive in TTFT (prefill)
• long outputs are expensive in ITL (decode)
• and they stress different hardware resources.
• Context length isn't free because it bloats the KV cache and directly reduces batch capacity.
• GPU utilization during decode can drop to 30% even on a fully loaded server, because the bottleneck is memory bandwidth, not arithmetic.
• The fix isn't more compute, it's faster memory, a smaller cache, or better batching.

When someone tells you their model is slow, the first diagnostic is whether it's slow to start (prefill-bound, optimize TTFT) or slow to stream (decode-bound, optimize ITL).

​To master the full LLMOps cycle with code, start here →​

We published the above LLMOps course, which covers the fundamentals of AI engineering & LLMs, Building blocks of LLMs like tokenization, embeddings, attention, architectural designs and training, decoding, generation parameters, the LLM Application Lifecycle, context engineering, prompt management, defense, control, memory, temporal context, evaluation, tool use, red teaming, Adaptive LLMs, and Serving.

👉 Over to you: are you running into TTFT or ITL bottlenecks in your deployments, and what's worked for you?

THAT'S A WRAP

Partner with US

ADVERTISE TO 950k+ AI Professionals

Our newsletter puts your products and services directly in front of an audience that matters, including thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.

Get in touch today by replying to this email.

Today’s email was brought to you by Avi Chawla and Akshay Pachaar.

​Update your profile | Unsubscribe​

Looking for more? Unlock our premium DS/ML resources.

© 2026 Daily Dose of Data Science