From 6 Hours to 2: Fixing the Memory Wall in Batch AI Pipelines

Most teams hit the same wall: you're processing tens of thousands of documents through an LLM, the pipeline takes 6+ hours, and someone suggests "just add more GPUs." That's the wrong answer.

The real bottleneck isn't compute — it's the Memory Wall. Your GPUs are sitting idle most of the time, waiting on memory bandwidth. Here's how we cut a 50K daily transcript pipeline from 6 hours to 2 hours while reducing costs by 30%.

The Memory Wall Problem

Auto-regressive generation (how LLMs produce tokens) is fundamentally memory-bound, not compute-bound. Each token generation step reads the entire Key-Value cache from GPU memory, but only performs a tiny matrix operation. Your expensive H100s? They're operating at a fraction of their theoretical FLOPS.

The red box dominates your wall-clock time. More GPUs won't fix a memory bandwidth bottleneck — smarter memory management will.

Optimization 1: Continuous Batching + PagedAttention

The problem with static batching: Every request in a batch must finish before the next batch starts. One long response holds up 63 short ones.

Continuous batching dynamically inserts new requests and evicts completed ones at the iteration level. Combined with PagedAttention — which stores the KV cache in non-contiguous memory pages instead of one giant block — you eliminate memory fragmentation and fit dramatically more concurrent requests.

Impact: Near 100% GPU utilization instead of the 30-40% typical with static batching.

Optimization 2: Speculative Decoding

This is the single biggest win for batch workloads.

The idea: pair your large "target" model with a small, fast "draft" model. The draft model rapidly guesses the next N tokens. The target model verifies all N tokens in a single forward pass — accepting the longest correct prefix.

Why this works mathematically: Verification is parallelizable. Instead of 8 sequential forward passes, you do 1 pass for 8 tokens. The output distribution is identical to the target model — no quality loss.

Optimization 3: Cross-Call KV Cache Reuse

In batch workloads, thousands of prompts share the same system instructions, few-shot examples, or context documents. Without caching, you're recomputing the exact same KV cache for the shared prefix on every single request.

Semantic caching preserves the KV cache of shared prefixes across API calls, completely eliminating redundant prefill computation.

Before:  [System Prompt | Few-Shot | Context | Query A] → full prefill
         [System Prompt | Few-Shot | Context | Query B] → full prefill (again!)

After:   [System Prompt | Few-Shot | Context] → cached once
         [Query A] → only new tokens prefilled
         [Query B] → only new tokens prefilled

When 80% of your prompt is shared boilerplate, this cuts prefill time by ~80%.

The Combined Effect

These three optimizations compound:

The Takeaway

Scaling batch AI pipelines is a memory engineering problem, not a compute scaling problem. Before requisitioning more hardware:

1. Profile your GPU utilization — if it's under 60%, you have a batching problem
2. Measure your acceptance rate for speculative decoding — aim for 70%+ token acceptance
3. Audit your prompt overlap — any shared prefix over 500 tokens is a caching opportunity

The infrastructure layer is where the real leverage lives. Optimize the memory, and the throughput follows.

Building batch AI pipelines or hitting the memory wall? I'd love to compare notes — reach out at hello@sowmith.dev