Making LLMs fast and affordable in production

Two speeds in every request

Give a language model a two-page prompt and it reads all of it in a fraction of a second. Then it takes several more seconds to write a reply shorter than the prompt. That gap explains most of what makes LLM features slow or expensive, and almost every optimization we use in production attacks one side of it.

A request runs in two phases. In prefill, the model takes in the whole prompt at once. All the prompt tokens go through the network together, which keeps the GPU's arithmetic units busy, and the pass ends with the first token of the answer. In decode, the model writes the rest one token at a time. Each new token needs another full pass through the model, and each pass reads every weight from GPU memory to produce that single token. Prefill is limited by compute. Decode is limited by memory bandwidth.¹

So there are two latency numbers, not one. Time to first token is how long the user waits before anything appears, and it grows with the prompt. Time per output token is how fast the answer arrives after that. Total time is roughly the first plus the second multiplied by the number of tokens written.¹ When engineers at Databricks measured this, adding 512 input tokens increased latency less than generating 8 more output tokens.¹ If a reply feels slow, look at the length of the answer before the length of the prompt.

Providers price the same imbalance. Across the current lineups from Anthropic, OpenAI and Google, an output token costs five to eight times as much as an input token.^2,3,4

Two things in that chart shape the rest of this post. The dumbbells are all about the same length, so the output premium is close to an industry constant. And the rows span a range of more than a hundred, so which model you call matters more than almost any other setting.

There is one twist. Most product prompts are long: instructions, tool definitions, retrieved documents, chat history. A support assistant that sends 9,600 tokens of instructions and product docs and gets a 300-token reply sends 32 input tokens for every output token. At Claude Sonnet 5.5's list prices, input is almost nine tenths of that bill.² So the working rule is simple: input tokens set the bill, output tokens set the wait.

Never pay twice for the same prompt

During prefill the model turns every prompt token into keys and values, the intermediate results that attention looks back at. They stay in GPU memory as the KV cache, so decode never has to redo the prompt. Prompt caching keeps that work after the request ends. If the next request starts with exactly the same tokens, the server skips prefill for that part.

The savings are large. On Anthropic's API, reading from the cache costs a tenth of the normal input price on most models and a twentieth on Claude Sonnet 5.5 and Opus 5.5. Writing an entry that lives for five minutes costs 1.25 times the normal price.^2,5 When Anthropic launched the feature, a chat over a 100,000-token book went from 11.5 seconds to first token to 2.4 seconds, at 90 percent lower cost.⁶ OpenAI caches long prompts automatically,⁷ and self-hosted servers such as vLLM do the same with automatic prefix caching.⁸

The catch is the word "starts". The cache matches from the first token forward, and the first difference ends the match. Anthropic's documentation puts it plainly: a hit needs 100 percent identical prompt segments up to the cache breakpoint.⁵ One timestamp at the top of a system prompt makes every request look new.

So we treat prompt order as an engineering decision. Things that never change go first: instructions, tool definitions, reference documents, examples. Things that change go last: the date, the user's name, retrieved snippets, the new question. OpenAI's guide gives the same advice.⁷ Settings count too. On Claude, changing the thinking effort between turns of a conversation invalidates the cached messages.⁹ Every response reports how many input tokens came from the cache, and we watch that number the way we watch an error rate. A harmless-looking prompt edit can quietly turn a 95 percent discount into nothing.

The fastest token is the one you never write

Every output token costs a full pass through the model, so the cheapest optimization is writing fewer of them. Most products can cut a lot without anyone noticing.

• Ask for structure, not prose. A classifier should return a label, not a paragraph explaining it. An extraction step should return the fields you need and nothing else.
• Set a ceiling. A maximum output length caps the worst case, so one runaway answer can't hold a request open for a minute.
• Watch the tokens you don't see. Reasoning models think before they answer, and that thinking is billed as output, even when the API only shows you a summary.⁹ Set the reasoning effort per feature: checking an order status doesn't need the depth of a contract review.

Some of the wait can't be removed. A long answer still takes seconds to write. Streaming doesn't make it faster, but it changes what the user lives through. Jakob Nielsen's limits from 1993 still hold: about one second keeps a person's train of thought, and about ten seconds is as long as attention lasts.¹⁰ A streamed answer starts well inside the first limit. A buffered one can easily cross the second.

Streaming has a cost on the product side. The interface has to render partial text, cope with a stream that dies halfway, and hold back anything that must be checked before it is shown. Our usual pattern is to stream the words and validate the actions: text appears as it is written, but a refund button or a database write appears only after the complete, structured answer has passed its checks.

Send easy questions to small models

Look at the price chart again. Within one provider, the small model costs between a fifth and a hundredth of the large one. And most real traffic is not hard. Order status, opening hours and "how do I reset my password" do not need the most capable model on the market.

There are two common patterns. A router looks at each request and picks a model before answering. A cascade tries the cheap model first and escalates only when its answer fails a check. The research results are encouraging. FrugalGPT matched GPT-4's performance at up to 98 percent lower cost on its benchmarks by cascading models.¹¹ RouteLLM trained routers on human preference data and cut costs by more than half in some settings without hurting response quality.¹²

The hard part is the check. A good one is cheap and boring: does the output parse, does it pass the business rules, does it cite the retrieved sources, did the user ask the same thing again a moment later. None of this works without an evaluation set built from real requests. Without one, routing is a guess, and the savings show up long before the complaints do.

Two more price levers are worth knowing. Work nobody is waiting for, such as nightly summaries, tagging a backlog or re-indexing documents, can go through the batch APIs that Anthropic, OpenAI and Google all offer at half price, in exchange for results that arrive later.^2,3,4 And speed itself is now for sale: Anthropic's fast mode for Claude Opus 5.5 costs twice the normal rate for significantly faster output.² Latency has become a line item you choose.

Share every pass through the model

Everything so far works with any API. The next ideas are what inference providers do behind those prices, and what you need if you host open models on your own GPUs.

Remember that a decode pass reads every weight to write one token. If thirty-two users are each halfway through an answer, the server can run one pass that writes one token for each of them, reading the weights once instead of thirty-two times. This is batching, and it is where most of a server's throughput comes from.¹

The simple version, static batching, groups requests and runs the group until its longest answer is done. Short answers finish early and their slots sit empty. Orca, a research system from 2022, scheduled work per iteration instead: after every pass, finished requests leave and waiting ones take their place. On GPT-3 175B its authors measured 36.9 times the throughput of NVIDIA FasterTransformer at the same latency.¹³ The idea is now called continuous batching, and the major open-source inference servers use it.

Then memory becomes the limit. Every request in the batch keeps its KV cache on the GPU. For Mistral 7B that is 128 KB per token (keys and values, 32 layers, 8 key-value heads of 128 numbers, 2 bytes each),¹⁴ so a 4,000-token conversation holds half a gigabyte. Older servers reserved space for the longest possible answer up front, and the vLLM team found that existing systems wasted 60 to 80 percent of that memory to fragmentation and over-reservation.¹⁵ Their fix, PagedAttention, stores the cache in small fixed-size blocks, like pages in an operating system, and hands them out as tokens arrive. Waste fell under 4 percent, more requests fit in each batch, and throughput rose 2 to 4 times over the previous best systems at the same latency.^15,16

At larger scale, some systems go further and run the two phases on separate GPUs, so a burst of long prompts doesn't stall everyone else's decode. DistServe, which works this way, served up to 7.4 times more requests while meeting the same latency targets.¹⁷

Lighter weights, bolder guesses

The last two ideas attack the decode pass itself. One makes each pass read fewer bytes. The other gets more than one token out of each pass.

Quantization stores the model's weights with fewer bits. Most open models ship with 16-bit weights, so a 70-billion-parameter model needs 140 GB for its weights alone, more than one 80 GB NVIDIA H100 holds.¹⁸ Stored as 8-bit integers plus a scale for each small group of weights, it fits on one GPU. At 4 bits it takes 35 GB and leaves more than half the card for KV cache, which means bigger batches. And since decode is limited by how fast weights stream out of memory, fewer bytes also means faster tokens on the same hardware.

The real question is quality, because every rounding step moves a weight a little. LLM.int8() showed 8-bit inference with no measured loss on models up to 175 billion parameters.¹⁹ At 4 bits naive rounding hurts, so methods like GPTQ²⁰ and AWQ²¹ use a small calibration set to decide how to round. GPTQ reported negligible accuracy loss at 3 to 4 bits and about 3.25 times faster generation on an A100. AWQ found that protecting about 1 percent of the most important weights removes much of the error. In practice the loss depends on the task, so we run our own evaluation set before switching precision, the same way we would before switching models.

Speculative decoding spends the arithmetic that decode leaves idle while it waits on memory. A small draft model guesses the next few tokens. The large model checks all the guesses in a single pass, keeps the run that matches what it would have written, and adds its own token where they part ways.^22,23 Checking four guesses costs little more than writing one token, because the weights stream out of memory once either way.

The output follows the same distribution the large model would produce on its own; only the time changes. The original papers measured 2 to 3 times faster decoding on T5-XXL and 2 to 2.5 times on Chinchilla 70B.^22,23 EAGLE, which drafts using the large model's own internal features, reported 2.7 to 3.5 times on LLaMA2-Chat 70B.²⁴ It helps most when traffic is light and batches are small. Under heavy load the GPU's arithmetic is already busy, which is why vLLM aims it at medium-to-low traffic, memory-bound workloads.²⁵

Put together, the playbook is short. Keep the stable part of every prompt identical and at the front. Write less, and stream what you write. Send easy work to small models and patient work to batch queues. If you run your own GPUs, batch continuously, quantize carefully and let a draft model guess ahead. None of it is exotic. The skill is knowing which side of the request a problem lives on.

A model reads in one pass and writes one token at a time. Make the reading cheap, keep the writing short, and save the big model for the questions that need it.