LLM Production KPIs: A Practical Guide to Latency, Throughput, vLLM, and Cost Efficiency

KPI clusters for LLM production systems are the small family of numbers — latency, throughput, scalability, cost, energy, resource limits, queueing, integration overhead, reliability, and security overhead — that together tell you whether a deployed model is actually working, not just whether it "seems fine" in a demo. Each cluster answers a different question a demo never has to answer: how long does one person wait, how many people can wait at once, what does waiting cost in dollars and watts, and what happens when something upstream breaks. 📊

This matters because a model that answers beautifully in a notebook can still fail a business the moment real traffic hits it. A support bot with a lovely 2-second average latency can still make users furious if its 99th-percentile latency is 40 seconds during a Monday-morning spike. A summarizer that costs $0.002 per call in testing can quietly become a seven-figure line item once it runs on every inbound email. The KPI clusters below are the instruments that catch these problems before a customer, a CFO, or a regulator does. ⚠️

Original diagram: the twelve gauges a production LLM team actually watches.

📑 In This Post

1. Latency — the wait a human actually feels
2. Latency Decomposition — where the milliseconds hide
3. Throughput — how much work the fleet gets done
4. Scalability — growing without falling over
5. Cost Efficiency — the bill behind the answer
6. Energy Use — the watts behind the tokens
7. Resource-Constrained Performance — running on less
8. Queue Management — who waits, and for how long
9. Integration Overhead — the cost of talking to everything else
10. Reliability — staying up when it matters
11. Security Overhead — the guardrail tax
12. Speed vs. Accuracy vs. Perception — the three-way tug of war
13. Emerging in 2026 — what's joining the KPI panel next
14. Rolling This Out at Enterprise Scale
15. Common Mistakes
16. FAQ
17. References & Further Reading
18. Summary

🔀 Quick Comparison: Training-Time Metrics vs. Inference-Time KPIs

Every cluster in this post is an inference-time KPI — it describes the live, serving system, not the training run. Teams that only watch training-time metrics are flying blind in production, which is exactly the mistake this table is meant to head off.

Dimension	Training-Time Metric	Inference-Time KPI (this post)
Speed	Steps/sec, epoch time	TTFT, TPOT, end-to-end latency
Volume	Tokens processed per training run	Requests/sec, tokens/sec under load
Spend	GPU-hours to convergence	Cost per request, cost per judge call
Failure mode	Loss spikes, divergence	Timeouts, queue backlog, guardrail false positives
Who owns it	ML research / training infra	SRE / platform / applied ML together

1. Latency — The Wait a Human Actually Feels

📌

Kid analogy: imagine ordering food at a counter. Latency isn't just "how long until the whole tray arrives" — it's also "how long until someone even looks up and says 'coming right up.'" A kitchen that stays silent for two minutes and then hands you everything at once feels much slower than one that acknowledges you in five seconds, even if the total wait is identical.

In an LLM system, that first acknowledgment is called time to first token (TTFT): the gap between sending a request and seeing the first piece of the answer appear. The rest of the reply streams in afterward, one token at a time, at a pace called time per output token (TPOT), also known as inter-token latency. Together, end-to-end latency is roughly TTFT plus (output tokens minus one) times TPOT — a long answer can be slow even when both component numbers look healthy, simply because there are more tokens to generate.

✅ Worked example: NVIDIA's own inference-benchmarking documentation defines TTFT as covering tokenization, the model's "prefill" read of the whole prompt, and de-tokenization of the first output token — and explicitly excludes empty first responses from the measurement, because a TTFT computed on a blank token is meaningless. That precision is why serious teams don't just say "it feels fast" — they instrument the exact boundary.

Without this split, teams optimize the wrong thing. A team chasing a lower average latency might ship a change that helps typical requests but leaves a painful tail of slow ones untouched — which is why latency should only ever be read as percentiles (p50, p95, p99), never as an average. The OpenTelemetry project has standardized this thinking into official semantic conventions, defining gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token as histogram metrics so different vendors' dashboards can be compared apples-to-apples.

🎯 Use this when you're deciding what "fast" even means for a feature — a document summarizer and a live chat widget need completely different TTFT targets.

2. Latency Decomposition — Where the Milliseconds Hide

📌

Kid analogy: if a letter takes three days to arrive, decomposition is opening up that "three days" and finding out one day was the mail truck driving, one day it sat in a sorting bin, and one day it sat on a doorstep before anyone picked it up. You can't fix "three days" directly — you can only fix one of its parts.

Original diagram: the five waiting rooms a single request passes through.

A single request's total time is really network transit in, queue wait for a free GPU slot, the prefill phase (reading and encoding the whole prompt, which is what actually produces TTFT), the decode phase (generating tokens one at a time, governed by TPOT), and network transit back out. Standards work in this space — including an active IETF Internet-Draft on LLM benchmarking terminology — separates prefill latency, queue wait time, and inter-token latency into distinct, individually measurable stages precisely so teams stop lumping them together.

💡 Contrasting example, tying back to the counter analogy above: prefill scales with how long the prompt is, so a "good TTFT" is not one fixed number — a request with a 50-token question and a request with a 5,000-token pasted document have legitimately different TTFT floors. Comparing them on a single shared dashboard target is a common source of false alarms.

🎯 Use this when p95 latency spikes and nobody knows why — decomposition tells you whether to blame the network team, the scheduler, or the model itself.

3. Throughput — How Much Work the Fleet Gets Done

📌

Kid analogy: latency is how long it takes to serve one customer at a lemonade stand; throughput is how many cups the whole stand can pour in an hour once there's a line out the door. You can have a fast individual pour and still run a slow stand if you can only serve one person at a time.

Throughput is usually tracked as tokens per second across all concurrent requests, plus requests per second. The single biggest lever here is how requests are batched onto a GPU. Older "static batching" waits for every request in a group to finish before releasing any of them, so one long answer holds seven short ones hostage. Continuous batching — iteration-level scheduling that swaps a finished request out and a waiting one in at every decode step — fixes that.

✅ Worked example: Anyscale's widely-cited benchmark of the vLLM serving engine measured continuous batching plus its PagedAttention memory manager delivering up to roughly 23x higher throughput than naive static-batched serving on the same hardware, with the gap widening as answer lengths become more variable — exactly the traffic pattern a real chat product produces, where some replies are one sentence and others are three paragraphs.

Without watching throughput as its own KPI, a team can ship a model that looks snappy in a single-user demo and then collapses the moment ten people hit it at once, because nobody measured how the serving stack behaves under concurrency rather than in isolation.

🎯 Use this when capacity-planning for a launch — throughput under realistic concurrency, not single-request latency, tells you how many GPUs you actually need.

4. Scalability — Growing Without Falling Over

📌

Kid analogy: a see-saw works fine with two kids. Scalability is asking what happens when twenty kids want to ride it — do you add more see-saws smoothly, or does the one see-saw just snap?

Scalability measures whether latency and throughput stay healthy as load, model size, or context length grow — and whether the system can add capacity (more GPU replicas, more shards, more regions) without a painful manual scramble. Two failure shapes matter: vertical limits (a single GPU running out of memory for the KV cache as context windows grow) and horizontal limits (a fleet that can add machines but whose shared bottleneck — a database, a rate limiter, a single guardrail service — doesn't scale with it).

💡 Key warning, building on the batching example above: continuous batching itself has a ceiling — pack too many concurrent sequences onto one GPU and per-request TPOT degrades because each decode step is now doing more work. Autoscaling that only watches CPU or request count, and ignores GPU memory pressure or batch occupancy, will scale too late.

🎯 Use this when planning for a traffic-driving event (a product launch, a marketing push) — load-test the whole path, not just the model endpoint in isolation.

5. Cost Efficiency — The Bill Behind the Answer

📌

Kid analogy: two vending machines can sell the same candy bar, but one runs an inefficient old motor that burns through electricity for every purchase. Cost efficiency isn't about the price on the label — it's about what it actually costs the business to produce each answer.

Cost efficiency is typically tracked as cost per request, cost per 1,000 tokens, or cost per resolved task, and it has to include more than the base model call: retries after a timeout, extra tokens spent on retrieved context, and — increasingly — the cost of LLM-as-judge evaluation calls used to score quality in production. A judge call that costs a fraction of the original generation sounds cheap until it runs on 100% of traffic rather than a sample.

This is also where cost interacts directly with the batching and throughput work above: continuous batching raises GPU utilization, and higher utilization is usually the single biggest lever for lowering cost per token, because the fixed cost of a GPU-hour gets divided across more completed requests.

🎯 Use this when a pilot feature is about to go from 1% of users to 100% — model a per-request cost, not just a monthly budget, before flipping the switch.

6. Energy Use — The Watts Behind the Tokens

📌

Kid analogy: a nightlight and a floodlight both "just turn on," but one sips electricity and one gulps it. A single AI answer is the nightlight; the real energy story is what happens when a billion nightlights are on at once, all night, every night.

A single text query to a modern chat assistant is a genuinely small amount of energy — Google has published a figure of roughly 0.24 watt-hours for a typical Gemini app query, and OpenAI's CEO has cited a comparable figure of around 0.34 watt-hours for a typical ChatGPT query, both in the same rough range as a short web search. What makes energy a real KPI cluster isn't the per-query number, though — it's aggregate scale. The International Energy Agency has reported that data centers worldwide consumed on the order of 415 terawatt-hours of electricity in 2024, with AI-driven demand a major factor in projected growth toward roughly double that figure by 2030.

💡 Key warning: per-query energy estimates vary noticeably by source and methodology (model size, hardware generation, whether cooling and networking overhead are included alongside the GPU itself), so treat any single "X watt-hours per query" headline as an order-of-magnitude signal, not a precise fact to cite without checking its date and methodology.

🎯 Use this when a sustainability report or procurement review asks for the environmental footprint of an AI feature — report a methodology and a range, not a single borrowed number.

7. Resource-Constrained Performance — Running on Less

📌

Kid analogy: packing for a weekend trip in a huge suitcase is easy. Packing the exact same weekend into a small backpack forces real choices about what actually matters. Running a model on a phone or a small on-prem box is the backpack version of running it in a data center full of GPUs.

This KPI cluster covers how well a model performs when memory, compute, or power are capped — on-device assistants, embedded systems, or cost-capped self-hosted deployments. The standard levers are quantization (representing weights with fewer bits), distillation into a smaller model, and aggressive KV-cache management, all of which trade some accuracy or context length for a smaller footprint. The guardrail-classifier world offers a clean, well-documented illustration of the same tradeoff at a smaller scale.

✅ Worked example: Meta's Llama Prompt Guard 2, an 86-million-parameter classifier built specifically to run fast, has been benchmarked at roughly 20–50 milliseconds on an H100 GPU — small enough to sit in the critical path of every request — compared to hundreds of milliseconds for a general-purpose large model doing the same safety judgment. The pattern generalizes: purpose-built small models frequently match a much larger model's accuracy on a narrow task while using a fraction of the memory and compute.

🎯 Use this when a feature has to run offline, on-device, or within a fixed hardware budget — benchmark the actual constrained target, not a beefy development laptop.

8. Queue Management — Who Waits, and For How Long

📌

Kid analogy: a single checkout line at a busy store, versus a store that opens more registers and lets clerks help whoever's ready next, versus a store that just locks the door once it's full. All three are ways of "managing a queue," and they feel very different to the person standing in it.

Queue management decides what happens to a request that arrives when the serving fleet is already busy: hold it, batch it in dynamically, prioritize it, or reject it outright with backpressure. Continuous batching (covered in the throughput section) is a queue-management technique as much as a throughput one — it is, at its core, a policy for deciding which waiting request gets the next free decode slot. Admission control — capping how many requests are accepted before quality degrades for everyone already being served — is the companion practice: it protects existing users' latency at the cost of turning some new requests away or queueing them explicitly.

💡 Key warning: a queue that grows silently is a latency incident that hasn't been noticed yet. Queue depth and queue wait time need their own dashboard panel, separate from end-to-end latency, or a rising queue gets averaged away into a still-acceptable-looking p50.

🎯 Use this when traffic is bursty rather than steady (support tickets that spike after an outage, homework help that spikes before a deadline) — steady-state capacity planning alone will underbuild for the burst.

9. Integration Overhead — The Cost of Talking to Everything Else

📌

Kid analogy: asking one friend a question is fast. Playing a game of telephone through five friends before the answer gets back to you is slow, even if each individual friend answers instantly — the delay lives in the handoffs, not in any one person.

A production LLM feature is rarely just "call the model." It's usually: retrieve documents, call the model, call a tool, wait for the tool's result, call the model again, run a guardrail check, then respond. Each hop adds its own network round-trip, serialization cost, and failure surface, on top of the model's own latency. Integration overhead is the KPI that captures everything that isn't the model call itself but still counts against the user's total wait.

Retrieval-augmented generation (RAG) pipelines are the most common place this bites teams: a vector search, a re-ranking step, and a final generation call are three sequential round-trips stacked before the user sees a single token, and each one needs its own latency budget rather than being lumped into "the AI is slow."

🎯 Use this when a multi-step agent or RAG pipeline feels sluggish end-to-end even though the core model benchmarks fast in isolation — trace every hop before blaming the model.

10. Reliability — Staying Up When It Matters

📌

Kid analogy: a toy that works nine days out of ten isn't "mostly a good toy" — on the tenth day, whoever's counting on it is stuck. Reliability is about how often the system is there when you need it, not how good it is when it happens to be working.

Reliability covers uptime, error rate, timeout rate, and graceful degradation — does the system fall back to a smaller or cached model when the primary is unavailable, or does it simply fail the request? Public LLM API providers illustrate why this needs constant, independent monitoring rather than a one-time check: as one industry example, OpenAI has stated that it does not currently publish a formal latency or uptime SLA for its API and instead directs enterprise customers with strict requirements to its live status page, which tracks incidents across a large number of individually-monitored components.

✅ Worked example: the practical response teams take to that reality is to build their own synchronous health checks against the exact endpoints they call, in the exact regions their users sit in, rather than relying solely on a vendor's own status page — because a vendor's own reporting can lag a real degradation by many minutes, and per-model slowdowns often never make it onto a public status page at all.

🎯 Use this when a feature is on a critical path (checkout, fraud review, medical triage support) — design an explicit fallback path, and monitor it as carefully as the primary one.

11. Security Overhead — The Guardrail Tax

📌

Kid analogy: a security guard checking every bag at a stadium entrance keeps everyone safer, but it also means the line moves slower. Security overhead is the honest acknowledgment that safety checks cost time, and that cost has to be budgeted on purpose rather than discovered by accident.

Production guardrail stacks typically layer fast, cheap checks first (rule-based filters and regex, on the order of single-digit milliseconds) ahead of small specialized classifiers for things like prompt-injection or toxicity detection (tens of milliseconds), reserving a full LLM-as-judge safety review (hundreds of milliseconds to several seconds) for only the cases the earlier tiers flag as uncertain.

💡 Key warning, continuing the small-classifier idea from the resource-constrained section above: stacking guardrails serially rather than running them in parallel compounds their latency directly — a handful of 50–100 millisecond checks run one after another can add several hundred milliseconds before a single token is generated, on top of whatever a synchronous LLM-based judge adds if the request escalates to that tier. There's also a compounding accuracy cost, not just a latency one: chaining several imperfect guardrails multiplies their individual error rates rather than simply adding them, so more layers is not automatically safer.

🎯 Use this when a chatbot's latency budget is tight (sub-second, conversational) — parallelize cheap checks, and reserve any synchronous LLM-judge call for genuinely high-risk paths.

12. Speed vs. Accuracy vs. User Perception — The Three-Way Tug of War

📌

Kid analogy: a student who blurts out the first answer that comes to mind is fast but often wrong. A student who checks their work three times is more accurate but slow. And a student who narrates their thinking out loud while working feels more trustworthy to a nervous parent watching, even at the same actual speed and accuracy as one who stays silent.

This is the tension underneath every KPI cluster above: reasoning models that "think longer" before answering can raise accuracy on hard problems at the direct cost of TTFT and end-to-end latency; smaller resource-constrained models trade some accuracy for speed and lower cost; guardrails trade both speed and occasionally accuracy (false positives) for safety. And perception doesn't always track the raw numbers — streaming a response token-by-token, as most chat interfaces now do, makes a system feel faster than one that waits and delivers the whole answer at once, even when the true end-to-end latency is identical, because the user's felt wait is closer to TTFT than to total generation time.

🎯 Use this when choosing between a smaller, faster model and a larger, slower one for a given feature — ask which of speed, accuracy, or perceived responsiveness actually drives the user outcome you care about, rather than defaulting to "biggest model available."

13. Emerging in 2026 — What's Joining the KPI Panel Next

You didn't ask about these directly, but they're already reshaping the twelve clusters above in current production systems, so a 2026-dated post would be incomplete without flagging them.

Prefill/decode disaggregation.

📌

Kid analogy: a restaurant where the same cook both preps every ingredient and plates every dish forces the tasting-menu prep work to keep interrupting quick orders. Splitting those into a prep station and a plating station lets both jobs run at their own pace.

Production LLM serving has the identical problem: prefill (reading the prompt) is compute-heavy and bursty, while decode (generating tokens) is memory-bandwidth-bound and needs a steady rhythm; running both on the same GPU means a long prompt can stall everyone else's in-flight tokens and spike p99 TPOT. Academic work including DistServe and Splitwise made the case for routing the two phases to separate GPU pools connected by a fast KV-cache transfer, and by 2026 this pattern — sometimes paired with the Mooncake-style distributed KV cache — is directly supported in major serving engines including vLLM, with the Red Hat–led llm-d project (a CNCF Sandbox project as of March 2026) reporting a meaningful cut to per-token latency against a standard vLLM baseline. This directly extends the latency-decomposition and scalability clusters above: it's a concrete answer to "the queue diagram earlier showed prefill and decode as two boxes — what if they didn't have to share a GPU at all?"

Model routing and cascades.

📌

Kid analogy: a parent doesn't call a specialist doctor for every scraped knee — a first-aid kit handles most of them, and only the unusual cases get escalated.

Production systems increasingly route a request to a small, cheap, fast model first, and escalate to a larger, slower, more expensive model only when the small model reports low confidence or the task genuinely needs it. This is a direct, practical lever on the cost-efficiency and speed/accuracy/perception tradeoff clusters above — it lets a team keep typical-case latency and cost low without giving up accuracy on the harder minority of requests.

Agent trajectory evaluation.

📌

Kid analogy: grading a student only on their final answer misses whether they guessed, copied, or actually solved the problem step by step; a good teacher checks the work shown along the way.

As more production systems became multi-step agents that call tools across several turns, evaluating only the final response started hiding exactly the failures that matter most — an agent can reach a correct-looking answer while having ignored a tool's error, retried a broken call, or quietly substituted its own training-data guess for what a tool actually returned. The 2026 practice is trajectory evaluation: scoring the full path (tool selection, argument correctness, whether the tool's actual output was used, error recovery, and task completion) rather than the final message alone.

💡 Key warning, and the single most important addition this section makes to every cluster above: per-step reliability compounds multiplicatively across a trajectory. An agent that succeeds at each individual step 95% of the time only completes an eight-step task correctly around two-thirds of the time end-to-end, and a twenty-step chain at that same per-step rate succeeds well under half the time. That means the reliability, integration-overhead, and security-overhead clusters from earlier in this post all get harder, not easier, as agentic workflows add steps — each additional tool call is another opportunity for latency, cost, and failure to compound, not just another feature.

🎯 Use this when a feature moves from "one model call" to "an agent that plans and calls tools" — re-derive your latency, cost, and reliability budgets per step, then multiply, rather than reusing the single-call numbers from earlier sections.

14. Rolling This Out at Enterprise Scale

Tracking these KPIs on one team's dashboard is easy. Making them mean the same thing across dozens of teams, models, and product surfaces is the actual governance problem, and it usually needs a few deliberate structures:

1. Clear ownership. A named team (often platform or SRE, working with applied ML) owns the shared KPI definitions and dashboards, so "latency" means the same measurement boundary everywhere rather than five slightly different stopwatches.
2. CI-gated checks for model and prompt changes. A change to a prompt, a model version, or a guardrail configuration runs through an automated latency, cost, and quality regression suite before it ships — the same discipline a code change gets from unit tests.
3. Access control and governance over evaluation data. Latency and quality traces often contain real user traffic; the same access controls and retention limits that apply to production data have to apply to the eval and monitoring pipeline that stores it.
4. Cost governance for LLM-as-judge and guardrail calls at scale. A judge or safety call that's cheap per-request needs an explicit sampling policy once traffic is high, or "cheap" quietly becomes a major cost center.
5. Separate dashboards for training-time and inference-time metrics. Conflating them (as the Quick Comparison table above illustrates) hides real production problems behind healthy-looking training numbers.
6. Alerting tied to user-facing thresholds, not just infrastructure thresholds. A GPU running at 95% utilization might be perfectly healthy; a p99 latency crossing a business-defined pain threshold is the number that should actually page someone.

🎯 Use this when a successful pilot is about to be handed off from one team to "the whole company" — that handoff is exactly where undocumented KPI definitions cause the most confusion.

Common Mistakes

These recur often enough across the clusters above that they deserve to be named directly, along with the reasoning that makes each one tempting in the first place.

• Reporting latency as an average instead of percentiles. An average flatters a system with a long, painful tail, because a handful of very slow requests get diluted by many fast ones — but those slow requests are exactly the ones a real user remembers.
• Treating one aggregate "quality score" as sufficient. A single number can look stable while masking a regression that only affects one narrow slice of traffic (a specific language, a specific request type); a metric suite catches what a single score averages away.
• Ignoring cost and latency as first-class evaluation dimensions. It's tempting to evaluate only "is the answer good," but a system that's accurate, slow, and expensive can still be a failed product — cost and latency need pass/fail thresholds of their own, not just a footnote.
• Treating an offline eval pass as sufficient without live monitoring. A golden test set is, by definition, a fixed snapshot; real user traffic drifts away from it continuously, so a model that passes offline can still degrade in ways only live monitoring will catch.
• Letting golden datasets and load-test traffic patterns go stale. Test sets and synthetic traffic built for last year's usage patterns understate real cost, latency, and queueing behavior once user behavior shifts — new features, new prompt lengths, new burst patterns.
• Stacking guardrails without checking compounding accuracy or latency cost. As the security-overhead section above showed, more layers of imperfect checks can lower overall correctness even as each individual layer looks reasonable, while also silently eating the latency budget.

❓ FAQ

Is a lower average latency always better?

Not by itself. A lower average can hide a bad tail. Watch p95 and p99 alongside the average, since those are the requests a real user actually remembers as "slow."

What's the difference between throughput and scalability?

Throughput is how much work the current fleet completes right now; scalability is whether that throughput keeps rising smoothly as you add more capacity, or whether some hidden bottleneck caps it.

Do guardrails always make a system slower?

They add some latency, but well-designed guardrail stacks run cheap checks in parallel and reserve expensive LLM-as-judge calls for uncertain cases, keeping the typical-path overhead small while still catching most problems.

Is energy use really something a product team needs to track?

Per-query energy is small, but aggregate energy use scales with total traffic and is increasingly part of corporate sustainability reporting, procurement reviews, and, in some jurisdictions, disclosure requirements — so yes, at meaningful scale.

Why does perceived speed matter if the real latency is unchanged?

User satisfaction tracks the felt wait, which streaming ties closely to TTFT rather than total generation time — so improving TTFT (even without touching total latency) can measurably improve how "fast" a product feels.

🔗 References & Further Reading

• OpenTelemetry — Generative AI semantic conventions (metrics): opentelemetry.io/docs/specs/semconv/gen-ai
• NVIDIA — LLM inference benchmarking metrics documentation: docs.nvidia.com/nim/benchmarking/llm
• vLLM Project — official documentation and repository: github.com/vllm-project/vllm
• Anyscale — continuous batching benchmark repository: github.com/anyscale/llm-continuous-batching-benchmarks
• International Energy Agency — Electricity 2024/2025 and data-centre demand reporting: iea.org/energy-system/data-centres
• Google — published estimate of Gemini app text-query energy use: blog.google/technology/ai
• Meta AI — Llama Guard / Prompt Guard model documentation: ai.meta.com/llama/purple-llama
• OWASP — Top 10 for Large Language Model Applications: owasp.org/www-project-top-10-for-llm
• OpenAI Help Center — API latency guarantees and status page policy: help.openai.com
• IETF Datatracker — LLM Benchmarking Terminology Internet-Draft: datatracker.ietf.org
• CNCF — llm-d project (prefill/decode disaggregated serving), Sandbox status March 2026: github.com/llm-d/llm-d
• arXiv — Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving" (OSDI 2024): arxiv.org/abs/2401.09670
• arXiv — Qin et al., "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving" (FAST 2025): arxiv.org/abs/2407.00079

Product and vendor names above (NVIDIA, vLLM, Anyscale, Google, Meta, OpenAI, OWASP, IETF/OpenTelemetry) are trademarks of their respective owners, referenced here solely to attribute real, verifiable practices.

📝 Summary

• Latency is the wait a person feels, split into TTFT and TPOT.
• Latency decomposition breaks that wait into network, queue, prefill, and decode stages.
• Throughput is how much total work the fleet gets done, unlocked mainly by continuous batching.
• Scalability is whether throughput keeps rising smoothly as load and capacity grow.
• Cost efficiency tracks the real dollars per request, including judge and retry costs.
• Energy use is small per query but significant in aggregate, and worth reporting with a stated methodology.
• Resource-constrained performance is about doing the same job with a smaller footprint.
• Queue management decides who waits, gets batched in, or gets turned away under load.
• Integration overhead is every non-model hop that still counts against the user's wait.
• Reliability is whether the system is there at all when it's needed, with a real fallback path.
• Security overhead is the honest, budgeted cost of keeping a system safe.
• The speed/accuracy/perception tradeoff ties every other cluster together into one design decision.
• Emerging in 2026: prefill/decode disaggregation, model routing/cascades, and agent trajectory evaluation are already reshaping the twelve clusters above.

None of these numbers matter in isolation — a production LLM system is healthy when the whole panel of gauges looks reasonable together, not when any single needle points the right way. Happy measuring! 👋