Skip to main content

Quality, Safety, Cost, Reliability: The Four-Axis Framework for Production-Ready AI Systems

Calculating read time…

"Working" is not one measurement — it's four. A production AI system has to be helpful (quality), it has to stay inside policy and privacy limits even in rare edge cases (safety), it has to run for a sustainable price (cost), and it has to actually be there and behave consistently when someone needs it (reliability). Optimize for only one of these and you get a system that looks great in a demo and fails in a way nobody budgeted for. 🧭


This matters because "quality" is the axis everyone measures, and the other three are the ones that actually cause outages, lawsuits, and blown budgets. A team can ship a genuinely well-written, accurate assistant that quietly leaks confidential data, costs ten times what anyone approved, or becomes unreachable the one day a customer actually needs it — and by any single-score quality metric, that team would look successful right up until it wasn't. This post walks through each of the four axes with a real, documented, named example for each one — not a hypothetical — and one running example (an internal travel-and-expense policy assistant) to show how the four axes actually pull against each other in practice. 🧮

Diagram showing four questions behind whether an AI system is working: quality (helpful and correct), safety (respects policy and privacy), cost (stays within budget), and reliability (consistent, available, and falls back safely)

🔀 Quick Comparison: Quality-Only View vs. Four-Axis View

  Quality-only view Four-axis (Q/S/C/R) view
Release question "Does it sound right?" "Is it right, safe, affordable, and actually there?"
What gets hidden A rare but severe privacy or policy failure, buried in a good average score Nothing gets averaged away — each axis has its own gate
Cost blind spot Invisible until the invoice arrives Tracked per-request, budgeted before launch
Outage handling Not part of the evaluation at all Fallback behavior is a release requirement, not an afterthought

1. What "Working" Actually Means in Production

🧒 Kid version: a toy car that's the fastest one on the shelf isn't automatically the "best" car. If its wheels fall off after five minutes, if it needs a battery that costs more than the car itself, or if it only turns on when it feels like it — speed alone doesn't make it good. You'd want it to also be safe to play with, affordable to keep running, and there when you actually want to play.

Most teams building an AI assistant start by asking one question: is it helpful? That's a quality question, and it's the easiest one to test — you can read ten sample outputs and get a gut feeling for whether they're good. The problem is that "good on ten sample outputs" says nothing about whether the assistant will leak something it shouldn't on the eleventh, whether it can be run at the volume the business actually needs, or whether it stays up during the exact moment someone depends on it.

A useful way to frame production readiness is to treat it as four separate pass/fail questions rather than one blended score. Quality asks whether the system is helpful, correct, complete, and appropriate for the task. Safety asks whether it respects policy, privacy, security, and risk limits — including in the rare cases that don't show up in a quick demo. Cost asks whether running it is sustainable at real volume: tokens, retrieval calls, tool calls, and the infrastructure underneath all of it. Reliability asks whether it behaves the same way today as it did yesterday, and whether it's actually reachable — latency, uptime, tool availability, and what happens when something fails.

✅ Worked example, set up: picture a mid-sized company's internal travel-and-expense assistant. Employees ask it things like "can I expense a client dinner over $150?" or "what's the mileage rate for my personal car this quarter?" It reads the company's expense policy, checks a few internal systems, and answers in plain language. This is the running example for the rest of the post.

🎯 Use this when: someone on the team says a release "looks good" — ask which of the four axes they actually checked, because "looks good" is almost always a quality-only judgment.

2. Quality: Helpful, Correct, Complete, Appropriate

🧒 Kid version: if you ask a friend for directions and they confidently send you the wrong way, being confident doesn't help you get there. Quality isn't about how sure or how polished an answer sounds — it's about whether it's actually correct and actually answers what you asked.

For the expense assistant, quality means the policy it quotes is the real, current policy — not a plausible-sounding guess at what the policy probably says. That distinction sounds obvious until you look at what happens when a company lets a chatbot answer policy questions without hard grounding in the actual source document.

💡 A real, named quality failure: in November 2022, Jake Moffatt asked Air Canada's website chatbot about the airline's bereavement fare policy after his grandmother died. The chatbot told him he could apply for the bereavement fare retroactively, any time within 90 days of the flight. That wasn't true — Air Canada's actual policy required the application to be submitted before travel, not after. Moffatt booked his flights based on the chatbot's answer, was later denied the refund, and took the airline to Canada's Civil Resolution Tribunal. Air Canada's defense was that the chatbot was "a separate legal entity" responsible for its own words. The tribunal rejected that argument outright, held the airline legally accountable for what its own chatbot had told him, and ordered Air Canada to pay the promised refund plus damages. The chatbot was pulled from the website within months.

What makes this a quality failure specifically, rather than a safety or reliability one, is that the assistant did exactly what it was supposed to do structurally — it answered fluently, in context, on-topic — and the content was simply wrong. No system crashed, no data leaked, nothing was slow. It was a confident, complete-sounding, incorrect answer to a real question, and that was enough to create real legal and financial liability. For the expense assistant, the equivalent failure would be confidently quoting last year's mileage rate, or inventing a plausible-sounding approval threshold that doesn't actually exist in the policy document.

Quality, done properly, means checking correctness against the actual source of truth (not the model's general sense of what a bereavement policy "usually" looks like), completeness (did it mention the conditions that apply, not just the headline number), and appropriateness for who's asking (an employee's plain-language question deserves a plain-language answer, not a copy-pasted policy paragraph).

🎯 Use this when: an assistant answers policy, pricing, or eligibility questions — quality testing has to check the answer against the live source document, not just whether the answer reads well.

3. Safety: Policy, Privacy, Security, Risk

🧒 Kid version: a babysitter who's fun and plays great games is still the wrong babysitter if they'll hand your house keys to a stranger who asks nicely. Being good at the main job doesn't matter if the same system fails at the "don't do the genuinely dangerous thing" part.

Safety failures are dangerous precisely because they can be rare. A system can pass a thousand quality checks and still have one catastrophic safety gap that a thousand checks never happened to trigger. This is also why safety failures often come from the humans using the system, not just the system itself — which is exactly what happened at Samsung.

💡 A real, named safety failure: in March 2023, weeks after Samsung's semiconductor division lifted an internal ban on ChatGPT, three separate incidents surfaced within about 20 days. One engineer pasted confidential source code into ChatGPT while asking it to help debug a database issue. Another employee fed it code related to defective equipment, asking for an optimization. A third recorded an internal meeting, transcribed it, and asked ChatGPT to turn the transcript into meeting minutes. There's no indication any of the three employees intended harm; each was simply reaching for a fast, free tool to solve an ordinary work problem. But because ChatGPT's consumer product could retain submitted data, Samsung's proprietary source code and internal meeting content had effectively left the building, with no way to pull it back. Samsung responded by banning generative AI tools company-wide and warning that violations could lead to termination.

Notice what this incident actually is: a data-handling and system-design failure, not a model quality failure. The tool worked exactly as designed — it answered helpfully, and it stored what was sent to it, because that's what a consumer product with no enterprise data agreement does. The safety gap was that nothing in the system stopped confidential source code from reaching a third-party service that retains inputs. For an internal expense assistant, the equivalent risk isn't hypothetical: if it can see an employee's submitted receipts, home address for mileage reimbursement, or salary-linked approval thresholds, then anywhere that data could leave the company's control — through logging, through a third-party model provider, through an overly broad tool permission — is a safety gate, not a quality one.

This is also why safety needs zero-tolerance thresholds rather than an acceptable error rate the way quality often does. A 2% rate of slightly awkward phrasing is a quality nuisance. A 2% rate of leaking someone's home address is not a rounding error — it's a policy violation happening roughly one in fifty times, and it should block a release regardless of how good the assistant is on everything else.

🎯 Use this when: designing what data an assistant can see or which tools it can call — ask what happens if this exact request pattern occurs, not just what happens with the typical request.

4. Cost: Tokens, Retrieval, Tools, Infrastructure

🧒 Kid version: a lemonade stand that makes the best lemonade in town still goes out of business if each cup costs more to make than you sell it for. Quality and cost are separate questions — "is it good" and "can we actually afford to keep doing this" don't have the same answer.

Cost is the axis that's easiest to ignore during development, because a handful of test queries against an API costs pennies. The problem shows up only once volume is real — and by then, the architecture decisions that determine the actual bill were made a long time ago.

💡 A real, documented cost failure and fix: Dataherald, a company that built a natural-language-to-SQL engine using a retrieval-augmented generation pipeline, published a case study on discovering that their system was consuming roughly 150,000 tokens per query. After implementing proper production monitoring (via LangSmith) to actually trace where those tokens were going, the team found a bug that was inflating token usage well beyond what the task required. Fixing it took per-query consumption down to roughly 25,500 tokens, an 83% cut, and the system's accuracy didn't move. Nothing about the model's intelligence changed. The fix was entirely about seeing, for the first time, where the tokens were actually being spent.

The pattern behind that case study is common enough to have a name in cost-optimization circles: a cost spiral. It typically starts because development-time testing happens at low volume, where a wasteful prompt or an over-eager retrieval step is invisible — a few extra thousand tokens per call doesn't show up on anyone's radar until that same call is happening a million times a day. For the expense assistant, the equivalent risk is retrieving the entire expense policy document on every single question instead of just the relevant section, or calling an internal lookup tool for information the assistant already has in context. None of these mistakes are visible in a quality review, because the answer still comes out correct — it's just far more expensive to produce than it needs to be.

Cost as a production axis means tracking cost per request (not just total monthly spend, which hides which feature or team is driving it), setting a budget before launch rather than reacting to the first invoice, and treating a cost spike as an incident with a root cause to find — the same way a latency spike would be treated — rather than a number to shrug at.

🎯 Use this when: a prototype is about to move from a handful of test users to real company-wide rollout — cost per request should be measured and budgeted before that jump, not after the first bill.

5. Reliability: Consistency, Latency, Uptime, Fallback

🧒 Kid version: a fire alarm that works perfectly nine days out of ten isn't a good fire alarm — reliability is exactly the property you need most on the one day everything else goes wrong. A tool that's excellent most of the time but silently absent right when it's needed hasn't actually done its job.

Reliability is unusual among the four axes because a system can be flawless on quality, safety, and cost, and still fail this one completely — simply by not being there, or not behaving the way it did an hour ago. And reliability failures aren't limited to the system a team builds; they can come from a dependency several layers upstream.

💡 A real, dated reliability failure: on the morning of September 3, 2026, OpenAI, Anthropic, and xAI all experienced overlapping outages within roughly a 90-minute window. A brief window of routing trouble at OpenAI knocked out ChatGPT and Codex access for a subset of users beginning around 7:43 am PT, and the company had it working again roughly half an hour later. Anthropic's status page described a partial outage affecting several Claude models, which it said it had identified and resolved by around 9:16 am PT. xAI's parent company, SpaceX, separately pointed to an infrastructure failure at its Memphis compute site as the cause of Grok's downtime. Downstream products that depend on these providers — including coding tools built on top of them — went down too, simply because the infrastructure underneath had failed. No single company caused all three outages, and no shared cause was confirmed, but the effect for anyone building on top of any of these providers was identical: the system they depended on simply wasn't there.

This kind of incident illustrates two separate reliability requirements at once. The first is straightforward uptime and latency — is the model endpoint actually responding, and how fast. The second, easier to overlook, is fallback behavior — what does your product do when the answer is no? A well-designed system degrades gracefully: it might queue the request, fall back to a cached answer, or clearly tell the user "this service is temporarily unavailable, try again shortly" instead of hanging indefinitely or returning a confusing error. For the expense assistant, a reliability failure isn't only "the model is down" — it's also "the internal policy-lookup tool timed out, and the assistant answered the question anyway using only what it already had in context," which can quietly turn a reliability problem into a quality problem downstream.

Reliability, measured properly, includes p95 latency (not just average latency, which hides the worst experiences), uptime of every dependency the system touches, tool call success rate, and — critically — whether the fallback path was actually tested, not just assumed to exist.

🎯 Use this when: designing any AI feature that depends on an external model provider or internal tool — write down, on purpose, what the user sees when that dependency is unavailable, rather than discovering it during an actual outage.

6. How the Four Axes Pull Against Each Other

The four axes aren't independent dials you can each turn to maximum. Improving one routinely costs you on another, and pretending otherwise is how teams end up making tradeoffs by accident instead of on purpose.

  • Quality vs. cost: retrieving more context and using a larger model generally improves correctness — and increases token spend and latency. The expense assistant could pull the entire policy handbook into every prompt "to be safe," which would help quality slightly while multiplying the cost of every single question.
  • Safety vs. quality: tightening what the assistant is allowed to say about sensitive topics (salary bands, personal reimbursement details) reduces safety risk but can make it refuse or hedge on legitimate questions, hurting the quality score reviewers actually see.
  • Reliability vs. cost: adding redundancy — a backup model provider, retry logic, cached fallback answers — improves reliability and adds infrastructure and engineering cost that has to be budgeted for, not treated as free insurance.
  • Cost vs. reliability under load: the cheapest way to control cost is often to cap retries and set aggressive timeouts — which is exactly the setting that makes a system look unreliable the moment an upstream dependency gets slow, rather than briefly degraded.

💡 Key warning: none of these tradeoffs are inherently wrong to make. They become a problem only when they're made silently. A team that decides "we'll accept a 5% cost increase for a meaningful drop in policy-violation risk" has made a defensible, explicit choice. A team that just watches cost creep upward without ever deciding whether the quality gain was worth it hasn't made a decision at all — they've just let one happen.

🎯 Use this when: reviewing a proposed change to any one axis — ask out loud which of the other three axes it's quietly trading against, and whether that trade was actually intended.

7. Turning Four Axes Into Ship/Block Decisions

Naming four axes is only useful if each one produces an actual yes-or-no answer before release, not just a general impression. A few practices make that concrete for a team of any size:

  1. Give each axis its own threshold, not a blended score. A single "overall health" number can go up while a critical axis quietly goes down underneath it. The expense assistant shouldn't ship on a great average score if its safety gate — no exposure of another employee's reimbursement data — has any failures at all.
  2. Make safety failures zero-tolerance, and make quality, cost, and reliability failures budget-based. A rare but severe safety violation should block release outright. A 2% dip in helpfulness on an edge-case question is something a team can consciously accept within a defined budget.
  3. Define what happens on failure before it happens. For each axis, write down the rollback trigger in advance: a spike in policy violations, a sustained latency increase, a cost overrun beyond the agreed budget. Shipping isn't a one-way door, and knowing the exit criteria in advance is what makes a fast rollback possible instead of a panicked one.
  4. Different products need different bars on the same four axes. An internal expense assistant with medium-stakes mistakes can tolerate a different safety and reliability bar than, say, a customer-facing healthcare or banking assistant. The four axes are the same everywhere; the thresholds are not.

🎯 Use this when: a release review turns into a debate about whether something "feels ready" — replace the feeling with four written thresholds and check each one.

8. Mistakes That Keep Happening

  1. Treating quality as the whole evaluation. It's the easiest axis to eyeball, which is exactly why teams over-invest in it and under-invest in the other three.
  2. Discovering cost after launch instead of before. Development-time testing volumes hide the per-request economics that only become visible — and only become a problem — at real scale.
  3. Treating safety as a rate instead of a threshold. A "low" rate of privacy violations is still a real number of real incidents once volume is high enough; safety needs a hard floor, not an average.
  4. Assuming reliability is someone else's job. "The model provider had an outage" is true and also not an excuse — the September 2026 incident shows that even flawless in-house engineering can go down because of a dependency three layers removed.
  5. Never testing the fallback path. A fallback that exists on paper but has never actually been triggered in testing usually fails in some unexpected way the first time it's needed for real.
  6. Making axis tradeoffs invisibly. Cost, safety, and reliability changes routinely happen as side effects of a "quality improvement" nobody explicitly signed off on trading against.

🎯 Use this when: auditing an existing AI feature — check which of these six mistakes is currently happening quietly, since most teams have at least one active.

❓ FAQ

Isn't quality the most important axis, since it's what users actually notice?
Users notice quality first, but they notice safety, cost, and reliability failures hardest. A single leaked document, a service outage, or a shut-down feature due to runaway cost tends to matter far more than a slightly less polished answer.

Do all four axes need the same rigor for every AI feature?
No. A low-stakes internal tool and a customer-facing financial or healthcare assistant use the same four axes but need very different thresholds — the framework doesn't change, the bar does.

Can a safety failure really come from good-faith, non-malicious use?
Yes — Samsung's 2023 incident involved employees trying to do their jobs faster, not anyone acting maliciously. Safety failures are often a system-design gap, not a bad actor.

Is cost really an evaluation concern, or just a finance problem?
It's both, but by the time it reaches finance as a problem, the architecture decisions that caused it were made much earlier. Tracking cost per request during development is what keeps it out of the finance-escalation category.

Can reliability fail even if my own code and prompts are perfect?
Yes. The September 2026 outage affecting OpenAI, Anthropic, and xAI simultaneously shows that reliability depends on the full dependency chain, including infrastructure your team doesn't control.

🔗 References & Further Reading

  • Civil Resolution Tribunal of British Columbia. Moffatt v. Air Canada, 2024 BCCRT 149 (14 February 2024) — official tribunal decision on chatbot-caused negligent misrepresentation.
  • Bloomberg. Samsung Bans ChatGPT and Other Generative AI Use by Staff After Leak (May 2, 2023) — original reporting on the Samsung source-code and meeting-notes incidents.
  • ZenML LLMOps Database. Dataherald: Optimizing LLM Token Usage with Production Monitoring in Natural Language to SQL System — documented case study of the 150,000-to-25,500 token reduction.
  • OpenAI and Anthropic public status-page disclosures for the September 3, 2026 outage, as reported by AI Chat Daily and Axios — status.openai.com and status.anthropic.com remain the authoritative sources for the incidents themselves.

This post explains and synthesizes publicly reported incidents and documented case studies in original wording, illustrated with an original worked example (the internal travel-and-expense assistant) and an original diagram — it does not reproduce any source's text or structure verbatim. Specific facts (dates, dollar figures, token counts, timestamps) are drawn from the cited sources; consult the originals for full detail before citing them elsewhere. Company and product names mentioned are trademarks of their respective owners, cited here for factual attribution only.

📝 Summary

  • "Working" in production spans four separate axes — quality, safety, cost, and reliability — not one blended score.
  • Quality failures can happen with no crash and no leak at all: Air Canada's chatbot answered fluently and was simply wrong, which was enough to create real legal liability.
  • Safety failures are often system-design gaps triggered by ordinary, well-intentioned use, as Samsung's 2023 incident showed — and they need zero-tolerance thresholds, not an acceptable error rate.
  • Cost is invisible at development-time volume and can hide architectural waste, like the 83% token reduction Dataherald found only once it actually traced where tokens were going.
  • Reliability depends on the entire dependency chain, not just your own code — the September 2026 outage took down services that never touched a faulty line of their own.
  • The four axes routinely trade against each other; the goal isn't to avoid every tradeoff, it's to make each one explicit instead of accidental.
  • Each axis needs its own threshold and its own rollback plan defined before release, not a single "looks ready" gut check.

The habit worth keeping: before calling anything "done," check it against all four questions separately — is it right, is it safe, can we afford to keep running it, and will it actually be there when someone needs it. A yes on three out of four is still a no. Thanks for reading. 👋

Comments