LLM Training Data Quality Gates: A Guide to Provenance, Deduplication & Contamination Detection

Data quality gates are the checkpoints a training corpus has to pass — for provenance, deduplication, toxicity, format consistency, instruction hygiene, coverage, and contamination — before a single training step ever runs on it. 🚦

A training run is not something you can quietly patch after the fact the way you'd fix a buggy release. Once a model has trained on duplicated text, on content that overlaps a benchmark you plan to report scores on, or on data nobody can confirm the right to use, that's baked into weights that took days or weeks and real money to produce. Retraining from scratch because a gate that should have run on day one didn't is one of the most expensive mistakes in this field — expensive in compute, in schedule, and in trust once a contaminated benchmark score gets published and later has to be walked back. 🧯

📑 In This Post

1. Why Training Data Needs Its Own Quality Gates
2. Provenance
3. Deduplication
4. Toxicity Screening
5. Format Consistency
6. Instruction Hygiene
7. Coverage
8. Contamination Detection
9. Hands-On Lab: Run a Mini Gate Pipeline Yourself
10. Rolling This Out at Enterprise Scale
11. Common Mistakes
12. FAQ
13. References & Further Reading
14. Summary

🔀 Quick Comparison

Gate	What It Catches	What Breaks Without It
Provenance	Unclear source, licensing, or consent for a document	Legal and trust exposure discovered only after the model ships
Deduplication	Exact or near-exact repeated text	Wasted compute and a model that memorizes and regurgitates repeated passages
Toxicity screening	Harmful, hateful, or unsafe content	A model that reproduces harmful patterns it was never meant to learn
Format consistency	Malformed records that don't match the expected schema	A training job that fails partway through, or silently trains on garbage fields
Instruction hygiene	Near-duplicate instructions padding a fine-tuning set	A dataset that looks large but is narrow, overfitting the model to a few patterns
Coverage	Missing domains, languages, or task types	Blind spots that only surface once real users hit them in production
Contamination detection	Benchmark test data leaking into the training set	Inflated eval scores that don't reflect real capability, discovered after publication

1. Why Training Data Needs Its Own Quality Gates

🧒 Kid analogy: a bakery checks every ingredient before it goes into the mixing bowl — is the flour actually flour, is the milk still fresh, did a stray eggshell get in? Nobody checks the ingredients by tasting the finished cake and hoping for the best, because by then it's too late to pull the eggshell back out. Training data works the same way: once it's mixed into a model through training, you can't reach back in and remove just the bad part.

Each gate in this post answers one specific, checkable question about a candidate document or instruction before it's allowed into the training set: where did this come from, have we seen it before, is it safe, is it structured correctly, is it genuinely new, is it representative, and does it secretly contain the answer key to a test we'll use later? None of these questions can be fully answered by looking at overall model quality after training — by then, whatever got through the gates is already part of the weights.

🎯 Use this when: designing any pretraining or fine-tuning data pipeline — treat each gate below as a required stage with its own pass/fail criteria, not an optional cleanup pass at the end.

2. Provenance

🧒 Kid analogy: if a friend hands you a toy and says "here, keep it," you'd probably want to know whether it was actually theirs to give away, or borrowed, or found somewhere. Provenance for training data is the same question, at scale: for every single document, can you say where it came from and whether you're actually allowed to use it?

🏢 Real-world example: the Allen Institute for AI's Dolma corpus — a fully open, three-trillion-token pretraining dataset released alongside a public data sheet and toolkit — treats provenance as a first-class deliverable rather than an afterthought. Its creators describe explicitly documenting which sources went into the corpus and why, consulting with legal reviewers on sourcing decisions before finalizing the mix, and publishing that reasoning openly enough that outside researchers can rebuild the dataset from scratch using the released toolkit. That openness is itself the point: a team that can point to exactly where every slice of its training data came from is in a fundamentally different position, legally and reputationally, than one that can only say "it's mostly from the web."

In practice, a provenance gate records, per source or per document: where it was obtained, under what license or terms, when it was collected, and whether any removal or opt-out mechanism applies to it. That record needs to survive independently of any one engineer's memory — it's the artifact a legal or compliance review will ask for later, and "we're pretty sure it came from a public dataset" is not an answer that holds up under scrutiny.

✅ Worked example: a training pipeline that tags every ingested document with its source dataset name, collection date, and license at ingestion time — before any filtering happens — so that if a source is later found to be unusable, every affected document can be located and removed by that tag alone, rather than by re-scanning the entire corpus.

💡 Key warning: provenance tracked only at the source level ("this batch came from Common Crawl") isn't the same as provenance tracked per document. If one specific site within a broad web crawl later needs to be excluded, source-level tracking can't isolate it — the whole batch is either kept or discarded.

🎯 Use this when: any external or web-sourced data enters the pipeline — that's the point where provenance has to be captured, because it's far harder to reconstruct after the fact.

3. Deduplication

🧒 Kid analogy: if you're making a scrapbook and accidentally glue in the same sticker five times, the book doesn't get more interesting — it just gets thicker with the same thing repeated, and you wasted stickers you could have used on something new. A model trained on the same passage repeated many times spends its limited "attention" during training re-learning something it already knew, instead of learning something new.

🏢 Real-world example: Dolma's documented pipeline runs deduplication in three distinct passes, each coarser-to-finer than the last: first at the URL level, dropping web pages that were crawled more than once; then at the document level, removing pages from different URLs that turn out to contain exactly the same text (mirrors and syndicated re-posts, for instance); and finally at the paragraph level, stripping out repeated boilerplate — like navigation menus or cookie notices — that recurs inside otherwise-unique documents. All three passes are implemented using a Bloom filter, a probabilistic data structure chosen specifically because it can check "have I seen this before?" against billions of documents without needing to store every document's full text in memory.

The staged ordering matters mechanically: URL-level checks are cheap and remove the most obvious repeats first, shrinking the corpus before the more expensive paragraph-level comparison has to run on it — running the finest-grained check first on an un-deduplicated corpus would waste enormous compute re-comparing text that a cheaper check would have already caught.

✅ Worked example: a news aggregator's crawl picks up the same wire-service story republished on forty different outlets' websites — different URLs, near-identical text. Document-level deduplication catches this even though URL-level checks alone would have let all forty copies through.

💡 Contrasting case: two documents that are 95% identical but differ in one meaningfully updated paragraph (a news article updated after breaking developments, for instance) are a harder case than exact duplicates — exact-match deduplication would treat them as entirely different documents and keep both, near-duplicate detection is needed to catch this without over-triggering on legitimately similar-but-distinct content.

🎯 Use this when: ingesting any large web crawl — mirrored and syndicated content is close to guaranteed at that scale, not an edge case.

4. Toxicity Screening

🧒 Kid analogy: before you put fruit in a basket for a fruit salad, you check each piece and set aside anything that's gone bad — otherwise one rotten piece can spoil the pieces touching it. Toxicity screening is that check for training data: setting aside content that's harmful before it ever gets mixed in with everything else.

🏢 Real-world example: Dolma's toolkit implements toxicity screening as a blend of two approaches applied depending on the data source — rule-based filters for patterns that are reliably identifiable through fixed heuristics, and classifier-based filters, trained models that score text for the likelihood it contains toxic or harmful content, for cases where fixed rules would be too blunt. Their documentation is candid that this is not a solved problem: they describe explicitly weighing trade-offs, noting for example that aggressively removing toxic content can also reduce a model's ability to recognize and correctly respond to hate speech when a downstream application actually needs it to.

That trade-off is the core design tension in this gate: a threshold set too loose lets harmful content through; a threshold set too aggressive can strip out legitimate content that merely discusses sensitive topics (news coverage of violence, educational material about historical atrocities, medical text about self-harm), degrading the model's ability to handle those topics appropriately rather than simply avoiding them.

✅ Worked example: a customer-support training corpus includes real messages like "this is the worst service I've ever received" — a well-calibrated classifier lets this through as ordinary frustration rather than flagging it as toxic, keeping the corpus useful for training a model that has to recognize and respond to upset customers.

💡 Contrasting case: that same classifier, tuned too aggressively, could also strip out a customer's message disclosing a self-harm risk or reporting abuse — precisely the kind of message a support-response model most needs to learn to handle carefully, not the kind it should never have seen. A threshold tuned purely to maximize removal can quietly damage the use case the model exists for.

🎯 Use this when: setting a toxicity classifier's threshold — treat it as a tunable parameter to validate against real examples from your own domain, not a fixed setting you configure once and never revisit.

5. Format Consistency

🧒 Kid analogy: if everyone in a group project agrees to save their slide in the same file format, the presentation opens smoothly for everyone. If one person saves theirs differently, the whole slideshow can break when it's time to present — not because their content was bad, just because it didn't match what everyone else expected. Training data needs the same agreement: every record has to follow the same structure, or the pipeline that reads it breaks.

🏢 Real-world example: OpenAI's fine-tuning documentation specifies an exact structural contract for training files — each line of the uploaded file must be a valid JSON object containing a messages field holding a list of role-and-content pairs — and the platform validates uploaded files against that structure before a fine-tuning job is allowed to start, rejecting files where a record is missing a required field or uses an outdated field name rather than silently attempting to train on malformed data.

This gate is less about content quality and more about structural predictability: a training pipeline downstream of this gate is written to expect a specific shape of input, and any record that doesn't match that shape is either a pipeline bug waiting to happen or, worse, silently ignored fields that quietly reduce how much a model actually learns from that example.

✅ Worked example: here's the shape of a minimal format-validation gate, explained before the code. It reads a training file one line at a time and does three checks on each line before letting it through: (1) confirm the line parses as valid JSON at all — a single unescaped quote elsewhere in the file shouldn't corrupt every other line; (2) confirm the required top-level field is present; and (3) confirm every message in that field has both a role and non-empty content. Anything that fails any check is written to a separate rejects file with a reason, rather than silently dropped, so a human can review exactly what got excluded and why.

valid, rejected = [], []
for line_num, raw_line in enumerate(open(training_file)):
    try:
        record = json.loads(raw_line)
        messages = record["messages"]
        assert all(m.get("role") and m.get("content") for m in messages)
        valid.append(record)
    except Exception as err:
        rejected.append({"line": line_num, "reason": str(err)})

💡 Key warning: a record can pass every structural check above — valid JSON, the right field present, a non-empty string — while that string is still an empty-looking placeholder like a single space, or boilerplate copy-pasted from a template. Format validation confirms shape, not substance, so it has to run alongside gates like deduplication and toxicity screening, never as a replacement for them.

🎯 Use this when: training data is assembled from more than one source or generation process — that's exactly when subtle schema mismatches (an extra field, a renamed key) tend to creep in undetected.

6. Instruction Hygiene

🧒 Kid analogy: a worksheet with twenty questions that all secretly ask the same thing in slightly different words isn't really twenty questions — it just looks like it. Instruction hygiene is making sure a set of training instructions is actually as varied as it appears to be, not the same handful of requests wearing different outfits.

🏢 Real-world example: the Self-Instruct method, published by Wang et al. and later used as the data-generation approach behind well-known open instruction-tuned models, generates new candidate instructions from a language model itself and then explicitly filters them before they're added to the training pool: any new instruction is compared against every instruction already kept using ROUGE-L similarity, a text-overlap metric, and discarded if its similarity to any existing kept instruction exceeds a 0.7 threshold. On top of that similarity filter, the method also applies keyword-based filtering to remove instructions asking for capabilities the model doesn't have (referencing an image or a graph, for instance), and removes generated instances that are exact duplicates or that pair the same input with a different, presumably inconsistent, output.

The mechanical reason this matters: an instruction-tuning dataset's value comes largely from the diversity of tasks it demonstrates, not its raw example count. A dataset padded with near-duplicate phrasings of the same handful of instructions can look large on a spreadsheet while teaching the model far less than its size suggests — every additional near-copy adds training compute without adding new signal.

✅ Worked example: a team generating customer-service instructions produces eight different surface forms of "How do I cancel my subscription?" — including "I want to end my subscription, how?" and "Please explain the cancellation process." ROUGE-L filtering against the first kept version catches the other seven as near-duplicates, keeping the pool at one genuinely distinct instruction instead of eight that all teach the model the same thing.

💡 Key warning: the Self-Instruct paper's own authors reported that a manual review of 200 random generated instructions found roughly 46% had some quality problem — a reminder that similarity-based filtering catches redundancy, not correctness, and generated instruction data still benefits from spot-checking beyond automated filters alone.

🎯 Use this when: instruction data is generated synthetically at scale (by a model, rather than written entirely by hand) — that's exactly the process most prone to producing large volumes of superficially different, substantively repetitive examples.

7. Coverage

🧒 Kid analogy: if a test covers five chapters but you only studied chapter one really thoroughly, being an expert on chapter one won't help with the other four. Coverage is checking, before the test, that you've actually studied everything that's going to be on it — not just your favorite or easiest chapter.

🏢 Real-world example: Dolma's own documentation describes coverage as an explicit, evaluated design decision rather than an incidental outcome of whatever data happened to be available. Its creators note including Wikipedia text specifically because it measurably improved performance on K-12 science knowledge benchmarks, and separately describe an acknowledged tension in deciding how much code to include: adding code documents improves code-related task performance but can reduce performance on general text benchmarks, so the mixture ratio itself becomes a deliberate, tested decision rather than a default. They're equally direct that their own evaluation suite has blind spots — for example, that it can't fully isolate the effect of adding code to an otherwise text-heavy corpus, since many code benchmarks require additional instruction-tuning to measure properly.

The general pattern is a coverage matrix: define the domains, languages, formats, and task types the model actually needs to be good at, measure what fraction of the current corpus represents each cell of that matrix, and treat sparsely-covered cells as an active gap to fill — not something to discover for the first time when a real user's query falls into one.

✅ Worked example: a team building a multilingual support model crosses five supported languages against ten product categories in a coverage matrix, and finds Spanish-language tickets for their newest product line are almost entirely absent — so they specifically source or generate data for that exact cell, rather than adding generic Spanish text that wouldn't close the actual gap.

💡 Key warning: coverage measured only along single dimensions can hide a gap at their intersection. A corpus can look well-covered in "Spanish" and well-covered in "billing questions" separately, while Spanish-language billing questions specifically are almost absent — the aggregate percentages for each dimension alone won't reveal that.

🎯 Use this when: defining the target use cases for a model before data collection begins — coverage gaps are far cheaper to catch against a stated target than to notice only after deployment.

8. Contamination Detection

🧒 Kid analogy: if you accidentally studied using the actual answer key to tomorrow's test, acing the test the next day wouldn't prove you understood the material — it would just prove you could recognize answers you'd already memorized. Contamination detection is checking, before training, that none of tomorrow's test questions accidentally ended up in today's study materials.

🏢 Real-world example: this is one of the most consistently documented gates across major model releases, and different teams have published different concrete thresholds for it. The original GPT-3 technical report defines a 13-gram (a run of 13 consecutive words or tokens) overlap between a training document and a benchmark test example as evidence of contamination, and filters affected training content out on that basis. OpenAI's GPT-4 technical report uses a related but distinct method, treating a 50-character substring overlap as its contamination signal. Meta's Llama 2 paper takes a different technical approach again, matching on tokenized prompts directly rather than raw character or word n-grams. What all three converge on is the same underlying goal — catching near-verbatim overlap between what a model trains on and what it will later be tested on — using different specific mechanics.

The stakes here are not hypothetical. Published contamination analyses have reported meaningful overlap even in major model releases — Llama 2's own contamination analysis found over 10% of MMLU benchmark samples were highly contaminated in at least one of the models they studied, and GPT-4's technical report documented roughly 25% of the HumanEval coding benchmark showing contamination in their training data. These are the model developers' own disclosed findings, not third-party accusations — which is itself the point: contamination is common enough at web scale that treating detection as a required gate, and disclosing what's found, has become standard practice among major labs rather than an exceptional admission.

✅ Worked example: before reporting scores on a coding benchmark, a team runs an n-gram overlap check between their training corpus and the benchmark's public test set, and finds three of the benchmark's exact solutions embedded in a scraped forum thread. Those specific documents get removed before training begins — the same kind of catch Section 3's deduplication gate makes on repeated text, just checked against an external test set instead of the corpus itself.

💡 Key warning: n-gram and character-overlap methods catch near-verbatim overlap but can miss contamination that's been paraphrased or translated — text that conveys the same benchmark question in different words won't trigger a string-matching check, which is why some teams supplement n-gram detection with embedding-similarity or model-based methods for a second layer of coverage.

🎯 Use this when: any benchmark you plan to report scores on has a chance of appearing in your training data's source pool (which, for anything web-scraped, is essentially always) — run the check before training, since a contaminated score discovered after publication is a credibility problem, not just a data problem.

9. Hands-On Lab: Run a Mini Gate Pipeline Yourself

This lab uses a tiny, disposable list of sentences you type yourself — no real dataset, no risk, and you'll see three of the seven gates catch something in real time.

1

Write down eight short "training candidate" sentences of your own invention. Make two of them identical on purpose, make two more say nearly the same thing in different words (like "The cat sat on the mat" and "A cat was sitting on the mat"), and pick one to serve as your pretend "benchmark test question" — then copy that exact sentence, word for word, into your list of eight as if it had leaked in.
Expect to see: a list that looks like eight distinct sentences at a glance.

2

Run the exact-duplicate check by hand: compare every sentence to every other sentence, character for character.
Expect to see: your two identical sentences flagged immediately — this is the Section 3 deduplication gate's coarsest pass, exact matching, working exactly as intended.

3

Now look for near-duplicates: which two sentences convey the same meaning using different words?
Expect to see: your "cat on the mat" pair standing out once you're looking for meaning rather than exact text — this is the harder case from Section 3's callout and the same idea behind Section 6's ROUGE-L instruction filtering, just done by eye instead of by a similarity score.

4

Finally, run the contamination check: compare your one designated "benchmark question" against the rest of your list for an exact match.
Expect to see: it matches itself, obviously — but notice that if you'd only paraphrased the benchmark question instead of copying it exactly, this simple check would have missed it entirely, which is precisely Section 8's warning about paraphrased contamination slipping past n-gram methods.

Bridging to production: you just ran, by hand, on eight sentences, a simplified version of what Dolma's Bloom-filter deduplication and Self-Instruct's ROUGE-L filtering do automatically across billions of documents and tens of thousands of instructions. The gap between "eight sentences checked by eye" and "billions of documents checked by a Bloom filter" is scale, not concept.

10. Rolling This Out at Enterprise Scale

Data quality gates need the same operational discipline as any other production pipeline once more than one team or training run depends on them. Concretely, that means a versioned corpus never reaches a training run without first passing through a CI-gated flow:

1. A dataset or gate-configuration change is proposed: a new data source is added, a toxicity threshold is adjusted, or the contamination benchmark list is updated.
2. The change runs against a fixed regression sample first: a held-out set of known-good and known-bad examples for each gate, small enough to run quickly, checks that the change catches what it should and doesn't reject what it shouldn't.
3. A human reviews the regression results and the diff in what got filtered: not just whether the regression sample passed, but a sample of what changed in the full corpus as a result, since a technically-passing change can still shift filtering behavior in unintended ways.
4. The full gate pipeline runs against the entire corpus, producing a new versioned artifact: the resulting training-ready corpus is tagged with exactly which gate configuration produced it, never overwriting the previous version silently.
5. Only a versioned, gate-passed artifact is eligible to be used in a training run: a training job references a specific corpus version explicitly, so what was actually trained on is always reconstructable after the fact.

Around that flow, a handful of governance questions come up repeatedly at scale:

• Gate ownership and dataset versioning: every training corpus that passes through the gates should be versioned as a distinct artifact, with a record of exactly which gate configuration (which toxicity threshold, which contamination method, which benchmark list) produced it — so a quality issue found later can be traced to a specific version rather than "sometime last quarter."
• Access control and data governance: raw pre-gate data frequently contains unfiltered, unreviewed content — including potentially toxic material and unverified-provenance documents — so access to it needs tighter controls than access to the cleaned, post-gate corpus.
• Cost governance for classifier-based gates: toxicity classifiers, quality classifiers, and LLM-based filtering steps all cost compute per document, and that cost scales linearly with corpus size — track cost per gate separately so a classifier that's technically accurate but prohibitively expensive at full corpus scale gets caught in planning, not mid-run.
• Observability for gate pass/fail rates: track what fraction of incoming data each gate rejects, over time and per source. A sudden spike in the format-consistency gate's rejection rate, for instance, usually means an upstream data-generation process changed — that's exactly the kind of drift worth alerting on rather than discovering when a training run comes up short on usable examples.
• Contamination re-checks on benchmark updates: when a new benchmark is adopted for reporting, re-run contamination detection against the existing corpus for that benchmark specifically — a corpus that was clean against last year's benchmark list isn't automatically clean against this year's.

🎯 Use this when: more than one training run or team draws from the same underlying corpus — at that point, an undocumented or unversioned gate configuration becomes a hidden dependency everyone downstream inherits.

11. Common Mistakes

• Treating "the corpus passed our one aggregate quality score" as sufficient. A single blended quality number can look healthy while masking a corpus that's badly contaminated on one benchmark and severely under-covered in one language — the seven gates in this post exist precisely because one score can't represent seven independent failure modes.
• Running contamination detection only after an expensive training run, not before. Checking benchmark overlap as a post-hoc audit means the compute is already spent by the time contamination is found — the whole value of this gate comes from running it before training starts, not from confirming a problem afterward.
• Stopping at document-level deduplication and skipping paragraph-level. As Section 3 showed, document-level checks miss repeated boilerplate embedded inside otherwise-unique documents — a corpus can look fully deduplicated by one measure while still containing enormous amounts of repeated text at the paragraph level.
• Letting a synthetically generated instruction set skip similarity filtering because "the model wrote it, so it must be diverse." Section 6's example showed the opposite is often true — generated instructions cluster around a few underlying patterns unless a similarity filter is actively applied.
• Using a stale toxicity or quality classifier indefinitely. Language, slang, and harmful-content patterns shift over time; a classifier validated once and never re-checked against current examples can quietly drift out of alignment with what it's actually supposed to catch.
• Not versioning gate configurations alongside the datasets they produced. When a gate's threshold or logic changes without a corresponding version record, nobody downstream can answer "what exactly was filtered out of this corpus, and by what rule" months later, when it matters most.

❓ FAQ

Do all seven gates need to run in the exact order listed in this post?

Not strictly, but cost-conscious pipelines generally run cheaper, coarser checks (like exact-match deduplication) before more expensive ones (like classifier-based toxicity screening or contamination detection), for the same reason Dolma's staged deduplication runs URL-level checks before paragraph-level ones — it's more efficient to shrink the corpus early with cheap checks before running expensive ones on what remains.

Is a 13-gram or 50-character overlap the "correct" threshold for contamination detection?

There isn't one universally correct threshold — GPT-3, GPT-4, and Llama 2 each documented a different specific method and threshold. What matters more than picking the exact same number is running some form of overlap detection at all, and being transparent about which method and threshold were used when reporting results.

Can deduplication accidentally remove content that should have been kept?

Yes, particularly at the near-duplicate stage. Two documents that are mostly similar but differ in one meaningfully updated detail can be wrongly treated as duplicates if a similarity threshold is set too aggressively — this is why deduplication methods are usually tuned and validated against real examples rather than set to a maximally strict default.

Does passing all seven gates guarantee a high-quality training corpus?

No. These gates catch specific, checkable failure modes — they don't guarantee the content is accurate, well-written, or ideally balanced for a given task. They're a necessary floor, not a complete substitute for downstream evaluation of the resulting model.

Why did major model developers publish their own contamination findings instead of hiding them?

Contamination is common enough at web scale that finding some is close to expected, not a sign of a uniquely flawed process. Disclosing the method and the measured overlap lets others interpret reported benchmark scores appropriately, which has become the more credible path than presenting scores without that context.

🔗 References & Further Reading

• Allen Institute for AI — Dolma: 3 Trillion Token Open Corpus for Language Model Pretraining (Ai2 Blog)
• Allen Institute for AI — Dolma Toolkit (GitHub, Ai2)
• Soldaini et al. — Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (arXiv, Ai2)
• Wang et al. — Self-Instruct: Aligning Language Models with Self-Generated Instructions (ACL Anthology)
• Brown et al. — Language Models are Few-Shot Learners (arXiv, OpenAI's GPT-3 paper)
• OpenAI — GPT-4 Technical Report (arXiv)
• Touvron et al. — Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv, Meta)
• OpenAI — Fine-tuning (OpenAI API Docs)

All product, company, and dataset names (OpenAI, GPT, Meta, Llama, the Allen Institute for AI, Dolma, and others referenced) are trademarks or names of their respective owners.

📝 Summary

• Data quality gates catch problems before training, because a training run can't be selectively un-learned afterward.
• Provenance records where every document came from and whether it's actually usable — ideally tracked per document, not just per source.
• Deduplication runs in coarse-to-fine stages — URL, document, then paragraph — so cheap checks shrink the corpus before expensive ones run.
• Toxicity screening blends rules and classifiers, and its threshold is a genuine trade-off, not a "set once" default.
• Format consistency is a structural contract, validated before training starts rather than discovered mid-run.
• Instruction hygiene uses similarity filtering (like ROUGE-L) to stop a dataset from looking diverse while actually repeating itself.
• Coverage is a deliberate, measured design decision against a defined target, not whatever the raw data happened to contain.
• Contamination detection checks the corpus against the exam beforehand — every major lab referenced here found and disclosed real overlap.
• The hands-on lab lets you trigger three of these gates yourself, by hand, on eight sentences.
• At enterprise scale, gates need the same versioning, CI-gating, and observability discipline as any other production pipeline.

That's data quality gates end to end — from a single mislabeled sticker in a scrapbook to a Bloom filter checking billions of documents. Go check what's actually in the corpus before the training run starts, not after. 🚪