Data leakage is when information reaches training or evaluation that will not legitimately be available at prediction time. The model looks brilliant offline and falls apart in production, and the gap between those two numbers is not noise or drift or bad luck. It is that the exam was compromised. Leakage is the most reliable producer of results that are too good to be true in all of applied machine learning, and the reason it deserves an essay in 2026 is that the LLM era did not retire it. It gave it new places to hide.
The classic forms first, because they still account for most of the damage, and because the new forms are the same diseases in new tissue.
Target Leakage: The Feature That Knows the Answer
Target leakage is a feature that is a consequence of the label rather than a predictor of it. The churn model with a cancellation_reason column. The readmission model trained on fields the hospital only fills in after readmission. The fraud model using an account-status flag that the fraud team sets while investigating. In each case the pipeline ran clean and the AUC came back spectacular, because one column is the label wearing a hat.
The mechanics are always timing. A production feature store asks "what did we know at the moment of prediction," while a training-set query against a warehouse asks "what do we know now," and every field that was updated between those two moments is a potential leak. The tell is distinctive: one feature towers over the others in importance, and the metric lands above anything the problem is known to support. A churn model at 99 percent AUC is not a breakthrough, it is a subpoena for your feature list. When a result would earn someone a promotion, audit it before anyone presents it.
Split Leakage: Fit Nothing Before You Split
The second family is contamination across the train-test boundary, and it has four recurring shapes.
Preprocessing before splitting. Fit a scaler, an imputer, a vocabulary, or feature selection on the full dataset and then split, and statistics of the test set have already shaped what the model trains on. The leak is small per transform and it compounds, and the fix is mechanical: every fitted transform lives inside the training fold, which is what pipeline abstractions are for.
Duplicates and near-duplicates. Real datasets are full of copies: the same document ingested twice, the same customer under two IDs, boilerplate paragraphs shared across records. Split randomly and the model gets quizzed on items it memorized from the other side. Deduplicate before splitting, and near-deduplicate for text, because a paraphrase is a duplicate as far as generalization is concerned.
Group leakage. Multiple rows from the same patient, user, or device land on both sides of the split, and the model learns to recognize the patient instead of the disease. Split by group, always, and treat any dataset with repeated entities as grouped until proven otherwise.
Temporal leakage. A random split on time-ordered data trains the model on Thursday to predict Tuesday. Features built with rolling windows leak the future just as effectively if the window was computed over the full series. Time-based splits are slower to set up and they are the only honest option for anything that will be deployed to predict forward.
The LLM Era Made It Sneakier
None of the above went away. What changed is that teams who would never random-split a time series now commit the same crimes one layer up, where the tooling does not flag them.
Benchmark contamination is train-test contamination at pretraining scale. Public benchmarks live on the public internet, pretraining corpora are the public internet, and so the famous exams end up inside the student. This is a known, studied problem, it is why vendors report on freshly authored private sets when they want to be believed, and it is why a model's delta on a public leaderboard transfers so unreliably to your private workload. The practical rule costs nothing: never make a model decision on public benchmark numbers alone. Your own task, your own data, your own eval set, every time.
Prompt iteration on your eval set is the leak teams commit personally, by hand, while feeling rigorous. You run the golden set, read the failures, tweak the prompt, run it again, and repeat until the number is good. That loop is gradient descent with a human in the middle, and after enough iterations the prompt is fitted to those specific fifty cases, upweighted by every quirk they happen to contain. The score is now partly memorization, and the first genuinely new input finds the gap. The fix is the oldest trick in the field, applied to prompts: a dev set you iterate against and a held-out set you touch rarely, refreshed as it wears out. With eval sets this small, wear-out is fast, and the sample sizes involved were marginal to begin with.
Fine-tuning on your own exam is the same thing with weights. Training data assembled from the same pool as the eval set, near-duplicates across the boundary, or yesterday's eval failures helpfully folded into tomorrow's training run. The remedy is also the classic one: deduplicate across the boundary with n-gram overlap or embedding similarity before any run, and quarantine the eval pool from the data pipeline entirely, because a set that feeds training is not an eval set anymore.
RAG Benchmarks Leak by Construction
Retrieval evals have a leakage mode of their own that almost nobody prices in: the questions are usually written by someone looking at the documents. A question authored from a chunk shares its vocabulary, its phrasing, and its framing, so the retriever is being tested on paraphrase matching, which is the easy case. Real users ask questions in their own words, about their own situations, with the document's terminology nowhere in sight, and retrieval that aced the paraphrase exam goes noticeably dumber.
I will eat my own cooking here: my chunking benchmark runs on SQuAD, whose questions were written by crowdworkers reading the passages. The comparisons survive, because every splitter faces the same questions, and the relative gap between fixed and structure-aware chunking is the honest finding. The absolute recall numbers, though, flatter every strategy on the board, and your production recall will be lower than any benchmark built this way predicts. When you build your own retrieval eval, source at least some questions from real search logs or real tickets, written before the answer was in front of anyone. Those questions are worth ten synthetic ones each.
The Checklist That Catches Most of It
Leakage is not exotic, and the defenses are not clever. They are a short list applied without exceptions.
Split first, fit later: no transform sees anything outside its training fold. Split by time when there is time, by group when there are repeated entities, and deduplicate, including near-duplicates, before the split exists. Keep a held-out set that iteration never touches, for prompts exactly as for models, and refresh it on a schedule. Check train-eval overlap mechanically for any fine-tune. Distrust public benchmark deltas on principle, and distrust your own numbers hardest when they are best: the emotional signature of leakage is delight. A model that suddenly clears the ceiling did not usually get smart. It usually got the answers.
The through-line from a leaky churn feature in 2016 to a contaminated benchmark in 2026 is the same single question, asked at every boundary in the system: could this information have been here, at this moment, honestly? Ask it about every feature, every split, every prompt revision, and every training corpus, and most of this essay becomes unnecessary. Skip it, and your metrics will keep writing checks your production system cannot cash.
FAQ
What is data leakage in machine learning?
Information reaching training or evaluation that will not legitimately be available at prediction time. Metrics inflate offline and collapse in production. The two classic families are target leakage, a feature that encodes the label, and train-test contamination, where evaluation data influences training.
Target leakage vs. train-test contamination?
Target leakage is a feature problem: a column that is a consequence of the outcome, so the feature knows the answer. Contamination is a split problem: the same information lands on both sides, via duplicates, shared users, random splits on time series, or transforms fitted before splitting.
How do you detect data leakage?
Audit whatever looks too good: a dominating feature, a metric above the problem's known ceiling, a model that dies on fresh data. Mechanically: fit transforms inside folds only, split by time and by group, deduplicate first, and check train-eval overlap for text with n-gram or embedding similarity.
Is benchmark contamination data leakage?
Yes, contamination at pretraining scale: public benchmark items end up in the training corpus, so the model has seen the exam. It is why public benchmark deltas transfer poorly and why fresh, private eval sets beat famous ones for real decisions.