A PaperBridge long-tail question
How can you tell whether an AI paper may have data leakage?
Map the path from data collection to evaluation and check whether target information entered features, training data, prompts, tools, or human review before prediction. Common signals include related samples across splits, test questions in pretraining, random row splits for time data, and repeated test-set use during tuning.
Check whether split units are independent
Random rows do not isolate nearby samples from the same user, patient, device, document, or video. The model may recognize source-specific cues rather than learn the target ability.
Look for tests split by user, entity, time, or source. When a paper only says random split, inspect the appendix, code, or data statement for the unit.
Trace what pretraining and outside tools could access
Large-model pretraining corpora are rarely listed in full. Check the benchmark release date, web visibility, public answers, and whether the authors ran contamination detection or deduplication.
Retrieval systems, search engines, and evaluation agents may return a test answer directly. Separate model ability from tool access and memory.
Check tuning and human evaluation for answer exposure
Choosing prompts, thresholds, model versions, or post-processing after viewing test results turns the test set into a validation set. A final untouched test or hidden evaluation is stronger.
Human raters can also be biased when they see method names, reference answers, or earlier scores. Look for blinding, randomized order, and a scoring rubric.
Six signs of data leakage
- Do related entities or sources cross training and test?
- Does a future period form the time-series test?
- Could test questions and answers appear in pretraining?
- Can an outside retrieval tool find the answer directly?
- Did the test set guide prompt or threshold selection?
- Was human evaluation blinded and randomized?
Primary research and official documentation
These sources support the facts. Workflow and comparison guidance is PaperBridge's synthesis of research, official documentation, and engineering practice.