A PaperBridge long-tail question
How do you actually clean training data for a world model?
There is no universal world-model cleaning script. Define the state, action, and prediction target first; then validate decoding and timestamps, align modalities, deduplicate by scene, prevent split leakage, filter for task-relevant quality, audit rights and privacy, and test each rule with a small downstream ablation. For video world models, continuity and scene-level leakage often matter more than whether every frame looks pristine.
1. Define the sample before cleaning it
A video-prediction sample may be 16 or 32 consecutive frames. A robotics sample may contain observation, action, next observation, task instruction, and episode ID. If those fields are unstable, cleaning only organizes the noise.
Cross-robot data also needs consistent coordinate frames, action rates, gripper states, camera names, and task labels. Standardizing these schemas is a central part of Open X-Embodiment.
2. Remove structurally invalid records first
Check complete decoding, declared versus observed duration, monotonic timestamps, missing sensor frames, legal action ranges, successful resets, and end-to-end episode completeness.
Do not silently replace a missing action with zero. Zero may mean no movement, while a missing value means the causal record is unknown.
- Video: decode failures, black frames, frozen runs, broken frame rates, clips that are too short.
- Robotics: joint-limit violations, saturated actions, clock drift, failed resets, sensor dropout.
- Multimodal: text-video mismatch, instructions arriving after actions, incompatible units or coordinates.
3. Deduplicate and split at the scene level
Use exact hashes for identical files and perceptual signatures plus metadata for near-duplicate clips. More importantly, keep clips from the same source video, robot session, or driving run in one split group.
Random clip-level splitting can put nearly adjacent frames in training and test sets, producing an unrealistically high score.
4. Make quality task-dependent
Sharpness, compression, overlays, and cuts can be scored automatically, but thresholds should reflect deployment. Motion blur and occlusion may be valid operating conditions rather than disposable noise.
A useful taxonomy is clean, recoverable, hard-but-valid, and invalid. Delete invalid data, repair recoverable data, and retain hard valid examples with separate monitoring.
5. Verify that actions and consequences agree
For labeled actions, replay sampled transitions and verify direction, scale, and delay. With latent actions, as in Genie, excessive cuts, jumps, and loops still need attention because editing patterns can be mistaken for environment dynamics.
Failures, collisions, occlusions, and recovery behaviors should not all be removed. They are often the safety-critical tail and should be labeled and sampled deliberately.
6. Track rights, privacy, and provenance
Keep source, license or authorization state, collection time, cleaning version, and rule history. Faces, plates, screens, location, and speech may require removal, masking, or access controls.
Preserve immutable raw data and train from versioned manifests so that a problematic source can later be removed precisely.
7. Validate filters with small ablations
Every rule should improve something measurable: prediction error, control success, long-horizon consistency, or safety. DataComp demonstrates the useful pattern of holding training and evaluation fixed while comparing curation strategies.
Run no-filter, basic validation, deduplication, and task-filtered variants on a smaller model before scaling. A filter that only makes the dataset look tidy may reduce valuable diversity.
Minimum viable cleaning checklist
- Define the schema, target, episode, and scene grouping key.
- Validate decoding, duration, timestamps, missing frames, units, and action ranges.
- Deduplicate files and scenes; split by source or session group.
- Label cuts, freezes, occlusions, failures, and rare events.
- Audit authorization, privacy, source, and removal rights.
- Keep immutable raw data plus versioned manifests and rule reasons.
- Ablate each filter on a small model before scaling.
Primary research and official documentation
These sources support the facts. Workflow and comparison guidance is PaperBridge's synthesis of research, official documentation, and engineering practice.