Storage and IO
Storage and IO describe how data moves from disk or network storage into CPU memory and then accelerator memory. Many training jobs are bottlenecked by input pipelines rather than model computation.
End-to-end step time can be approximated as:
If input time dominates, accelerators wait:
Key Ideas
- Small random reads stress metadata and latency; large sequential reads stress bandwidth.
- Network storage, local scratch, object storage, and memory caches have different failure modes.
- Preprocessing can shift the bottleneck from storage to CPU.
- Caching helps only when cache keys include data and preprocessing versions.
- Validation and inference should use the same data interpretation as training.
Data Path
A training input often follows:
Profiling should identify which stage is limiting throughput before changing architecture or batch size.
Cache Contract
A cached feature is valid only for a specific source and transform:
If any part changes, stale cached features can silently invalidate evaluation.
Failure Modes
- Many workers overload shared storage with small random reads.
- Local scratch is faster but not cleaned, versioned, or sized.
- Cached features are reused after preprocessing changes.
- Data loader masks corrupt or missing examples instead of failing visibly.
- Evaluation reads data through a different path than training.
Practical Checks
- Are files many small objects, large shards, archives, databases, or streaming records?
- Is the bottleneck metadata, bandwidth, decompression, CPU transform, or transfer to GPU?
- Are workers overloading shared storage?
- Is local scratch cleaned and sized for the job?
- Are cached features invalidated when raw data or preprocessing changes?
- Does the data loader surface corrupt, missing, or partial records?
- Are train, validation, and inference using the same data interpretation?
- Is throughput measured together with model utilization?