Deep Learning I/O Benchmark

Deep Learning I/O (DLIO) Benchmark is a benchmark suite aiming at emulating the I/O pattern and behavior of deep learning applications. The benchmark is delivered as an executable that can be configured for various deep learning workloads. It uses a modular design to incorporate different data loaders, data formats, dataset organizations, and use training configuration parameters similar to the actual deep learning applications. It is able to represent the I/O process of a broad spectrum of deep leanrning applications.

The main features of DLIO include:
  • Easy-to-use configuration through YAML files which represent the I/O process of different deep learing applications.

  • Easy-to-use data generator capable to generate synthetic datasets of different formats, different data organizations and layouts.

  • Full transparency over emulation of I/O access with logging and profiling at different levels with DFTracer.

  • Supporting emulating both sequential training and distributed data parallel training.

GitHub repo: https://github.com/argonne-lcf/dlio_benchmark.


How to contribute

Resources

Acknowdgments

Indices and tables