Speaker
Description
High Energy Photon Source（HEPS）is expected to generate a huge amount of data, which puts extreme pressure on data I/O in computing tasks. Meanwhile, inefficient data I/O significantly affect computing performance.
Firstly, the data reading mode and limitations of computing resources are taken into account，and we propose a method for automatic tuning of storage parameters, such as data block size, for optimizing the data reading speed.
Secondly, we designed a data processing pipeline scheme for reducing I/O latency and maximizing IO bandwidth utilization while processing high-throughput data. The computing task is split into multiple steps, i.e., data loading, data preprocessing, data processing and data writing, which are executed asynchronously and in parallel.
Finally, due to the limited storage, the lossless compression methods are applied for further optimizing the I/O speed. However, it will incur additional performance overhead. Thus, we put forward an intelligent lossless compression method, which judges whether compression is beneficial to data I/O, and compresses the suitable data to reduce I/O footprint and required storage resources.
| Consider for long presentation | No |
|---|
