Please visit Jefferson Lab Event Policies and Guidance before planning your next event:
May 8 – 12, 2023
Norfolk Waterside Marriott
US/Eastern timezone

A High-Speed Asynchronous Data I/O Method for HEPS

May 11, 2023, 12:00 PM
Marriott Ballroom V-VI (Norfolk Waterside Marriott)

Marriott Ballroom V-VI

Norfolk Waterside Marriott

235 East Main Street Norfolk, VA 23510
Oral Track 2 - Online Computing Track 2 - Online Computing


Fu, Shiyuan (IHEP)


High Energy Photon Source(HEPS)is expected to generate a huge amount of data, which puts extreme pressure on data I/O in computing tasks. Meanwhile, inefficient data I/O significantly affect computing performance.

Firstly, the data reading mode and limitations of computing resources are taken into account,and we propose a method for automatic tuning of storage parameters, such as data block size, for optimizing the data reading speed.

Secondly, we designed a data processing pipeline scheme for reducing I/O latency and maximizing IO bandwidth utilization while processing high-throughput data. The computing task is split into multiple steps, i.e., data loading, data preprocessing, data processing and data writing, which are executed asynchronously and in parallel.

Finally, due to the limited storage, the lossless compression methods are applied for further optimizing the I/O speed. However, it will incur additional performance overhead. Thus, we put forward an intelligent lossless compression method, which judges whether compression is beneficial to data I/O, and compresses the suitable data to reduce I/O footprint and required storage resources.

Consider for long presentation No

Primary authors

Fu, Shiyuan (IHEP) Wang, Lu (Imstitute of High Energy Physics Chinese Academy) Cheng, Yaodong Hu, Yu (IHEP, CAS) Liu, Rui (Institute of High Energy Physics, CAS) Liu, Jianli (Institute of High Energy Physics, CAS) Sun, Haokai (Institute of High Energy Physics, CAS) Wang, Lei ( Institute of High Energy Physics, CAS) Wang, Shuang ( Institute of High Energy Physics, CAS) Qi, Fazhi ( Institute of High Energy Physics, CAS)

Presentation materials

Peer reviewing