Please visit Jefferson Lab Event Policies and Guidance before planning your next event: https://www.jlab.org/conference_planning.

May 8 – 12, 2023
Norfolk Waterside Marriott
US/Eastern timezone

Evaluation of Rucio as a Metadata Service for the Belle II experiment

May 9, 2023, 4:45 PM
15m
Norfolk Ballroom III-V (Norfolk Waterside Marriott)

Norfolk Ballroom III-V

Norfolk Waterside Marriott

235 East Main Street Norfolk, VA 23510
Oral Track 1 - Data and Metadata Organization, Management and Access Track 1 - Data and Metadata Organization, Management and Access

Speaker

Panta, Anil (University of Mississippi)

Description

Rucio is a Data Management software that has become a de-facto standard in the HEP community and beyond. It allows the management of large volumes of data over their full lifecycle. The Belle II experiment located at KEK (Japan) recently moved to Rucio to manage its data over the coming decade (O(10) PB/year). In addition to its Data Management functionalities, Rucio also provides support for storing generic metadata. Rucio metadata already provides accurate accounting of the data stored all over the sites serving Belle II. Annotating files with generic metadata opens up possibilities for finer-grained metadata query support.
We will first introduce some of the new developments aimed at providing good performance that were done to cover Belle II use-cases like bulk insert methods, metadata inheritance, etc. We will then describe the various tests performed to validate Rucio generic metadata at Belle II scale (O(100M) files), detailing the import and performance tests that were made.

Consider for long presentation No

Primary authors

Serfon, Cédric Panta, Anil (University of Mississippi) Ito, Hiro (Brookhaven National Laboratory) De Stefano Jr, John Steven (Brookhaven National Laboratory) Hernandez Villanueva, Michel (DESY) Mashinistov, Ruslan Laycock, Paul (BNL) Miyake, Hideki (KEK/IPNS) Ueda, Ikuo

Presentation materials

Peer reviewing

Paper