Speaker
Description
Since 2009 Port d’Informació Científica (PIC), in Barcelona, has hosted the Major Atmospheric Gamma Imaging Cherenkov (MAGIC) Data Center. At the Observatorio del Roque de Los Muchachos (ORM) in the Canary Island of La Palma (Spain), data produced from observations by the 17m diameter MAGIC telescopes are transferred to PIC on a daily basis. More than 200 TB per year are being transferred to PIC in order to be stored, distributed, analyzed, or reprocessed.
In 2012, PIC developed a data transfer system based on the CERN File Transfer Service (FTS) to orchestrate the data ingestion, transfer submission, data transfer and registration on the file catalog. This system has been running for a decade managing the data transfers to the disk and tape storage endpoints with no major problems, except some manual intervention when some network or database problem appeared.
We present a new transfer system for MAGIC based on RUCIO, an open-source framework to organize and manage large scale datasets, originally developed in the context of the ATLAS experiment at CERN. RUCIO has been adapted to the particularities of the MAGIC context including following the transfer guidelines in the MAGIC namespace. Moreover, the RUCIO scripts are able to update the transfer status from the Data Transfers database consistently.
The RUCIO instance is deployed on top of a Kubernetes cluster based on a Continuous Integration pipeline on Gitlab and interfacing with the local FTS instance. All of them managed and deployed at PIC. In this contribution we will present the main upgrades, functionalities and performance of using RUCIO orchestration in comparison with the previous FTS based system. We will pay special attention to the aspects introducing the robustness for the MAGIC use case.
| Consider for long presentation | Yes |
|---|
