Please visit Jefferson Lab Event Policies and Guidance before planning your next event: https://www.jlab.org/conference_planning.

Indico is being upgraded to version 3.3.4 on October 15, 2024. There are no impacts to events expected. There are no major feature updates – only minor feature improvements and bugfixes. See the news link for more information.

May 8 – 12, 2023
Norfolk Waterside Marriott
US/Eastern timezone

Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory

May 11, 2023, 2:00 PM
15m
Marriott Ballroom II-III (Norfolk Waterside Marriott)

Marriott Ballroom II-III

Norfolk Waterside Marriott

235 East Main Street Norfolk, VA 23510
Oral Track 4 - Distributed Computing Track 4 - Distributed Computing

Speaker

Dr Karavakis, Edward (Brookhaven National Laboratory)

Description

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three days and produce tens of petabytes of raw image data and associated calibration data. More than 20 terabytes of data must be processed and stored every night for ten years.

The Production and Distributed Analysis (PanDA) system was evaluated by the Vera C. Rubin Observatory Data Management team and selected to serve the observatory’s needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, it’s support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed and alerts issued within 60 seconds.

This presentation will briefly describe the Vera C. Rubin Data Management system and its use at both the Interim Data Facility (IDF) hosted on the Google Cloud Platform (GCP) and the United States Data Facility (USDF) hosted at the SLAC Shared Scientific Data Facility (S3DF). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Vera Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.

Consider for long presentation No

Primary authors

Adelman-McCarthy, Jennifer (Fermi National Accelerator Laboratory) Barreiro Megino, Fernando (University of Texas at Arlington) De, Kaushik (University of Texas at Arlington) Dubois, Richard (SLAC National Accelerator Laboratory) Gower, Michelle ( National Center for Supercomputing Applications ) Guan, Wen (Brookhaven National Laboratory) Jenness, Tim (AURA/Rubin Observatory) Dr Karavakis, Edward (Brookhaven National Laboratory) Klimentov, Alexei (Brookhaven National Laboratory) Korchuganova, Tatiana (Universidad Andres Bello) Kowalik, Mikolaj (National Center for Supercomputing Applications) Lin, Fa-Hui (University of Texas at Arlington) Maeno, Tadashi (Brookhaven National Laboratory) Nilsson, Paul (Brookhaven National Laboratory) Padolski, Sergey (Brookhaven National Laboratory) Wenaus, Torre (Brookhaven National Laboratory) Yang, Wei (SLAC National Accelerator Laboratory) Yang, Zhaoyu (Brookhaven National Laboratory) Ye, Shuwei (Brookhaven National Laboratory)

Presentation materials

Peer reviewing

Paper