Please visit Jefferson Lab Event Policies and Guidance before planning your next event:
May 8 – 12, 2023
Norfolk Waterside Marriott
US/Eastern timezone

Extracting Columnar Event Data From Experiment Specific Data Formats At Scale

May 11, 2023, 11:30 AM
Hampton Roads Ballroom VIII (Norfolk Waterside Marriott)

Hampton Roads Ballroom VIII

Norfolk Waterside Marriott

235 East Main Street Norfolk, VA 23510
Oral Track 6 - Physics Analysis Tools Track 6 - Physics Analysis Tools


Galewsky, Ben (University of Illinois)


We will describe how ServiceX, an IRIS-HEP project, generates C++ or python code from user queries and orchestrates thousands of experiment-provided docker containers to filter and select event data. The source datafiles are identified using Rucio. We will show how the service encapsulates best practice for using Rucio and helps inexperienced analysers get up to speed quickly. The data is returned as flat root or parquet files in an object store. We will show how ServiceX is deployed into a modern analysis facility as part of a scalable analysis workflow. Recent work includes support for CMS MiniAOD files, robust failover across file replicas, CERN OpenData support, Autoscaling of worker pods.

Consider for long presentation No

Primary authors

Galewsky, Ben (University of Illinois) Watts, Gordon (University of Washington) Dr Vukotic, Ilija (University of Chicago)

Presentation materials