Please visit Jefferson Lab Event Policies and Guidance before planning your next event: https://www.jlab.org/conference_planning.

Streaming Readout III

US/Eastern
Room 201 (Freeman Center at Christoper Newport University)

Room 201

Freeman Center at Christoper Newport University

Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
Description

Meeting of the EIC Streaming Readout Consortium

Organizers E. Brash (CNU), C. Cuevas (JLab), M. Diefenthaler (JLab), G. Heyes (JLab)

Remote meeting via BlueJeans (Meeting ID 493409952): 

Join via browser or dial in via phone: (888) 240- 2560 (see also international numbers). 

Previous workshops (part of EIC Ad-hoc Meeting Series): 

01/2018 Streaming Readout II

01/2017 Trigger/Streaming Readout

CNU - Campus Map
CNU - Campus Map in 3D
Group Picture
JLAB - Map
Streaming Readout IV - Announcement
  • Monday, December 3
    • 9:00 AM 10:00 AM
      Welcome Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
      • 9:00 AM
        Welcome 15m
      • 9:15 AM
        Status of EIC Streaming Readout Initiative 30m
        Summary of Streaming Readout II workshop, EIC R&D and EICUG meetings, and funding situation
        Speaker: Dr Douglas Hasell (M.I.T.)
        Slides
      • 9:45 AM
        Discussion 15m
    • 10:00 AM 12:00 PM
      Data control and validation Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
      Convener: Dr Marco Battaglieri (INFN-GE)
      • 10:00 AM
        CBM streaming readout DAQ and validation 30m
        Speaker: Dr Volker Friese (GSI)
        Slides
      • 10:30 AM
        BDX triggerless DAQ and validation 30m
        Speaker: Dr Andrea Celentano (INFN-Genova)
        Slides
      • 11:00 AM
        Discussion 1h
        Slides
    • 12:00 PM 2:00 PM
      Lunch break (on your own) 2h Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
    • 2:00 PM 6:00 PM
      Electronics, timing and synchronization Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
      Convener: Chris Cuevas (Thomas Jefferson National Accelerator Facility)

      Electronics, timing and synchronization session summary

      The afternoon session on Monday 3 December was completely filled with presentations that covered the session topics. 

      Streaming Electronics

      Ben Raydo from Jefferson Lab covered existing triggered DAQ systems at JLAB, and presented the existing performance for high trigger rates [100Khz] and large data rates approaching 1GB/s for Hall D during the fall 2018 run period.  The existing JLAB DAQ front end modules were described and examples of the CLAS12 trigger system were explained in detail.

      Designing the front end hardware using ASIC in a true streaming mode cannot have ASIC that require a trigger signal to begin digitization or to begin reading buffered data. There are several new ASIC companies including large commercial companies that have high speed waveform digitizers that will work in streaming mode. Ben gave examples where ASICs are used to readout existing 12GeV detectors, including possible ASICs that we know are in development.

      Plans for a full VXS crate of 250Mhz 12bit ADC boards [256 channels] were explained and this test will show the full streaming performance including new software that will handle the new data flow. This test will be setup in the new INDRA lab area. If this test works well, the full CLAS12 DAq could be tested with this mode, and rough estimates show data rates could approach 50GB/s.

      A multi-channel digitizer board for the BDX experiment

      Fabrizio Ameli delivered a presentation that was focused on the new digitizer board that was inherited from the KM3Net experiment. The BDX experiment specifications were listed and this front end board contains 12 channels of 14-bit/250Msps ADC data. The board is designed with a trigger-less method and contains a time-variable window and includes a zero-suppression feature. 

      The board interfaces directly to front end SiPM and the bias voltages are generated on-board. There is a commercial System-On-Module[SOM] mezzanine card that contains a FPGA which is based on the Zynq-7045 from Xilinx. This SOM manages the fiber transceiver data including the Ethernet RJ45 and USB 2.0 interfaces. Other board features includes flexible timing resources including White Rabbit, and PLL clock interface that is shown to be highly stable with extremely low jitter to daisy-chain the timing signaling with multiple boards. Other high speed interfaces are provide also, such as SATA connections. Test results were presented for the bias voltage{HV] monitoring feature and clock timing jitter.

      Future work will focus on timing resolution with waveform reconstruction, DMA implementation for higher throughput and developing slow controls for EPICS or other standards.

      Status of the JLAB Streaming Data Acquisition Test Stand

      Ed Jastrzembski gave an update on the streaming DAQ test stand that the Jefferson Lab Data Acquisition Group is building. It uses the new SAMPA ASIC developed for the ALICE experiment at the LHC.  The 32-channel SAMPA chip has sophisticated DSP features and is designed to be read out continuously.  Jefferson Lab is interested in seeing how they can take advantage of this new technology and the continuous readout concept in their experiments.

      Rather than design a system from scratch to support the SAMPA, the Jefferson Lab group chose to exactly copy the ALICE readout architecture.  On a Front End Card (FEC) data from five SAMPAs is concentrated onto two high speed optical links (GBT) that connect to a Common Readout Unit (CRU).  The links also carry timing and synchronization signals from the CRU to the FEC.   A single CRU can receive data from up to 24 FECs.  The ALICE designed CRU was not available so a functionally equivalent readout unit designed for the ATLAS experiment is used instead (FELIX CRU). The Jefferson Lab test stand is now assembled and consists of five ALICE FECs (800 channels) connected to a single FELIX CRU. 

      At this point the Jefferson Lab group is learning how to configure and read out data from the SAMPA chips.  Their goal is to connect the system to a prototype GEM detector at the lab.  They will study the SAMPA’s response to detector signals and find out how to best utilize the chip’s DSP features.  They will also learn how to deal with the continuous data stream produced.

      Streaming Readout and Timing

      William Gu from Jefferson Labpresented recent test results using a Xilinx KCU1500 accelerator card in conjunction with a Dell host computer. 

      1. Using the Dell 7920 computer, we can achieve 5200 MB/s read from and 5600 MB/s write to (simultaneously) the KCU1500 FPGA accelerator card via DMA and PCIe3x8.  
      2. The 10 Gbe Ethernet can be easily implemented.  
      3. A multi-channel streaming TDC using a data format with the time stamp up to many seconds was implemented on the VETROC board.  
      4. The clock and synchronization ideas were presented.

      Further work in the next few months (or years):  First: combine the above mentioned work (1) and (3), and stream the data to computer for streaming readout software development in the ASTRA lab.  Second: do some performance tests (distributed clock jitters, synchronization precision etc.) using some existing hardware.  Third: produce a (or several) new streaming TDC board to further test higher data throughput, clock distribution and synchronization, if budget permits. 

      Alphacore - Multi-Channel Readout ICs

      Phaneendra Bikkina, from Alphacore presented the company history and specifications for several custom ASIC for nuclear physics experiments. Several of these ASIC projects are at the DOE STTR Phase II level, with parts delivered in January 2019 for initial testing.  The chips will be qualified for performance of 12bit and 10bit performance at sample speeds of 100Msps and 50Msps respectively. Chips are fabricated with a 180nm CMOS process, which features radiation hardening for SEL, SEU and SEFI prevention. The Alphacore chips include charge sensitive preamplifiers, high speed ADCs and ADCs with JESD204B high speed readout.

      New developments include a continuous ADC with 8 channels, 10bit, with sampling speeds up to 10Gsps on a 28nm CMOS fabrication. The timing resolution is impressive at <1ps and relatively low power for an eight channel device. 

      Design Study for a sPHENIX based EIC Detector - Progress report

      Jin Hyuang from BNL presented a very comprehensive review of the sPHENIX design study with a view toward using the same electronics model for an EIC interaction area.  The sPHENIX detector includes 7 large detector systems ranging from the inner tracker to the outer HCal. These different detector types use different technologies and interface front end boards that can be readout using the FELIX2 PCIe DAq board to stream data from the different detectors. The front end interface chips are SAMPA for the TPC, and pixel sensors for the MVTX. This offers a very flexible but common method to provide a readout interface independent of the front end technology. 

      Jin presented a sPHENIX - EIC DAQ strategy based on the PCIe-based FPGA card [FELIX2] with full streaming data as the main concept to acquire all the front end data. The FELIX2 board has the capabilities to manage the front end data and is built on a commercial [PCIe] standard for computing hardware.

      Recent test stand results were presented for readout of a GEM + zigzag pad with the SAMPA front end and FELIX2 DAQ. Special SAMPA chips are requested for an 80ns shaping time and testing will continue as part of an R&D EIC project.

      Summary points are copied as follows:

      • PHENIX and sPHENIX builds long experience for streaming readout front end
      • We are exploring one way to build EIC detector streaming readout and trigger less DAQ based on architecture of sPHENIX DAQ. Looks promising
      • Preliminary simulation show disk rate for EIC collision signal (100 Gbps) expected lower than sPHENIX disk rate (200 Gbps)
      • Controlling background for high-L low-σ collisions would be important for collider, detector and DAQ designs.
      • BNL-712/FELIX type DAQ architecture fits EIC purpose. Similar architecture have wide support in 2020+ for high throughput DAQ e.g. ATLAS, ALICE, LHCb, CBM, Bell-2, and cost effectively bridges custom front end with commodity computing.
      • Productions planned for BNL-712/FELIX DAQ interface, more EIC R&D interests welcomed

      Discussion

      The presentations offered a wide view of the different hardware projects that are either extensions of previous work [JLAB 12GeV], or developments for new experiments that have streaming readout as the main focus[sPHENIX]. 

      We had contributions from a commercial ASIC company, Alphacore. This growing company  is dealing directly with our Physics community to develop modern high speed digitizer chips for front end systems that will include high speed serial readout options.

      For an EIC detector, the streaming readout format is clearly needed, and our efforts now continue to emphasize the details and definitions. For example, holding data in the front end electronics memory until a ‘trigger’ signal is received,  is a concept that should be well understood as not needed for a true streaming readout definition. Data suppression for high channel count detectors close to the virtex area may indeed need different control to begin streaming, but holding data and waiting for a ‘trigger’ is not desired.  The data rates from each of the EIC detector sub-systems has not been completely defined nor simulated, but these studies continue and the hardware specifications presented at this meeting show that EIC data rates are within our capabilities.

      Distribution of timing synchronization and precision clock signaling is of course required, but it is not clear yet on the best method of distribution nor the resolution and stability of these signals. Higher sampling ADCs will demand extremely low jitter [sub-ps], so close consideration of timing/synch distribution systems will be essential.

      There was discussion about adding definitions for the essential signaling of timing and synchronization to the draft EIC User Handbook. We have plenty of information for the DAQ section and hopefully we can add this information soon for review. 

      • 2:00 PM
        Streaming Electronics 30m
        Speaker: Benjamin Raydo (Jefferson Lab)
        Slides
      • 2:30 PM
        The BDX wave board 30m
        Speaker: Fabrizio Ameli (Sapienza University of Rome)
        Slides
      • 3:00 PM
        Status of the JLAB Streaming Data Acquisition Test Stand 30m
        Speaker: Dr Edward Jastrzembski (Jefferson Lab)
        Slides
      • 3:30 PM
        Streaming Readout and Timing 30m
        Speaker: William Gu (Jefferson Lab)
        Slides
      • 4:00 PM
        Alphacore - Multi-Channel Readout ICs Development Status 30m
        Speakers: Daniel Mazidi (Alphacore, Inc.), Phaneendra Bikkina (Alphacore, Inc.)
        Slides
      • 4:30 PM
        Design Study for a sPHENIX based EIC Detector - Progress on streaming demonstrations for EIC 30m
        Speaker: Dr Jin Huang (Brookhaven National Lab)
        Slides
      • 5:00 PM
        Discussion 30m
    • 6:00 PM 8:00 PM
      Dinner (on your own) 2h Smoke BBQ Restaurant & Bar

      Smoke BBQ Restaurant & Bar

      10900 Warwick Blvd, Newport News, VA 23601.
  • Tuesday, December 4
    • 9:00 AM 10:00 AM
      INDRA tour at Jefferson Lab F110 (Turn right at cafe, through glass doors continue ahead past elevators). (CEBAF Center at Jefferson Lab)

      F110 (Turn right at cafe, through glass doors continue ahead past elevators).

      CEBAF Center at Jefferson Lab

      Conveners: David Abbott (Jefferson Lab), Dr Graham Heyes (Jefferson Lab)
    • 10:00 AM 12:00 PM
      Streaming Readout Software L102 (CEBAF Center at Jefferson Lab)

      L102

      CEBAF Center at Jefferson Lab

      Convener: Dr Markus Diefenthaler (Jefferson Lab)
      • 10:00 AM
        TRIggerles Data acquiSition (TRIDAS) for the KM3NET experiment 30m
        Speaker: Prof. Tommaso Chiarusi (Istituto Nazionale Fisica Nucleare - Sezione di Bologna)
        Slides
      • 10:30 AM
        Software aspects of streaming readout 30m
        Speaker: Dr Markus Diefenthaler (Jefferson Lab)
        Slides
      • 11:00 AM
        Discussion 1h
    • 12:00 PM 2:00 PM
      Lunch break and return to CNU 2h Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
    • 2:00 PM 4:00 PM
      Intermediate network and streaming data Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
      Convener: Dr Jan Bernauer (Stony Brook University / RBRC)
      • 2:00 PM
        Intermediate network and MPEG presentation Layer 45m
        Speaker: Dr Jan Bernauer (Stony Brook University / RBRC)
        Slides

        The Tuesday afternoon session was dedicated to Intermediate network and streaming data.


        The first talk, a combination of talks on  "Intermediate network" and "MPEG program stream" was given by Dr. Bernauer. Any streaming network can be logically represented by a data flow network, connecting data "sources" (detectors) to "filters" which receive one or more data streams, process them, and produce one or more new streams, and data "sinks", which receive data for display or storage. The talk took "stream merging" as a common example for a filter and discussed different merging approaches.
        In the second part, the OSI network layer was discussed with example solutions for each layer. The presentation layer is identified as the layer a common protocol has to be defined for to allow inter-operability. Lower layers can be implementation dependent, but the talk recommends ethernet and TCP/IP as a good default.
        The remainder of the talk discussed the MPEG standard. While not directly usable as a presentation layer for a streaming readout, some design decisions are applicable, ensuring efficient implementations in software and hardware.

        The discussion focused mainly on the question of congestion, i.e. how a situation should be handled in which the data transport network is too slow and data has to be discarded. No clear solution was found and this points warrants for investigation in the future.


        In the second talk, Dr. Blyth discussed ProIO, an event-based I/O stream format based on ProtoBuf. ProIO is possible contender for a presentation layer. The use of ProtoBuf makes interfacing with many languages easy, as template accessor functions and data structures can be generated automatically.
        He compared speed and data size with ROOT, ProIO is on-par or exceeds the speed of root in the selected benchmark.

        In the discussion, mainly features and (few) constraints of ProIO were discussed.

        The question of presentation layer is important, and the required feature set is not quite clear.  Markus Diefenthaler and Jan Bernauer will develop a protoype to gain experience and further insight into the requirements.

         

         

      • 2:45 PM
        ProIO 30m
        Speaker: Dr David Blyth (Argonne National Laboratory)
        Slides
      • 3:15 PM
        Discussion 45m
    • 4:00 PM 5:00 PM
      Closing Room 201

      Room 201

      Freeman Center at Christoper Newport University

      Christoper Newport University Freeman Center, Room 201 1 University Plaza Newport News, VA 23606, USA
      Conveners: Dr Jan Bernauer (Stony Brook University / RBRC), Dr Marco Battaglieri (INFN-GE)