Streaming readout gives opportunity to streamline workflows and to take advantage of other emerging technologies, e.g., artificial intelligence (AI) or machine learning (ML). In the INDRA-ASTRA project, we have explored the possibility for automated calibrations using AI / ML which would allow a rapid turnaround from data taking to physics results.
We introduce the ADWIN algorithm for detecting changes in streaming data and present an example implementation of ADWIN to detect sudden and gradual changes in a sample of ZEUS MC data. We compare the algorithm between one-dimensional and two-dimensional cases and give an example of integrating a simple, near-real time calibration method with ADWIN.
Optimal confidence values for controlling the rate of detecting anomalies
Adapting ADWIN to use higher moments instead of the mean
Limiting the maximum window size to improve computational efficiency of ADWIN
Change Detection Based on Multiscale Basis30m
We have used the Multiscale basis to detect the sudden change. For the sudden change, the mean information of data is enough to detect the change. In the next step, we plan to detect the gradual change, this requires us to explore the higher moment information of the data. The corresponding multiscale basis also has a higher vanishing moment property.
The figure 'mean_variance_change_detect.jpg' shows the detect results for the dataset 'hit_bbgem_1034.root'. We use the mean and standard deviation methods to detect the outlier. The results show that we can detect all the peaks that can be observed by humans.
The figure 'merge_shift_mean_variance.jpg' shows the results for the gradual change. We create simple gradual change and use the test function with order 1 vanish moment. The results show that gradual change can be detected in a large scale. At this moment, the algorithm can not verify the type of change (sudden change or gradual change).