Speaker
Description
INFN-CNAF is one of the Worldwide LHC Computing Grid (WLCG) Tier-1 data centers, providing computing, networking and storage resources to a wide variety of scientific collaborations, not limited to the four LHC experiments. The INFN-CNAF data center will move to a new location next year. At the same time, the requirements from our experiments and users are becoming increasingly challenging and new scientific communities have started or will soon start exploiting our resources. Currently, we are reengineering several services, in particular our monitoring infrastructure, in order to improve the day-by-day operations and to cope with the increasing complexity of the use cases and with the future expansion of the center.
This scenario led us to implement a data streaming infrastructure designed to enable log analysis, anomaly detection, threat hunting, integrity monitoring and incident response. Such a data streaming platform has been organized to manage different kinds of data coming from heterogeneous sources, to support multi-tenancy and to be scalable. Moreover, we will be able to provide an on demand end-to-end data streaming application to those users/communities requesting such a kind of facility.
The infrastructure is based on the Apache Kafka platform, which provides streaming of events at large scale with authorization and authentication at the topic level for data isolation and protection. Data can be consumed by different applications, such as Opensearch, which provides the capability to index a large amount of data and implements appropriate access policies.
In this contribution we will present and motivate our technological choices on defining the infrastructure, the configuration of the implemented services and finally the use cases we plan to address with such a data streaming platform.
| Consider for long presentation | No |
|---|
