Please visit Jefferson Lab Event Policies and Guidance before planning your next event: https://www.jlab.org/conference_planning.

May 8 – 12, 2023
Norfolk Waterside Marriott
US/Eastern timezone

Backfilling a cloud with grid jobs

Not scheduled
1h
Hampton Roads Ballroom and Foyer Area (Norfolk Waterside Marriott)

Hampton Roads Ballroom and Foyer Area

Norfolk Waterside Marriott

235 East Main Street Norfolk, VA 23510
Poster Poster Poster Session

Speaker

Whitehouse, Dan (Imperial College)

Description

Imperial College London hosts a large Tier-2 WLCG grid site based around a HTCondor batch system; additionally it provides cloud computing facilities using Openstack to non-WLCG activities. These cloud resources are open to opportunistic usage, provided the impact on the primary cloud users remains low.
In common with most Tier 2 sites we see constant job pressure from the WLCG VOs, while the usage of our cloud is typically much more intermittent. To allocate grid jobs to the available opportunistic cloud resources we implemented a lightweight backfill system based on the Openstack python API. We execute these grid jobs in a dedicated openstack project, the quota of which is adjusted according to the amount of unused resources in other projects. A second process takes care of starting new instances and erasing completed or stalled ones. We build a dedicated image for the virtual workernode instance once a day, using the Jenkins Continuous Integration tool. This ensures the image is always up-to-date and security updates are applied promptly. In this image we include a client for the HTcondor batch system and a script to shutdown the worker node if no work is underway to avoid having empty worker nodes taking up resources. For monitoring, we collate the key parameters of the backfill system using Ganglia.
Here we present our implementation of this lightweight backfill system and we report on its first year in production.

Consider for long presentation No

Primary authors

Whitehouse, Dan (Imperial College) Fayer, Simon (Imperial College)

Presentation materials

Peer reviewing

Paper