Abstract

GLAST LAT Science Data Processing - Four Months Experience After Launch

Richard Dubois (Stanford Linear Accelerator Center)

The GLAST Observatory was launched on June 11, 2008 and the Large Area Telescope (LAT) was activated on June 25. Some 15 GB of data is downlinked daily, transformed into 750 GB in the event reconstruction process, spread out over approximately 8 contacts per day. Each data run is farmed out to several hundred computing cores and results merged back together in our processing pipeline. The pipeline is designed to execute complex processing trees defined in xml and to handle multiple tasks simultaneously, including prompt data processing, simulations and data reprocessings. Our system has a pair of Oracle servers at its center to maintain all the state and dataset bookkeeping. Batch processing happens in the SLAC LSF batch farm with more than 2000 shared cores. The xrootd filesystem is used for high throughput and management of large disk pools. Nagios and Ganglia are used for problem alerts and tracking resource usage. The HEP-like instrument event reconstruction lives in a Root world, while high level science is done in FITS. Users have access to the data via web query engines that slice and dice the data to their needs, also executing the queries in the processing pipeline. Two operations simulations were performed before launch as full scale exercises with a day, then a week of fully simulated data rolled out in real time. These were key to flushing the final issues out before actual launch operations, which have been remarkably smooth.

Mode of presentation: oral