The Wide-Field Astronomy Unit (WFAU), Institute for Astronomy, University of Edinburgh specializes in building and maintaining science archives from astronomical surveys. The bulk of the data we currently process come from two infrared telescopes: VISTA in the southern hemisphere and UKIRT in the north.
The surveys carried out by these telescopes have produced around 100-200 TB of data. WFAU holds the pixel data (images) in flat files whilst the object catalogues, derived from the pixels, are held in MS SQL databases.
Recently the number of stars in the catalogues from two adjoining surveys of our own Milky Way, the VISTA Variables in the Via Lactea (VVV) and the Galactic Plane Survey (GPS) exceeded one billion. It was decided to mark this milestone by trying to produce a zoomable/pannable image that displays these one billion stars.
Image Processing
To produce the final image several thousand individual images from each of three passbands had to be stitched/mosaiced together. This was preformed using SWarp, astronomy resampling software that can handle arbitrarily large images. Unfortunately as we store the original images in a compressed format which SWarp is unable to currently read, the individual images and their confidence images also required uncompressing. Astromatic’s STIFF was then used to combine the three separate passbands into an RGB colour pyramidal TIFF image
OK so it wasn’t that simple! In the end we had to split up the mosaic into five chunks to get it through the stitching. The resulting five TIFFs were bolted together using the VIPS image processing software.
We now had a 1,267,500 x 120,000 pixel TIFF image weighing in at around 100GB, so over to IIPImage. We had opted for IIPImage over other technologies as it offered a lightweight fast image server able to work from a single image (and other formats if we later go that route) and a javascript viewer that looked like it would be customizable for our needs, i.e. coordinate display and a goTo function.
Early testing with IIPImage and excellent support from Ruven had already shown that it would likely cope with such an image but how would it fare in practice?
Putting 150 Gigapixels Online
At the end of March we put out a press release at the National Astronomy Meeting. This was picked up by various news sites and blogs. For a short while it was the most read story on the BBC website. Over 300,000 unique visitors later the image is still there for all to see:
The image is hosted on a 64bit Linux box with an 8 core 2.6 GHz AMD Opteron Processor and 32 GB RAM. Around 3TB of disk space was needed whilst the image was being created, though the final image is a mere 100GB (LZW compressed). Initial tests on tile size showed that 128×128 resulted in very slow performance. In the end we went with 768×768 with the reasoning being that this would reduce the number of requests and given a typical browser window size the user would be able to pan a bit without initiating new requests.
We used the latest IIPMooViewer version 2.0 code to which the coordinate display has been added and we’ve also altered the viewer to adjust the contrast at different zoom levels.
At peak usage we did encounter a few slow-downs (which Memcached might help with) but generally it works very well and the response from fellow astronomers and the public has been fantastic.
Many thanks to Ruven and the IIPImage forum members for all their help!
by Mike Read, Institute for Astronomy, University of Edinburgh
