The National Gallery of Art (NGA) has been using IIPImage since the 2010 online release of The History of the Accademia di San Luca research project. The zoom feature of this website uses a slightly customized IIPImage JavaScript client to permit the viewer to inspect 1,000 high-resolution scans of historical manuscripts dating back to the sixteenth century. Our experience with IIPImage on the San Luca project was encouraging. The IIPImage server software performed well under load and was able to quickly render zoomed portions of imagery to web-browser-based clients.
In 2012 we embarked on a sustained internal development effort to redesign the NGA public website from the ground up. An important objective of the redesigned site was to make high-resolution art images available for as much of the Gallery’s collection as possible and to make ultra-high-resolution images available for the highlights of the collection. Although IIPImage seemed to us a natural choice, we nonetheless conducted a fresh study in spring 2012 of the available zoom servers and clients, which only reinforced our prior conclusion that IIPImage is the best image zoom package for viewing art objects.
Important requirements for the zoom server on the redesigned public website include:
- Must provide a fast and efficient server-side engine using open standard file formats
- Must provide client software based on open standards, i.e. pure HTML + JavaScript with an upgrade path to HTML5 rather than proprietary technologies requiring a browser plugin
- Must be able to operate from as few source images as possible (ideally a single zoom file) rather than requiring a multitude of pre-generated zoom tiles for each image
- Should retain and reproduce accurate color information from source imagery to the web browser on the client side
- Preferably based on open source software and released under open source license in case it needs to be modified to suit our purposes
- For future endeavors, the tool should provide auxiliary services to generate on-the-fly renditions of zoom images of arbitrary size and location
The IIPImage solution was able to meet most of these requirements. In April 2013 when the redesigned www.nga.gov website launched, it included dozens of ultra-high-resolution images for collection highlights and thousands of high resolution images of paintings. Since launch the NGA’s rapid capture project added tens of thousands of additional high-resolution digital photographs of works on paper, including (by September 2014) the full set of images of watercolors from the Gallery’s Index of American Design. Rapid Capture is an ongoing endeavor that seeks to produce high-resolution digital imagery for nearly all works on paper within the Gallery’s collections.
A year later, in April 2014, a major set of enhancements to www.nga.gov was released. National Gallery of Art Online Editions debuted its first edition, Dutch Paintings of the Seventeenth Century, as part of the Getty Foundation’s Online Scholarly Catalogue Initiative (OSCI). The site includes scholarly research on more than 120 paintings and dozens of artists as well as X-ray, infrared, and ultraviolet band art images, some of which have led to new insights and fascinating discoveries.
The HTML5 IIPMooviewer client was customized to allow registered technical images to be faded in and out with a transparency control while overlaid on top of the visible light image. In addition, when registered technical images are selected, the zoom and pan settings can be locked so that one set of controls is used to navigate two images.
Image Acquisition and Production
Images of Gallery art objects are produced by the Division of Imaging and Visual Services. The different methods of digitizing artwork are summarized here. Direct digital capture is the highest quality digital reproduction produced at the Gallery. Artwork is captured using a professional, large-format, high-resolution digital camera back that is calibrated for color. The camera back’s multishot feature allows it to eliminate moiré interference patterns in the reproduction. The image is also corrected for uneven distribution of lighting, which results from lens falloff and non-uniform light distribution on the surface of the artwork. Each image is proofed against the original artwork under ISO proofing conditions and manually corrected when the color calibration does not produce a satisfactory visual match.
Producing Ultra-High-Resolution Photography of Paintings
Using the camera’s full resolution to capture just one section at a time, we are able to increase dramatically the amount of detail captured. Our computer-controlled motorized easel allows us to photograph a painting in precise regions (tiles) with repeatable accuracy within 0.05 inches. The tiles are then “stitched” together as a whole into what becomes an ultra-high-resolution digital image that provides extraordinary detail, revealing small brushstrokes and hairline cracks that may be invisible to the naked eye.
With advice from our technical partners, we designed and fabricated our easel to move the painting while keeping the camera and lights stationary, the opposite of many other step-and-repeat systems. By not forcing a fixed relationship between the camera and lights, we allow maximum flexibility to use any camera and to vary the direction and quality of the light, the distance of light to the artwork, or the type of light used in order to produce the desired reproduction. Options include standard visible light photography, raking light photography, polarized light photography, and ultraviolet-induced visible fluorescence photography, all possible without compromising any detail.
Registration of Technical Images with Primary Art Object Image
In order to compare the various imaging modalities, image registration must be carried out and the NGA has developed it’s own registration/mosaicking algorithm, which corrects for several kinds of distortion, small rotation and scale errors, and keystone effects between images. Thus images acquired with different cameras, illumination, and geometries can be registered/mosaicked. See the publication Automatic registration and mosaicking of conservation images for more details.
Web Derivatives and Zoom Images
The source images, some of which are greater than 20,000 pixels in the long dimension, are made available to the web systems as color-corrected uncompressed TIFF files. These large images require fast CPUs and a lot of RAM. A single ultra-high-resolution source image file can be greater than 1GB. The NGA’s web image processing server is a VMWare ESXi virtual machine (vm) running Red Hat Enterprise Linux 6. This vm is provisioned with 24GB of RAM and 3TB of SAN based storage and is used to produce pyramidal TIFF files (PTIFs) and JPEG derivatives in a variety of resolutions for the Gallery’s public website.
New imagery is marked for publication and pulled by a Perl script from the DAM (Portfolio). The script downloads each source image and calls ImageMagick’s convert routine to extract the embedded color profile. It then produces the set of JPEG derivatives, each of which is injected with an embedded sRGB ICC profile after the image’s color space is converted to sRGB from the original ICC profile. Metadata about the art object (title, artist, etc.) is embedded into each image file. The script also saves metadata about each image in an imaging database that is subsequently used by the website to locate art object imagery.
Once the JPEG derivatives are produced, the script then calls a VIPS routine to produce a single pyramidal TIFF file to feed the IIPImage server. We found that VIPS produces PTIFs a bit faster than ImageMagick’s convert tool. We also discovered a bug in convert that corrupts certain PTIF files in rare cases; VIPS is clearly the preferred tool for creating PTIFs. Each derivative image’s integrity is then verified using ImageMagick’s identify command. Specific options used when executing the convert and vips tools are outlined below.
Using convert to produce JPEG derivatives for the web
convert -filter Mitchell -unsharp 2x2 -scale WIDxHGT -quality 90 -density 72x72 \ -units pixelsperinch source_file.tif +profile '*' -profile source.icc \ -profile sRGBProfile.icc dest_file.jpg
- In our opinion, using the Mitchell filter and unsharp mask algorithms during the size reduction operation produces the best overall derivatives
- 90% JPEG quality produces very few compression artifacts while still resulting in a significant reduction in file size
- Density of 72×72 pixels per inch is a typical screen density and included for backwards compatibility with some internal systems that use web imagery in collection reports
- The +profile ‘*’ –profile source.icc –profile sRGBProfile.icc instructs ImageMagick to convert the color encoding in the original source image to the sRGB color space and to embed a standard sRGB color profile into the final JPG derivative. We use a standard sRGB color profile since these images are primarily for viewing through a web browser.
Using VIPS to produce PTIF zoom images for IIPImage
vips im_vips2tiff source_file.tif dest_file.ptif:jpeg:90,tile:256x256,pyramid
- VIPS retains embedded ICC profiles although IIPImage does not insert the ICC profile into each zoom tile; this results in slight color deviations on the Safari web browser which does not seem to default to using sRGB like other browsers.
- 90% JPEG compression is used to reduce the PTIF source file size while preserving quality
- 256×256 tile pixel dimensions setting was recommended by Ruven as a reasonable setting to use. Using smaller tile sizes results in a larger number of zoom level steps but larger PTIF file sizes and much slower client-side performance since many more tiles must be fetched to render each view. For example, a 12,000 x 15,000 pixel image with 256×256 tiles results in 7 zoom levels and a file size of 67 MB. Using 32×32 for the tile size doubles the size of the file and results in unacceptably slow rendering times on the client side. On the flip-side, using a larger tile size reduces image storage requirements, but increases the image data that must be transmitted to render each zoom view.
Implementing the IIPImage Server
High Level Architecture
Once the derivatives and zoom files are produced, a nightly process updates the public-facing file systems where images for the website are stored and updates the imaging database used by the public website. The IIPImage server runs in a dedicated virtual machine, as do the public-facing web servers and the database server.
The image comparison tool created as part of the OSCI initiative includes a drop-down enabling readers to change the images being viewed without having to close the IIPImage client window. The image switching feature requires metadata about all images for an art object and an AJAX based approach is used to fetch this data in JSON format from a service on the public web server. Due to a number of JavaScript and web browser security constraints, we opted to proxy this JSON service to make it appear that the IIPImage client software, the image metadata, and the zoom tiles are all being served by the image zoom server when in fact multiple servers are involved in this orchestration.
Required Software
- 64 bit Red Hat Enterprise Linux 6.5
- Apache 2.2.x (provided by Red Hat) with mod_proxy for accessing image meta-data services
- FastCGI development kit and mod_fcgid (provided by Red Hat)
- IIPImage Server 0.9.9
- ImageMagick 6.7 (provided by Red Hat)
- VIPS 7.30
- libtiff and libjpeg libraries
- GNU C/C++ compiler (provided by Red Hat)
Software Configuration
The IIPImage server runs as a CGI script under Apache web server. Standard CGI has a notoriously high overhead since the web server must invoke a new operating system process every time a CGI based service is requested. In the case of IIPImage, this overhead can be significant since each view of an image requires multiple zoom tiles to be fetched simultaneously. To alleviate the CGI overhead, IIPImage implements the FastCGI interface. FastCGI allows a process to idle in a loop rather than shutting down after servicing a request. This dramatically improves the response time of CGI processes. FastCGI has many configuration parameters, making it easy to fine tune performance for a given CGI application.
Configuring Apache to run IIPImage under FastCGI
# load 64 bit fcgid module LoadModule fcgid_module /usr/lib64/httpd/modules/mod_fcgid.so # # MOD_FCGID default performance tuning for ALL FastCGI applications, not just IIP # # named pipe used for communication between FastCGI module and CGI processes FcgidIPCDir /some_temporary_path # set the minimum number of processes per class FcgidMinProcessesPerClass 16 # maximum number of requests to service before exiting (protects against # memory leaks) FcgidMaxRequestsPerProcess 500 # maximum time for FastCGI to wait for a response from a CGI process # before timing out the response; scan interval is the period between # busytimeout checks FcgidBusyTimeout 30 FcgidBusyScanInterval 15 # maximum age of an idle process and max idle time of a process only if min # number of processes criteria is met; scan is period between these checks FcgidProcessLifeTime 300 FcgidIdleTimeout 60 FcgidIdleScanInterval 120 # max time fcgid will wait to get i/o with its application before killing it FcgidIOTimeout 30 # set up the Apache CGI configuration ScriptAlias /fastcgi/ /some_actual_server_path/fastcgi/ <Directory "/some_actual_server_path/fastcgi/" > # permit CGIs to be executed in this directory Options +ExecCGI # bind to FCGI module SetHandler fcgid-script # these are the final settings AllowOverride None # allow anyone to run the FastCGI scripts Order allow,deny Allow from all </Directory> # IIP configuration is defined through environmental variables # we can also override FastCGI default settings here if required FcgidCmdOptions /some_actual_server_path/fastcgi/iipsrv.fcgi \ InitialEnv LOGFILE=/another_actual_server_path/iipsrv.log \ InitialEnv VERBOSITY=1 \ InitialEnv JPEG_QUALITY=90 \ InitialEnv MAX_IMAGE_CACHE_SIZE=1024 \ InitialEnv FILESYSTEM_PREFIX=/server_path_to_images # use higher quality interpolation option when generating arbitrarily sized # JPEG renditions – we do not currently use this feature but might in the future # InitialEnv INTERPOLATION=0 \ # can set maximum resolution of on-the-fly rendered JPEGs – default is 5000 pixels # InitialEnv MAX_CVT=1200
Considerations for the future
IIPImage caches image tiles in memory, which improves performance of recently viewed images. It would also be advantageous to be able to use the disk to supplement this cache. Ruven has recommended both Memcached and Varnish cache to use in conjunction with the native caching built into IIPImage Server.
The IIPImage Server also supports rendering subsections of an image through the region API. The most recent releases of IIPImage should produce pixel accurate renditions. If used in conjunction with caching software such as Varnish or Squid, we could possibly eliminate the need to produce multiple resolutions for the public web site in advance. In addition, detail images could be represented by simply defining the region of interest rather than having to create such details using an image editing tool. In such a scenario, we would rely on the PTIF as the source for all web imagery and create then cache only the imagery required to meet demand. It is also possible to extend IIPImage Server with customized plug-in filters to be used during image resize operations, e.g. Mitchell filter, unsharp mask, etc. The NGA intends to explore that option in the future in order to maximize the quality of images produced by the IIPImage region operation. Of course, there is a trade-off to be made between quality and performance, so this will require careful study.
Since we are dependent on the web browser to render colors properly, it also seems important to embed the PTIFs ICC profiles within each separate zoom tile. Doing so would enable proper color rendering in the zoom client, possibly even by Safari which seems to treat images lacking an ICC profile differently from images containing one. In fact, a cursory review of web browser color management in general has shown Firefox and Chrome to have superior performance displaying accurate colors even when the correct profile is used. Ruven has indicated that he intends to add ICC profiles as a feature in a future release.
In summary, the Gallery is thrilled with the performance, quality, and stability of the IIPImage server and client software, and we intend to use it for years to come. The software is helping the Gallery fulfill its mission by providing to our audiences an ability to get closer to our works than they ever have before – and in some cases, closer than our protection officers would ever permit them to do in person.
by David Beaudet, Alan Newman & Kenneth Fleisher, the National Gallery of Art.
