A paper by Dr Warwick Cathro, Assistant Director-General Information Technology, and Mr Tony Boston, Director Digital Services, National Library of Australia, at EduCause in Australasia 2003
The National Library of Australia has developed an architecture for the management, discovery and delivery of its digital collections, embracing both digitised and born-digital information resources. The components of this architecture interoperate using recognised standards developed by communities working in the field of digital libraries. The Library uses the architecture to deliver access to over 100,000 digital objects which include archived web sites and digitised pictorial, map, sheet music, manuscript and audio collections. The main components of the architecture are: (i) A Digital Object Storage System. (ii) A Digital Archiving System which provides Library staff with the tools to gather and manage archival copies of selected Australian websites. (iii) A Digital Collections Manager which provides Library staff with the tools to support digitisation of the Library's collections and the management of the digital surrogates. (iv) A Metadata Repository and Search System. (v) A Persistent Identifier Resolver Service so that all digital collection objects can be allocated a Persistent Identifier and can be accessed via the web using this identifier. (vi) Delivery Systems for complex digitised collection items such as Manuscripts, Maps, Sheet Music and Audio recordings. This paper describes these components with an emphasis on the underlying technical standards which support interoperability, discovery, management and delivery of the digital collections.
In common with many other cultural and research institutions, the National Library of Australia is building and delivering digital collections to support research and learning. These digital collections do not belong to a separate sphere of activity: they are an integral part of the Library's total collection, although they obviously require different techniques for their acquisition, management and preservation. The digital collections, along with traditional collections such as books, journals, manuscripts, pictures and oral history interviews, together form a major national resource designed to meet the information needs of researchers and citizens.
The development and delivery of these collections is occurring within the context of the Library's statement of its directions for 2003-2005 [1]. This document signals that the Library will build collections of Australian electronic materials alongside its more traditional collections, and that its preservation activities will encompass these electronic collections. It also indicates that the Library will "digitise selected materials from [its] collections and assist users to access and explore, via the Internet, each type of collection material". The emphasis in the Directions statement is on delivering "fast and convenient access" to its collections and on developing new measures to improve access.
The digital collections include both digitised copies of collection items and born-digital electronic publications, including web sites captured for archiving.
In 1996, through its PANDORA Project, the Library began to collect archive copies of significant Australian web sites. There are now over 3400 web sites stored in the archive. In many cases, multiple snapshots of a web site have been captured, and the total number of gathered instances now exceeds 6600. Many of the sites (such as the Sydney Olympic Games site, and many election campaign sites) have already disappeared from the live Internet. The Digital Archiving System, described below, supports the process of gathering, managing and providing public access to the archive collection items [2].
In 1996 the Library also began to deliver access through its web site to some of the most significant items in its pictures collection, and to digitise all current acquisitions of pictures. In subsequent years the Library trialled digitisation techniques for other materials such as maps. In 2001 the Library embarked on a significantly expanded digitisation program. The material being digitised is an important part of Australia's cultural heritage and includes pictures, rare historical maps, early Australian sheet music and selected manuscript collections [3].
Digitisation has been undertaken in-house using capture technologies appropriate to the format of the collection material. The digitisation workflow is supported by the Digital Collection Manager system, described below. Typically, digital master images are held in high resolution (300 ppi) uncompressed 24 bit RGB TIFF version 6 format, while low resolution derivatives for viewing on the web are held in 72 ppi JPEG format.
Since 1997 the Library has been developing a technical architecture to support the management, discovery and delivery of its digital collections. It published an early description of this architecture in 1998 in its Digital Services Project Information Paper [4].
Key principles identified by the Library included the need to:
The following diagram shows an overview of the Library's digital services architecture:

This paper will first describe the bottom and top layers of the architecture, before focusing on the middle layers which are the main topic of the paper.
In June 2001, the Library acquired a dedicated server and tape library to store its digital collections. The initial capacity of the system was 2 terabytes of disk with a 4 terabyte tape library. This has now been upgraded to 4 terabytes of disk and 12 terabytes of tape storage. The system is upgradeable to 18 terabytes of disk and 50 terabytes of tape storage.
In early 2003 the system was storing about 4 terabytes of collection material, with the largest components being pictorial images (39%), oral history audio files (23%), sheet music images (19%) and the PANDORA archive (8%).
A structured file system has been developed to store images and sound files. The file system includes separate areas for digital masters, derivatives for viewing on the web and a working area where files are temporarily stored during creation, processing and transfer. Within these areas the file system reflects the division of material held by the Library into particular collections. Digital master files are stored on tape while web derivatives are stored on disk. Files on tape appear to the user to be online (they are actually nearline) and are automatically fetched in a few minutes from tape when requested by a user.
The entire Digital Object Storage System consists of:
Users commonly search for and discover digital collection items through a number of entry points. From the Library's institutional perspective, the prime entry point is its catalogue, since the digital collections form an integral part of the total collection. Consequently, the Library has ensured that all of the items in its digital collections are represented in its catalogue, and has also ensured that there are links from the catalogue records to the delivery systems using the Library's persistent identifier framework. (described below) In the case of the digitised collection items, access to the digital surrogates is provided through the catalogue record for the original item.
There are important entry points outside the institutional perspective. These include public search engines, and in recognition of this the Library is ensuring that, as far as possible, items in its digital collections are indexed by search engines such as Google. Other important entry points include federated discovery services such as Kinetica, PictureAustralia and the Register of Australian Archives and Manuscripts (RAAM). The word "federated" is used to signify that metadata has been gathered from many contributors to create a search service supporting simultaneous discovery of information resources residing in the collections of all the contributors. The Library itself supports a number of federated and specialised discovery services, using the component of its architecture called the Metadata Repository and Search System.
In order to support inter-operability between these resource discovery services and other national and international services, the Library requires that its resource discovery platform fully support the Z39.50 search and retrieval protocol.
One strength of the Library's architecture is that it is not dependent on any single Metadata Repository and Search System. Between 1999 and 2002, the platform for the Library's Metadata Repository and Search System was the Metastar Enterprise product from Blue Angel Technologies. The Library is now using the Teratext Database System [5] from Inquirion Pty Ltd, a company spun off from RMIT University. The change of platforms was driven by a need for greater scalability and performance in searching and indexing, and for greater standards compliance. The Library is progressively migrating its federated and specialised discovery services to this new platform.
An important feature of the architecture used by the Library is the implementation of a persistent identifier framework. This was developed following a consultancy undertaken for the Library in 2000 and 2001. The framework and the consultant's report have been published on the Library's web site [6]. The use of persistent identifiers ensures that every item in the Library's digital collections is citable in a persistent manner. The identifier can be used in a variety of contexts (including the Library's catalogue, federated discovery services, scholarly papers, bookmarks, and hyperlinks from other web pages) with confidence that this citation will never "break": it will always resolve to the location of the object on the Web. The Library assigns a unique and persistent identifier to every digital collection item and to each deliverable part of every digital collection item. The naming schemes are, where possible, based on existing naming schemes for physical items. Persistent identifier web addresses have two components - the name of the resolver which is a web server at the National Library, and the persistent identifier. They are of the form http://nla.gov.au/[persistent identifier]. For example:
The Library has developed a simple resolver service that redirects a user to the current location of an object based on its persistent identifier. If the Library's system receives a request in the form http://nla.gov.au/[pi], it will always pass this request to the resolver. The resolver is a Java servlet application that first parses the persistent identifier and then, based of a set of rules stored in an XML configuration file, determines the web location to re-direct the user to in order to access that resource. If the Library later changes the way it delivers a particular digital collection, it need only make a simple change to the configuration file to ensure continued access to that resource. This technique results in simple and fast resolution to a resource's current location but requires persistent identifiers that contain some "intelligence" so they can be parsed, and good data management practices to ensure consistent delivery of content within the Library's digital collections. An unexpected benefit of the resolver service has been an increased capability to measure the usage of the Library's digital collections regardless of the service used to discover the item, since all use is brokered by a single service.
The Library was able to implement the lower and upper layers of its architecture (the Digital Object Storage System and the Metadata Repository and Search System) using products available in the marketplace. However, this proved not to be possible for the middle layer, the digital object management layer, and here the Library was obliged to develop two significant software components itself. The Library hopes that it will be possible to replace these components with marketplace products within the next five years.
One of these components is the Digital Archiving System, which provides Library staff with the tools to gather and manage the items for the archive of selected Australian websites and electronic publications. This system, also known as PANDAS (PANDORA Digital Archiving System) has been described extensively elsewhere [2]. The system is now being used by a number of state libraries who are collaborating with the National Library to build the archive.
The system which manages the digital surrogates is called Digital Collections Manager. The Library made a strong effort to base this on a "digital asset management system" available in the marketplace, but it was not able to identify a system which was both affordable and closely matched to the Library's needs. Apart from affordability, the main issues were that systems in the marketplace did not sufficiently support a hierarchical digital library model, and did not sufficiently support the Library's digitisation workflows. In addition, the digital asset management systems available at that time did not have the required separation in their architecture between digital storage, management, discovery and delivery.
The Digital Collections Manager (DCM) supports the digitisation process, the creation of derivative objects for web delivery, and the storage of technical metadata and digital files [7]. DCM currently supports management of images (including pictures, maps, sheet music and manuscript materials) and is being enhanced to support management of audio recordings from the Oral History Collection. Later in 2003 it will be further enhanced to support recording of rights management information and information used for preservation management.
The metadata supported by the DCM data model includes:
The functional specifications for DCM have been published on the Library's web site [8].
The following screen capture shows the technical metadata stored within DCM for a digital master image of an item from the Pictures collection:

The following DCM screen details the copies of this picture held by the Library. These include the physical format original, a high resolution master and cropped "co-master" in TIFF format and low resolution thumbnail and view copies in JPEG format:

Several standards informed the development of the DCM data model, including the draft NISO standard Z39.87 Technical Metadata for Digital Still Images [9] and the Metadata Encoding and Transmission Standard (METS) [10].
METS is an XML document format and schema for metadata describing digital objects that can be used for the management of digital objects within a digital library and exchange of such objects between repositories. A future enhancement to DCM will allow the generation of METS instances for digital objects. These METS files will be archived along with digital masters for long-term preservation and may also be dynamically generated using the Open Archives Initiative protocol [11] when metadata about digital objects needs to be supplied to other repositories. The Digital Services Project Information Paper [4] specified requirements for a single Digital Object Management System that would meet the requirements currently delivered separately by PANDAS and DCM. However, the requirements for managing these types of collection material were sufficiently different to make the development of two different systems the most practicable course.
An important reference standard used by the Library was the Open Archival Information System (OAIS) Information Model [12]. This model defines core processes of ingest, data management, archival storage, access and administration. While these processes are common to both PANDAS and DCM, the difference is reflected in the level to which each system needed to support different workflows for the acquisition of digital objects. This process falls outside the scope of the OAIS Information Model.
One of the principles identified above was to "provide users with service delivery options appropriate to the category of object being accessed". To this end, the Library has developed web delivery systems for pictures, maps, sheet music and manuscripts.
These separate delivery systems have been developed in the context of a generic delivery framework. All of the delivery systems are based on descriptive and structural metadata sourced from the Digital Collections Manager. (The Manuscripts Delivery System, initially developed to deliver web access to the Sir Edmund Barton Collection, is currently being redeveloped to base it on DCM). The generic web delivery system returns a digital object in a contextual display to a user's web browser. The user interface allows the user to navigate the multiple pages of the object, to zoom in to view details of each page, and to view metadata pertaining to the object. The delivery system is accessed using the Library's resolver service.
In the case of manuscripts, the delivery system must be closely tied to the finding aid which describes the collection and which lists the series and individual items within that collection. Manuscripts and other multi-page items (such as music scores and rare books) also require a flexible page turning capability.
In the case of maps, users need a powerful zoom capability, and a means of identifying their current zoom location on the whole map. The MrSID technology [13] provides these features, and is an integral part of the Library's maps delivery system.
The Library is about to embark on an extension of the delivery framework to support delivery of audio files, including oral history interviews. The Library will use two streaming audio delivery technologies, with support for both low and high transmission rates in order to meet the needs of the maximum number of users.
In some cases, the form of the material may demand a delivery system which is different from the collection type implied by the Persistent Identifier (for example, it may be a picture or map within a manuscript collection). In such cases, the delivery system examines the "form of material" code stored in the DCM database and delivers the object accordingly.
Development of the Digital Collections Manager, the Web Delivery System and digitisation policies and procedures have proceeded in parallel, enabling the Library to respond rapidly to new requirements as more complex materials are digitised.
The National Library has developed a technical architecture to support its strategic objective of collecting and delivering access to digital collections. Key drivers have been the need to support integrated access to digital and print resources, to ensure that every digital collection item is citable in a persistent manner, to support a hierarchical digital library data model that reflects the complex nature of some collection items, and to ensure that the Library's ILMS has a central role in the architecture. This has required an architecture that separates digital storage, management, discovery and delivery into separate components. While the Library has plans to replace the components it has built in-house with a commercial product when one becomes available, there is still a tendency in the market place to develop products that focus on digital object discovery and delivery rather than the full set of activities required to acquire, manage and preserve these objects as integral parts of a larger collection.
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |