Digital Services Project

Overview

The Digital Services Project is a long-term initiative of the Library begun in 1998 as a strategy for ensuring effective management of its digital collections and their preservation for future access as technologies change. The aim of the project is to:

  • Provide the infrastructure for long-term management of digital material in the Library's collection through provision of hardware and software systems supporting integrated collection management in a digital environment.
  • Provide the Library with cost-effective technical solutions for efficient development and delivery of digital services to provide enhanced access to the Library’s collections and other documentary resources, and provide shared access to digital collections in cooperation with other institutions.

The Library defines digital services to mean services exploiting digital technologies to provide access to traditional documentary resources and to ensure the long-term preservation of, and access to, documents available only in digital form.

Summary of achievements

As a first deliverable, a Systems Architecture was developed that has been tested and modified over time to provide a framework for a wide set of IT development and procurement activities. This framework embraces all of the key library processes of selection, acquisition, storage, resource discovery, delivery, access control and preservation. It also covers both digitised and born-digital information resources.

The Library then tested the market to identify and acquire the infrastructure components needed to meet the aims of the project. At almost every stage the requirements proved to be specialised or leading edge. This resulted in the need to build some elements of the infrastructure in-house. The Library has used open source software whenever this has provided a functional and robust solution, and has also started to look at ways of sharing the software it has developed in-house through open source licencing arrangements.

Metadata Repository and Search System

The capability to separate resource discovery from resource management and delivery was a key element of the Digital Services framework from the start. The Library had a requirement to build resource discovery services based on a combination of metadata harvested from a variety of sources and databases accessed through standard search and retrieval protocols. From 1999-2002 the Library used the MetaStar Enterprise software from BlueAngel Technologies for this purpose.

In July 2002 the Library signed a contract with InQuirion Pty Ltd for the TeraText software which is now being used to provide federated, metadata-based resource discovery services such as PictureAustralia, MusicAustralia, LibrariesAustralia.and the Australian Research Online. Through TeraText, the Library also provides the 16 million record National Bibliographic Database (NBD) as a search target for other metasearch services.

In 2006, the Library selected the open-soure software product Lucene as a full text search engine for PANDORA (Australia's Web Archive). Recently, it has also been experimenting with Lucene as a metadata search engine in a laboratory environment, using a copy of the NBD to prototype relevance ranking and clustering of result sets and the use of FRBR to group related items.

Digital Object Storage System

One of the lessons learnt through testing the market was that the digital object storage requirement could be effectively separated from the digital object management requirement. In 2001 the Library took delivery of a Digital Object Storage System (DOSS) to handle the Library's digital object storage requirements for the period 2001 to 2006. The system consists of a Sun E450 server, a CLARiiON FC4700 disk array, a StorageTek Tape Library connected via a SAN switching infrastructure. The new system which became operational in July 2001 had been upgraded by 2005 to a capacity of 14 TB of disk and 60 TB of offline tape storage.

Digital Collections Manager

The Library also recognised at an early stage in the project that, while the underlying storage systems and information model might be the same, different workflow systems were needed to support the digitisation of documentary heritage collections and the capture and archiving of websites. A Digital Collections Manager (DCM) was developed in-house, replacing a number of existing in-house systems and providing for the acquisition, storage and long-term management by the Library of the products of digitisation and born digital collection items.

The current version of this software supports the digitisation of still images and audio material and stores descriptions of the source copy and structural, technical and preservation information about the resulting digital objects. It is also used to manage born digital collection items and to store information about the Library's analogue audio material that has yet to be digitised. The functional specifications for this system are still a good indicator of the gap between existing digital object management systems in the marketplace and the level of workflow support required for the digitisation of documentary heritage materials as an operational task.

Work has now begun on a system to support the recording of intellectual property rights and the Library is also working with the Australian Partnership for Sustainable Repositories (APSR) to develop systems to support the management of the preservation of the Library's digital collections and management of the preservation of physical format collection materials.

PANDAS - PANDORA Digital Archiving System

To support the separate need to capture and archive websites in an efficient and cost-effective manner, in 2001 the Library replaced an antiquated gathering system based on Harvest for the PANDORA Archive with the PANDAS system. A second version issued in 2002 provided improved workflow efficiencies and enabled archived material to be distributed nationally rather than being controlled centrally at the National Library.

In 2005 the UK Web Archiving Consortium began using PANDAS for its a test-bed for selective archiving of UK websites.

The software is currently being redeveloped to position it better to meet the needs of other agencies as a website curation tool, to take into account emerging web archiving standards and to meet future requirements such as support for deposit workflows and preservation management.

Digital collections, persistent identifiers and web delivery systems

In July 2001, the Library embarked on a major digitisation program to provide greater access to its collections through digitisation of traditional format library materials. This resulted in a new focus on the need for systems to support citation and delivery of digital collections.

In September 2001 the Library implemented a resolver service for digital collection items. The resolver accepts web requests which refer to a digital collection item by its persistent identifier and redirects them to the current storage location of that item. The service was implemented following the recommendations of a consultancy on Persistent Identifier Systems. Persistent Identifiers are now included at the bottom of every image (except thumbnails) delivered by the library's digital services.

The Library has also substantially implemented a generic web delivery framework and access architecture for its digital collections based on the Digital Collections Manager Database. Delivery systems for printed music and maps were released in July and September 2002 and for manuscripts, serials and monographs in 2003-2004. Delivery systems for audio material and pictures are currently under development. The manuscripts delivery system is based on the Encoded Archival Description. . The audio delivery system will also provide access to summaries and transcripts based on the Text Encoding Initiative guidelines.

The delivery framework is currently limited to material without access restrictions. The delivery system will be further developed to support access by authorised users to restricted material once systems to record rights management information are in place.

Metadata harvesting

An Open Archives Initiative (OAI) gateway has been implemented to enable metadata describing the Library's digitised collection items to be harvested by other agencies.

Future directions

Through its involvement in ARROW (Australian Research Repositories Online to the World) and APSR (Australian Project for Sustainable Repositories), the Library is currently investigating D-Space and Fedora as two alternative initiatives that may inform the ongoing development of its digital services framework. In particular, it is interested in aligning the information model and backend processes required to manage the digital objects under its care regardless of format, while continuing to support separate workflow systems for their creation and/or acquisition and ingest.

The Library has been monitoring developments in software and business models to support the digitisation of historical newspapers and the whole-of-domain capture of websites. This may require new software components to be incorporated into the Library's digital services framework and will certainly place new demands on the capacity of the Digital Object Storage System.

The Library is also investigating search engine software that will enhance the resource discovery process by providing access to the content of digital objects in Australian collections or federated metadata repositories and by enabling users to view different versions of an object over time.

Stop press

In October 2006 the Library commissioned a project team to review the achievements of the Digital Services Project and to define the IT architecture that will be needed to support the management, discovery and delivery of the National Library of Australia’s collections over the next three years.

The IT Architecture Project Report (pdf file, 627 kb) recommends three changes to the Library's current approach to building digital library services:

  • to implement a service-oriented architecture;
  • to adopt a single business approach; and
  • to consider open-source solutions where these are functional and robust.

Further Enquiries

Please direct feedback on the IT architecture project report or any other enquiries about the Library's digital library services by e-mail.

 

Page last updated 21 March 2007.

Back to the top