Staff Papers

National Library of Australia initiatives using the Open Archives Initiative protocol for metadata harvesting

A paper given by Tony Boston, Director, Digital Services at a National Archives of Australia International Seminar on the use of standards in the development of online access systems for Archives in Canberra in October 2003.

The National Library has embarked on a number of initiatives that use the Open Archives Initiative protocol for metadata harvesting both as a provider of discovery services based on metadata and as a data provider. The Open Archives Initiative (OAI) protocol has proved to have many advantages as a method of exchanging structured metadata conforming to any metadata schema in a standardised way. This paper discusses the benefits of using OAI, based on its use in National Library of Australia services such as PictureAustralia, the National Bibliographic Database and the Library’s digital collections, as well as briefly discussing software options to support the protocol.

Introduction

The OAI protocol for metadata harvesting [1a] provides a standard mechanism for sharing metadata over the Internet. An OAI data repository is available on the Internet and accepts requests for metadata records. OAI requests are encoded into a GET or POST web URL and the metadata response is encoded in XML format. There are six standard OAI requests that allow a user to find out about a repository (Identify), list the metadata formats (ListMetadataFormats) and the sets (ListSets) that it supports, list the identifiers for a set of records (ListIdentifiers), and get a set of metadata records (ListRecords) or a single record (GetRecord). In the OAI model, a data provider maintains an OAI repository that fields requests for metadata from service providers who use the metadata to build value-added services.

Benefits of using OAI

What are the benefits of using the OAI protocol? The protocol is an open, freely available standard created by the Open Archives Initiative. The Open Archives Initiative aims to “develop and promote interoperability standards to facilitate the efficient dissemination of content across the Internet” [1b]. The OAI protocol supports any metadata schema, with the base schema being simple unqualified Dublin Core. It is relatively easy to implement and supports what I call “incremental harvesting”, that is, a user can query an OAI repository and ask it to return just the new, changed or deleted records from a certain date or between two dates. The OAI protocol supports both data providers and service providers in the creation of federated discovery services or portals based on aggregated and shared metadata.

PictureAustralia and OAI

PictureAustralia (PA) [2a] is an example of a federated discovery service based on aggregated metadata. PictureAustralia provides search and browse access to images of Australian people, places and events from cultural agencies including libraries, archives, museums, galleries, universities and historical societies. The service was launched in September 2000 with about 470,000 images and 7 participants and now, about three years later, has nearly 1 million images and 31 contributors. It has been a very successful service for the National Library.

PictureAustralia users search a central database of metadata held at the National Library. The metadata is harvested from the web sites of PA contributing agencies. The National Library does not hold any images centrally, only metadata. So in the OAI model, contributing agencies are data providers and the National Library is the service provider.

When users of PictureAustralia search or browse and discover an image of interest, they can then navigate to the participant’s web site for a larger version of the image. From the participant’s web site, the user can read about any copyright restrictions for the item, order a high resolution copy of the image, navigate to their home page and search their online catalogue. PictureAustralia’s technical architecture is based on centralised metadata and distributed content.

PictureAustralia presents a single access point to the collections of a wide range of cultural agencies. The value of services like PA is that users do not need to know which agency holds a particular image. For example, if users are interested in images of Sydney, after searching PictureAustralia they will find results from the State Libraries of NSW and Victoria, the Australian Heritage Commission, the National Library of Australia and the Powerhouse Museum. The user would not necessarily expect the State Library of Victoria to have pictures of Sydney in its collection, but it actually has an extensive collection of Sydney images which are accessible through PictureAustralia.

PA employs a database of simple unqualified DC metadata. The metadata is held at the National Library for searching and thumbnail images are dynamically loaded into search result sets. Before we started using OAI, all metadata was re-harvested every two months from each participant as it was impossible to detect new or changed records. This is obviously inefficient, slow and costly in terms of Internet traffic and the service was only up to date six times per year.

The advantages of using the OAI protocol for metadata harvesting for PictureAustralia are its support for incremental harvesting, reduced harvest time and cost but the most critical advantage is that we can have a more up-to-date service. The PictureAustralia architecture is vulnerable to changes in the location of image pages on participants’ web sites. More frequent harvesting using OAI reduces the frequency of broken links caused by these changes. If all sites supported OAI, the service could be brought up-to-date every night with a reduction in the time that any broken link occurs in the service to less than one day.

The disadvantage of using the OAI protocol is that it requires some software at the data providers’ site and many of the agencies that have joined PA have very limited information technology resources.

In June 2003, the Library implemented OAI harvesting for the larger sites in PictureAustralia: the State Libraries of Victoria and NSW and the National Library itself. In October 2003, OAI harvesting from the Australian War Memorial was implemented which added another 160,000 records from their collection to PictureAustralia. We are still hoping that other agencies will move to OAI over time. However, at present we have a hybrid harvesting model as shown in figure 1 where we do incremental OAI harvests of the larger sites every day and web site harvests of the smaller sites once a week. PA is brought up to date once a week rather than every two months which represents a major advance in management of the service.

PictureAustralia hybrid metadata harvetsing model

Figure 1: PictureAustralia hybrid metadata harvesting model

Before OAI, web harvesting took about 14 days elapsed time every two months and the harvesting of large sites like State Library of NSW (SLNSW) and Victoria (SLV) could take up to five days each. The OAI protocol is much more efficient than web harvesting as the server response is a set of metadata records encoded in XML rather than a set of individual web pages from which metadata in the header is extracted and converted to XML. After implementing OAI, initial harvests of the State Library sites took about four hours (which only needs to be done once) and incremental harvests take less than one minute. From these figures it is clear how much more efficient OAI harvesting is.

The Library has defined PictureAustralia metadata schema [2b] for use with the OAI protocol which is based on Dublin Core. We use 12 of the 15 elements (all elements except language, type, and date) and have added two fields - the thumbnail image URL to allow us to display a thumbnail in PA result sets and the location URL to link to on the participating agency’s web site.

National Bibliographic Database and OAI

The National Library has also embarked on use of the OAI protocol for harvesting metadata records for the National Bibliographic Database (NBD) [3]. The NBD provides access to over 13 million bibliographic records describing 35 million items held by most Australian libraries. The NBD supports shared cataloguing and inter-library loans for Australian libraries and is currently a subscription service accessible via the Internet.

Traditionally libraries have submitted their records in MARC format via ftp to the NBD. The National Library will continue to support this arrangement but is also working on a new model that will allow OAI harvesting of records in a variety of formats in addition to MARC such as Dublin Core and MODS (Metadata Object Description Schema) [4]. The main benefits of using OAI for NBD data gathering are its support for incremental harvesting, as well as support for a wider variety of metadata schemas.

Digital Collections and OAI

The examples above illustrate the National Library’s role as a service provider. However, the National Library is also a data provider. The Library is digitising a significant component of its out-of-copyright special collection materials [5]. These include pictures, maps, sheet music, several manuscript collections as well as some rare books and serials. Over 66,000 pictures, 4,400 pieces of sheet music, 2,400 maps, and 4,400 manuscript items have been digitised to date.

The Library has developed a database called Digital Collections Manager (DCM) to manage these digital objects [6]. The database stores descriptive, administrative (including technical, rights management and preservation), structural and file level metadata about digital objects.

The Library has recently released an OAI interface to the DCM database [7]. Metadata about the Library’s digital collections is freely available through this interface. The National Library encourages wider use of its metadata, and would be grateful for comments on user experiences in harvesting its metadata and for information on its use in other Internet services.

The Library is providing metadata through this interface to a variety of Internet services including OAISter from the University of Michigan [8], and a similar service from the University of Illinois [9] that provide metadata about academically-oriented digital resources, as well as more specialised services like MacquarieNet [10] which is aimed at school students and the Cultural Materials Initiative (CMI) from the Research Libraries Group [11].

One collection – many paths

We are attempting to make our digital collections available by as many different access paths as possible. For example, all materials are described and available via the Library’s primary discovery service, its online catalogue. This information is also accessible via national federated discovery services like the National Bibliographic Database and, for pictorial items, via PictureAustralia. Using OAI, our digital collection records are also accessible through global federated services like OAISter and targeted services like MacquarieNet or the Research Libraries Group’s CMI. Finally we’re opening up our digital collections to search engines like Google. The Library believes and has statistics that show that increasing the number of alternate paths to information resources also increases the number of users of those resources.

For example, a user interested in finding a picture and other information about Edmund Barton and Alfred Deakin, Australia’s first two Prime Ministers, has multiple alternate access paths to the digital information resources held by the National Library. There is a photo of these gentlemen in the Library’s Pictures Collection, shown in figure 2, which has been digitised and is available on the Internet via a persistent web address [12].

Portrait of Edmund Barton and Alfred Deakin

Figure 2: Portrait of Edmund Barton and Alfred Deakin. http://nla.gov.au/nla.pic-an22948286

To find this item we can search the Library’s online catalogue [13] or pictures catalogue service [14]. However we can also search PictureAustralia, OAISter or even Google. Figures 3, 4, and 5 show the item listed in results from these services. From each of these results, the user can navigate to the digitised item in figure 2 via a persistent URL. It is interesting to note that the services based on structured metadata have a very low number of results for a simple keyword search for “Edmund Barton” and “Alfred Deakin”, namely four for the Library’s catalogue, two for the Pictures Catalogue, four for PictureAustralia, and three for OAISter while there is a much larger result set of 2,370 hits for Google.

PictureAustralia search results

Figure 3: PictureAustralia search results

OAISter search results

Figure 4: OAISter search results

Google search results

Figure 5: Google search results

OAI is a key enabler for metadata sharing, but equally important are persistent identifiers and persistent web addresses for digital objects. There is not much point in sharing metadata between discovery services if the objects being described move location on the Internet.

OAI software

In a technical sense, how can the OAI protocol be supported to make your collection management system records available? There is free software available at the Open Archives Initiative web site and it is easy to link this software to an existing database, metadata repository or collection management system. In fact, the Library’s OAI interface to the Digital Collections Manager database uses the OCLC OAICat [15] software which is available on the OAI web site.

There are also some commercial offerings that support the OAI protocol and it is hoped that support for the protocol by vendors of collection management systems will increasingly become commonplace. PictureAustralia uses the TeraText Metadata Publishing System software [16] to manage the harvesting of metadata for PictureAustralia. MPS can be used to do web harvesting, extraction and storage of metadata in a database which can respond to requests as an OAI data repository. MPS is an Australian product from a company called InQuirion which is based in Melbourne.

Figure 6 shows the MPS software used to manage PictureAustralia harvesting. Some of the sites from which the Library harvests are shown, as well as the number of records harvested from each site. There is a mixture of disk, web and OAI harvesting, with the majority being web harvesting, and harvests can be scheduled so that they run automatically at night when servers are least loaded.

MPS software used to manage harvests of PA participants' metadata

Figure 6: MPS software used to manage harvests of PA participants’ metadata

Conclusion

The Open Archives Initiative protocol for metadata harvesting is a proven, useful mechanism for sharing metadata over the Internet between organisations to create value-added services and new paths to information resources.

However, it should be remembered that OAI by itself is only a mechanism for sharing metadata and it does not solve the semantic interoperability issues that arise when metadata from different domains conforming to different standards is aggregated.

References

1. a. Open Archives Initiative. http://www.openarchives.org/
b. Organization. http://www.openarchives.org/organization/index.html

2. a. PictureAustralia. http://www.pictureaustralia.org/
b. PictureAustralia | Metadata schema. http://www.pictureaustralia.org/schemas/pa/

3. ALG: National Bibliographic Database (NDB). http://www.nla.gov.au/libraries/resource/nbd.html

4. Metadata Object Description Schema (MODS). http://www.loc.gov/standards/mods/

5. Digitisation of Traditional Format Library Materials. http://www.nla.gov.au/digital/program.html

6. Warwick Cathro and Tony Boston (2003). Development of a Digital Services Architecture at the National Library of Australia. http://www.nla.gov.au/nla/staffpaper/2003/cathro1.html

7. National Library of Australia Digital Object Repository. http://www.nla.gov.au/digicoll/oai/

8. OAISter Home. http://oaister.umdl.umich.edu/o/oaister/

9. University of Illinois Open Archives Initiative Metadata Harvesting Project. http://oai.grainger.uiuc.edu/

10. MacquarieNet. http://www.macnet.mq.edu.au/

11. RLG Cultural Materials -- What is it? http://culturalmaterials.rlg.org/

12. Persistent identifier scheme adopted by the National Library of Australia. http://www.nla.gov.au/initiatives/nlapi.html

13. Our Catalogue – National Library of Australia. http://www.nla.gov.au/catalogue/

14. Pictures Catalogue - National Library of Australia. http://www.nla.gov.au/catalogue/pictures/

15. OAICat [OCLC software]. http://www.oclc.org/research/software/oai/cat.htm

16. TeraText Metadata Publishing System. http://www.teratext.com/

Top

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us