National Library of Australia Staff Papers, 2001

Font Size:  Small  Medium  Large

Archiving the Web

The PANDORA Archive at the National Library of Australia

A paper by Warwick Cathro, Colin Webb and Julie Whiting.
Presented by Warwick Cathro Assistant Director-General, Information Technology, National Library of Australia at the Preserving the Present for the Future Web Archiving Conference, Copenhagen, 18-19 June 2001

Abstract

The National Library of Australia maintains PANDORA, an archive of selected, significant Australian web sites and web-based online publications. The purpose of PANDORA is to ensure that Australians of the future will be able to access a significant component of today’s Australian web based information resources.

This paper describes:

  • the overall framework for digital collection management at the Library which forms the context for PANDORA;
  • the reasons why the Library has undertaken a selective approach to archiving the Web;
  • the process of gathering, checking and presenting the archived copies of the web sites, and how this is supported by software developed recently by the Library;
  • the use of persistent identifiers for the items in PANDORA;
  • the approach which the Library has taken to access control;
  • the Library’s plans to preserve access to the content of the archive; and
  • possible opportunities for global cooperation, such as the sharing of information on the best ways to migrate obsolete files, and better information exchange between archives, libraries and researchers working in digital preservation.

Introduction

During the past eight years the World Wide Web has developed into an essential global mechanism for information access and delivery. A large proportion of information providers in the academic, government, commercial and community sectors use the Web to deliver current information about their services and activities. Hardly any of these information providers are concerned about the use of this information as a form of cultural heritage, as documentation to support future research.

Libraries, museums and archives have such a concern. Unlike most information providers, they have a genuine concern for the users of the future, a "moral imperative" to transmit the knowledge of the past not only to the present, but to the future 1.

Libraries and other memory institutions have always faced significant challenges in discharging this stewardship role, given the costs involved in storing and conserving their collections. In the era of digital information, these challenges have now increased manyfold.

At the National Library of Australia, we accepted in the early years of the Web that online publications are important social, intellectual and cultural resources and that our responsibilities in relation to this material were no less than for the material which we traditionally collected 2.

Computer technology is changing so rapidly that the interfaces, technical standards and file structures which will be used by computers even 20 years into the future are likely to be quite different from those used today. For national libraries and other research libraries, which have a tradition of building collections to meet the needs of scholars many decades or even centuries into the future, this presents a formidable challenge. The ability to access and read digital information in the future will depend on strategies such as migration (in which the data is migrated, if technically feasible, to new operating systems and data structures) or emulation (in which modern computers emulate the operating systems and data structures of previous eras).

A number of eminent commentators have expressed serious doubts about the likely success of digital preservation. For example, Peter Graham commented 3:

The investment necessary to migrate files of data will involve skilled labour, complex record-keeping, physical piece management, checking for successful outcomes, space and equipment. A comparable library [project for] data migration cost and complexity at approximately this order of magnitude would be the orderly photocopying of books in the collection every five years.

Many international research efforts are actively seeking practical solutions to the digital preservation dilemma. There is a great imperative to find a solution, or a combination of solutions. As Lynne Brindley (the British Librarian) said in a recent keynote speech 4 :

We simply do not have the luxury that our predecessors in their curation of printed materials enjoyed, of being able to rely on ‘benign neglect’ as a plank of our digital preservation strategy. Action has to be taken at the outset to ensure viable and sustainable access to digital content.

The PANDORA Archive

The first step in ensuring "viable and sustainable access" is to make sure that the resource survives, by getting it into a safe place where it can be stored, accessed and managed. In 1996, the National Library of Australia began to work on this first step, by starting to build an archive of selected, significant Australian web sites and web-based online publications - the PANDORA archive 5. This paper describes the experience of the Library in developing and maintaining PANDORA.

The name PANDORA derives from its goal of "Preserving and Accessing Networked DOcumentary Resources of Australia". PANDORA is a highly selective archive: in June 2001, it contained just 1250 web sites. Nevertheless, it already constitutes a strongly representative sample of Australian web publishing by academic, government, commercial and community organisations. Several of the web sites captured in the archive - including the official web site for the Sydney Olympic Games - have already disappeared from the live Internet. Moreover, about one-third of the 1250 web sites have been captured on multiple occasions, allowing the archive to build up a sequence of snapshots which demonstrate how these sites have changed over time.

In June 2001, the PANDORA archive:

  • contained about 5 million files and about 670,000 directories;
  • comprised about 134 Gb of storage; and
  • was growing at about 500 new titles and about 400 re-gathers per year.

PANDORA is now a routine part of the selection, acquisition and collection management processes of the Library. Organisationally, it is managed by the Electronic Unit which forms part of the Collections Management Division of the Library, alongside organisational units such as Monograph Acquisition and Cataloguing, and Serials Processing. PANDORA is characterised by the following features:

  • it is based on well defined selection guidelines which fit within the Library’s overall Collection Development Policy;
  • web sites are gathered and managed using software called PANDAS (PANdora Digital Archiving System) which is described below;
  • the gathering is undertaken by partner institutions (the State Library of Victoria, the State Library of South Australia and ScreenSound Australia) in addition to the Library itself;
  • permission of the publisher is sought and received prior to any site being included in the archive;
  • every web site is catalogued, and the catalogue entries are included in the Library’s Catalogue, the National Bibliographic Database and PANDORA web site, providing multiple paths for resource discovery which integrate with the discovery of other information resources; and
  • the archived version of every web site is subjected to quality checking to ensure that all files have been correctly captured.

The Context for PANDORA

PANDORA does not sit in isolation, but forms part of an overall framework for digital collection management at the Library. This framework embraces all of the key library processes of selection, acquisition, storage, resource discovery, delivery, access control and preservation.

The Library’s digital collection management framework covers both digitised and born-digital information resources. This framework is set out in the Digital Services Project Information Paper 6 which the Library published in December 1998.

In addition to its work in capturing Australian web sites for the PANDORA archive, the Library has an active digitisation program which is aimed at improving national and global access to its unique collections of pictorial, manuscript, rare map and music collections. The digitisation program and the PANDORA gathering process constitute the Library’s principal mechanisms for the acquisition of digital information resources.

In addition to these activities, the Library is examining the feasibility of capturing the entire Australian web domain, in a similar project to those undertaken in recent years for the Swedish, Icelandic and Finnish domains, and by the Internet Archive. The Library engaged a consultant in May 2001 to undertake this feasibility study.

The digital objects acquired through digitisation and the PANDORA gathering process will be stored using infrastructure which the Library is currently implementing through its Digital Object Storage System Project 7. The Library entered into a contract in June 2001 for the supply of two terabytes of disk storage (upgradeable to 20 terabytes) and for associated tape backup and storage management facilities.

Data about all of these digital objects will be stored and managed in a Digital Object Management System 8. This system will support the activities of the collection managers, and will contain the data necessary to support migration and other activities aimed at providing long term access to the objects. The Library has developed specifications for its Digital Object Management System, and is currently seeking the development or acquisition of software to meet these specifications.

The Library recognises that it will need to take action to preserve long term access to these digital collections and is planning steps, such as the migration of obsolete file types, to support this objective.

The Library supports access to these digital collections through mainstream pathways such as its catalogue and the National Bibliographic Database. This policy is based on the Library’s firm view that access to all information resources (digital and traditional) should be integrated through the use of such pathways. The Library has supplemented these pathways by developing specialised resource discovery services such as the one which it developed for its "Picture Australia" service. These services are based on the Blue Angel Technologies "MetaStar Enterprise" software which the Library acquired in 1999 following a procurement process 9.

PANDORA: Selection Issues

The Library and its partners do not attempt to capture everything published online that relates to Australia. A similar situation exists with traditional collections such as print publications, because much of what is published on the Web is the equivalent of "printed ephemera". The Library’s policy of selective acquisition is influenced by resource issues and by the Library's goal that what it collects and preserves must be accessible. Achieving this goal with digital materials is resource intensive, given that the Library aims to:

  • apply quality control to the collecting process;
  • negotiate explicit access rights with rights owners;
  • develop a cooperative relationship with creators and publishers;
  • maintain knowledge of new web design features and file formats which the archive will need to manage in the future;
  • record the metadata needed for future preservation decisions; and
  • strive to maintain access to the diverse and complex files in the archive.

The Library has developed a set of guidelines10 for its selection of web sites and online publications for the PANDORA archive. These guidelines take account of issues such as: Australian content, Australian authorship, the "authoritative status" of the author, research value, whether the publication exists in hardcopy form, whether there has been any "quality control process" in the online publishing, public interest in the subject matter, and whether the publication is indexed by a recognised indexing service.

PANDORA has concentrated on selecting titles which are available in online format only: in other words, if there is a print equivalent, the Library will collect the print version in preference to the digital version. This policy is influenced by the belief that print will be easier to preserve, as well as by the need to limit the number of publications eligible for inclusion in the archive. To date the application of this principle has eliminated about seventy percent of titles that would otherwise fall within the PANDORA selection guidelines. The Library will however continue to review this policy in the light of user expectations.

Other archiving agencies, such as the Internet Archive and the Royal Library of Sweden, have taken a different approach by aiming to collect and preserve everything on the Web that is relevant to their collecting focus. The National Library of Australia recognises that there are arguments in favour of a comprehensive approach, including the difficulty of predicting what will be important in the future, and the significantly lower gathering costs per site captured. For these reasons, as indicated above, the Library has commenced a feasibility study into the potential capture of the entire Australian web domain. Nevertheless, the Library believes that the continued existence of PANDORA as a quality-controlled, well-managed and accessible archive requires a continued commitment to a selective approach, even if this is supplemented by whole-of-domain capture.

Acquisition Issues and the PANDAS System

The process of gathering web sites and incorporating them into an archive raises a number of challenges. For example, the process must aim to preserve fidelity to the original resource, deal with links to external resources, and support an efficient workflow for quality checking and deal with sites where access is restricted.

To meet these challenges, a PANDORA Digital Archiving System (PANDAS) was developed in-house by the Library during 2000 and 2001. It was implemented in June 2001.

Specifications for the system were developed by the Library’s Electronic Unit, and the system was built by contract staff using WebObjects11 as an Application Development Environment. The system is designed to support the following functions:

  • manage the metadata about titles that have been both selected and rejected for inclusion in the archive;
  • initiate gathers of titles to be archived;
  • manage the quality checking and problem fixing process;
  • prepare the item for public display and generate a title entry page;
  • manage access restrictions; and
  • provide management reports.

PANDAS replaced less functional systems which were based on a MS Access database and some paper-based record keeping. It allows the PANDORA partners to manage a more efficient workflow, using the central PANDAS system. While currently all titles are stored and delivered from the Library’s server, the system will support distributed content as an option for the future.

PANDAS is accessed using Internet Explorer 5.5 (although public access to the archive may use any web browser). Web access to all functions of PANDAS ensures that the archiving partners have no need for special desktop software to use the system.

Administrative metadata

PANDAS allows collection managers to store administrative metadata about the web site being archived, such as:

  • descriptive information (title, URL, persistent identifier, publisher details, etc.);
  • permission status (whether permission to archive was granted or denied by the publisher);
  • type of resource;
  • status (selected, rejected, still being monitored);
  • subjects (selected from a list of about 20 broad categories);
  • collection name (if the title is being included as part of a collection of sites on a particular event or topic such as the Olympic Games, election campaigns);
  • restriction information (e.g. access in-house only for specified period, then made freely available ; access restricted to designated users only); and
  • gathering schedule.

The gathering schedule can specify that a site should be archived only once, or it can specify a gathering frequency (monthly, quarterly, annually etc.). It is also possible to enter a date list so that an item is gathered on specific dates, such as for elections or special events.

Gathering of sites

The majority of items in PANDORA are gathered from the publisher’s site. The "pull" approach minimises the impact of archiving on publishers and enables the Library to determine the most appropriate gathering schedule or capture time.

The gathering features of PANDAS have been designed to allow a variety of website copying software to be "plugged-in" while the user interface remains the same. Currently, the primary tool used is HTTrack12. PANDAS provides a wide range of options allowing the collection manager to control the scope and timing of the gathering process.

As part of the gathering, technical metadata about the file types captured is recorded by the system. This information is used to automatically generate public notes listing any plug-ins required for viewing the site. It will also be used for preservation analysis of the PANDORA files.

Checking the site

Since the items in PANDORA have been selected as nationally significant, the Library is committed to ensuring that the data is as complete and accurate as possible. A web site analysis tool (Linkbot) is used to automatically check the gathered site for missing files. While this is a useful process, much more work needs to be done to ensure that the look and feel of the sites captured in PANDORA is as close as possible to the original site. Staff do a manual check of the whole site to identify problems such Javascript not working correctly, missing files (some types don’t show up in link checks) or broken links due to coding errors or case sensitivity.

These are time consuming processes, and some problems cannot be completely rectified. If Electronic Unit staff cannot fix problems, the archived copy of the site is handed over to IT specialists for fixing.

Preparation of the site for public display

After a site has been gathered and checked for completeness, it must be prepared for public access in the PANDORA archive.

A script is used to automatically disable external links, forms, and mailto addresses. When users click on one of these links in the archived copy, a screen will appear giving them the option to leave the archived version of the site and access the link to the live site (if the latter still exists).

A public "title entry page" is generated for new titles, or is updated when a new instance of a title is gathered. The title entry page indicates whether the title is being gathered regularly, provides a link to the publishers live site, lists all the archived instances of the title, provides information about software plug-ins required to view it, and indicates any access restrictions. It also provides a link to the publisher's copyright statement if there is one, and to the Library’s own generic copyright warning.

The PANDORA subject and alphabetic title listings are also automatically updated when new titles are added to the archive.

Access: Persistent Identifiers

Since the files stored in PANDORA are intended to be preserved for posterity, they need to be accessible indefinitely and therefore require persistent identification.

These files (and other significant information resources on the web) may be cited in many places: in other web documents, in scholarly articles, in library catalogues and databases, and in bookmark files within browsers. Currently the only widely deployed standard for citing and locating a resource on the Web is the Uniform Resource Locator (URL). The URL describes the current location of a resource, and is subject to change if the resource is moved - which frequently happens in practice.

The rationale and requirements for a successful persistent identifier system have been described in a recent paper by Diana Dack13, who was engaged as a consultant by the Library. This consultancy assisted the Library to determine its policy on naming standards and its procedures for managing digital information resources for persistent access.

Given that a global, operational system of persistent identifiers does not yet exist, the Library will identify the digital resources which it manages using identifiers that are not dependent on a domain name to provide unique identification of the resource. While these identifiers would not rely on a domain name or a namespace identifier to make them unique, they could be used as the namespace specific string in the context of any naming system implemented at a later date.

In the case of PANDORA, it is necessary to automatically assign identifiers to each file because of the large number of files that are harvested. The form of identifier which has been developed for files stored in PANDORA is set out in the Appendix. This standard will ensure that each file will have a persistent unique identifier, that there will be sufficient granularity of identification to support preservation activities, and that there will be sufficient "intelligence" to enable a measure of grouping and relating of versions.

However, there still remains a significant issue: how to relate the archival persistent identifier to the original resource, so that those who have a citation to the original resource will be able to find the archival copy? Possible solutions to this problem include the use of metadata, the use of re-directs by publisher sites, and the use of multiple resolution if the DOI (Digital Object Identifier) or similar standards become widely deployed.

Access Control

Access restrictions are applied to a number of sites in PANDORA, and these are managed by the PANDAS system. Restrictions may be applied for commercial reasons, for privacy or cultural reasons, or as part of a policy decision for certain categories of material. Items can be restricted for a specified time period to onsite use only in one or more of the partner agencies, or can be password protected so that only designated researchers can obtain access.

When a title that is available only via purchase or subscription is selected for PANDORA, the Library negotiates with the publisher to determine access conditions that will not undermine the publisher’s commercial viability. The preferred model is for the publisher to allow onsite use (for example, within public reading rooms) at the partner agencies for an agreed time (say five years) after which the title would become freely available to external users. Agreement on this type of restriction has been reached with several commercial ebook and journal publishers. If the publisher is unwilling to allow this, other options (such as onsite use in only one agency or for a longer time period) may be negotiated.

Some material of a sensitive nature (such as indigenous material, personal accounts, and opinions expressed in closed discussion lists) may also require restricted access. For these categories of material suitable time restrictions are negotiated, or in some cases only password controlled access to the title by designated users is permitted.

In rare cases, no public access to a title may be allowed. In some cases where permission for access in PANDORA has been denied, a title is considered so significant that it has been captured in the hope that it can eventually be made available. Items which include libellous or other legally questionable material may be restricted to staff only. Some adult material has been selected for inclusion in PANDORA to reflect the widespread availability of such material on the Internet. While this material is publicly accessible on the Internet, a decision has been made to allow only password controlled access to the archived versions in PANDORA.

PANDAS manages all of these restrictions and allows only those users in designated locations (based on IP address) or with the required password to access the archived versions. As time periods for restrictions expire, the system automatically updates the title entry pages to indicate the changed access conditions.

The Preservation Process

The ultimate purpose of the PANDORA archive is to ensure that Australians of the future will be able to access a significant component of today’s Australian web based information resources, just as they will be able to access today’s more traditional information resources, such as books and newspapers. Achieving this goal will require the Library to undertake active measures to preserve access to the information resources stored in PANDORA.

As mentioned above, a first step in ensuring viable and sustainable access involves creating and managing a safe place where digital resources can be handled as resources of enduring value. However, this still leaves much work to do in ensuring that the resources gathered into the digital archive remain viable and accessible in the long term. To date, the Library has made reasonable progress with the first step, but is still in the early stages of developing a response to the long-term preservation challenge.

Given the diverse collections for which it is responsible, the Library is most likely to use a combination of strategies for the foreseeable future. These strategies will include:

  • some technology preservation, including maintenance of software and even some hardware;
  • negotiating with publishers to supply stable source files of some streaming or dynamic formats;
  • migration strategies for those file formats which correspond to compatible new formats, and which are amenable to mass conversion;
  • use of emulators, if they can be found or developed, for some file formats; and
  • simply keeping and refreshing some files, not amenable to migration or emulation, in the hope that a suitable access pathway will emerge.

Where migration is undertaken, the Library expects to store and manage multiple copies of archived files, maintaining both currently accessible versions and versions of files in their native formats.

The Library plans to commence a program of file migration in the PANDORA archive during the next year. In preparation for this, the Library has:

  • identified file formats held in the PANDORA archive, with a view to identifying an effective preservation path for each format; and
  • identified dead and deprecated HTML tags in the archive, which may not be recognised by later browsers.

An analysis undertaken in mid 2000 identified:

  • 127 tags dead in HTML 4.0;
  • 7 million tags due to be made non-standard in later versions of HTML; and
  • 14 million tags with deprecated attributes.

The Library will undertake a trial to migrate these obsolete and deprecated tags to equivalent tags that are valid in the latest version of HTML.

Preservation Metadata

A successful preservation process relies to a large extent on descriptions of the nature and history of the archived resources: on metadata, in short. Metadata is also required to guide the way archived objects are rendered to and understood by users.

So what metadata is likely to be adequate? In addressing this question, the Library in 1999 developed and published a draft set of digital preservation metadata14. In 2000, the Research Libraries Group (RLG) and the Online Computer Library Center (OCLC) invited the Library and a number of others to join an international working group charged with the task of proposing a draft international standard for preservation metadata.

The working group has taken as its starting point the approaches to preservation metadata of the NEDLIB Project in Europe, the Cedars Project in the UK, and Harvard University Library's Data Repository Service, as well as the Library’s draft. A White Paper15 issued in January 2001 compared these approaches as a springboard for defining a comprehensive, structured preservation metadata framework, applicable to a broad range of digital object types, archival processes and institutions.

The Need for Cooperative Action

Last year saw the release of a major report (familiarly known as the LC21 report) on the digital strategies of the Library of Congress 16. The report found that the Library of Congress "lags significantly in receiving and archiving the born-digital product of the nation". The Library, it said "cannot go on as before". Amongst the key actions which the LC21 report advocated were:

  • greater cooperation in digital collection building (across geographical and sectoral boundaries);
  • greater power - legislative and technical - to collect digital publications; and
  • a stronger engagement in digital preservation with electronic publishers, other major collecting institutions and the research community.

While the LC21 report was directed at the Library of Congress, it really speaks to all of us. We are all facing this challenge and all of our energies will be needed to address it.

Cooperation can take a number of forms. For example, there is a need to share information about the best ways to migrate obsolete file types. There is also a pressing need for rigorous scientific research to seek reliable and affordable solutions to the digital preservation problem, and for better information exchange between archives, libraries and researchers working in digital preservation.

Conclusions

The National Library of Australia has undertaken a selective approach to archiving the Web by developing the PANDORA archive and making it a routine part of the Library’s collection management activities. PANDORA constitutes a strongly representative sample of Australian web publishing by academic, government, commercial and community organisations. Several of the web sites captured in PANDORA have disappeared from the live Internet.

The content of PANDORA is based on a set of selection guidelines, and reflects the fact that every archived web site is subject to quality control, persistent identification, cataloguing and access permission from the publisher. As a commitment to its continuing support for PANDORA, the Library has recently developed and implemented software to make the site gathering and quality checking process more efficient.

The Library believes that such a "quality archive" can serve an important role in preserving access for future generations to today’s web based information resources.

References

  1. Battin, Patricia. Beyond crumbling books: preservation strategies for the nineties. Libraries: the heart of the matter: proceedings of the Australian Library and Information Association 2nd Biennial Conference (1992) : 15-18.
  2. Fullerton, Jan. The information environment in the 21st century: societal issues. Paper presented to the Australian Library and Information Association Biennial Conference, Canberra, 2000.
  3. Graham, Peter S. Intellectual preservation and electronic intellectual property. <http://www.ifla.org/documents/infopol/copyright/graham.txt>.
  4. Brindley, Lynne. Preservation 2000: keynote speech. <http://www.bl.uk/concord/otherpubs_speeches04.html>
  5. National Library of Australia. PANDORA: preserving and accessing documentary resources of Australia. <http://pandora.nla.gov.au/>.
  6. National Library of Australia. Digital Services Project Information Paper. <http://www.nla.gov.au/dsp/infopaper.html>
  7. National Library of Australia. Digital Object Storage System Request for Quotation. <http://www.nla.gov.au/dsp/doss/>
  8. National Library of Australia. Digital Object Management System Request for Quotation. <http://www.nla.gov.au/dsp/doms/>
  9. National Library of Australia. Metadata Repository and Search Service Request for Quotation. <http://www.nla.gov.au/dsp/rfq/index.html>
  10. National Library of Australia. PANDORA Selection Guidelines. <http://pandora.nla.gov.au/selectionguidelines.html>
  11. WebObjects5: the incredibly powerful way to create Java Server applications. <http://www.apple.com/webobjects/>
  12. HTTRack Website Copier 3. <http://www.httrack.com/>.
  13. Dack, Diana. Persistence is a virtue. Paper presented to the Online Information Conference, Sydney, January 2001. <http://www.nla.gov.au/nla/staffpaper/2001/dack.html>
  14. National Library of Australia. Preservation metadata for digital collections: exposure draft. <http://www.nla.gov.au/preserve/pmeta.html>
  15. Preservation metadata for digital objects: a review of the state of the art. <http://www.oclc.org/digitalpreservation/presmeta_wp.pdf>
  16. LC21: a digital strategy for the Library of Congress. Washington, DC : National Academy Press, 2000.

Appendix: PANDORA Persistent Identifier Standard

A recent consultancy recommended the following form of identifier for files stored in PANDORA:

<collection id>-<work identifier>-<archive date>-<publisher’s URI>-<generation code>

where

<collection id>: for the archival collections the collection ID will be "nla.arc"

<work identifier>: a unique number within the digital archival collections assigned to the parent work of which the resource is a component

<archive date> the date the file was archived in the format YYYYMMDD

<publisher’s URI> currently the host name, path name and file name of the resource on the publisher’s site

<generation code>: a two digit code representing the version of a resource which has been migrated from its original format.

Comments on this article

View all comments