Staff Papers
PANDORA, Australia's Web Archive, and the Digital Archiving System that Supports it
A paper written by Margaret Phillips, Director, Digital Archiving, National Library of Australia, published in DigiCULT.info, Issue 6, December 2003, p. 24.
The role of national libraries in Web harvesting
The primary role of a national library is to develop and maintain comprehensive collections of documentary material relating to its country and its people, to preserve them and to make them available for research now and in the future. During the last ten years, national libraries and other 'memory' institutions have had some difficult decisions to make regarding the extension of this role to electronic formats.
National libraries have accepted responsibility for online publications and a small number have embarked on programs to develop collections, or 'archives', of them.
Having accepted the responsibility, proceeding with it has been no easy task. In the print world the principles and procedures are straightforward and well established and usually supported by legal deposit legislation, requiring publishers to deposit one or more copies of every publication with the national library. Preservation is also a relatively simple task with print material. As long as attention is paid to the security of the collection and it is kept under controlled conditions of temperature and humidity, there is an excellent chance that most of it will last hundreds of years.
The situation to be addressed with online publications is much more complex (and therefore much more costly) in every respect and national libraries have responded to it in a variety of ways. One of the first and fundamental decisions to be made is whether to archive online publications comprehensively (whole domain harvesting) or selectively. There are advantages and disadvantages with both approaches1 and some national libraries such as Sweden, Norway, Finland and Iceland have taken the comprehensive approach, while Canada, Australia and Japan are archiving selectively.
As Web archiving technologies and practices mature, hybrid, more sophisticated approaches will be possible. An ideal situation would be a selective archive complemented by periodic snapshots of whole or part of a country's domain and/or subject-based snapshots such as the Election 2002 Web Archive2 developed by the Library of Congress and the Internet Archive. The National Library of Australia keeps a close watch on developments in whole domain archiving with this possibility in view. It is also interested in the more focused approach being explored by the Bibliothèque Nationale de France, which would used automated methods to identify and archive significant sites3 .
Another basic decision is whether the publications will be acquired, or 'ingested', by deposit from the publisher or by web harvesting using a harvesting 'robot', or 'crawler'. Almost all archives established to date rely on web harvesting, but the National Library of the Netherlands has built a major archive based on the deposit model.
Regardless of approach the archiving library must:
- Identify what is being published;
- Decide what is worth saving;
- Define roles and responsibilities of different stakeholders;
- Protect archived publications against loss;
- Preserve the integrity and guarantee the authenticity of publications; and
- Ensure long-term access and usability.
PANDORA: Australia's Web Archive
The National Library of Australia selectively harvests Australian online publications for inclusion in PANDORA, Australia's Web Archive. It does so in cooperation with eight other PANDORA partners all of whom have responsibilities for collecting and preserving Australian documentary heritage: the mainland State libraries, the Northern Territory Library, ScreenSound Australia (the National Film and Sound Archive) and the Australian War Memorial are all participants.
Each of the partner agencies identifies, selects, catalogues, and archives publications for which they accept responsibility as defined by their selection guidelines. After seven years of archiving Australian online publications, the National Library reviewed its guidelines4 in early 2003 and set priorities to focus effort on six main categories:
- Commonwealth and Australian Capital Territory government publications
- Conference proceedings
- E-journals
- Publications of tertiary education institutions
- Titles referred by indexing and abstracting agencies
- Topical sites
- Sites in nominated subject areas (defined in Appendix 2 of the selection guidelines) that will be collected on a rolling three year basis; and
- Sites documenting key issues of current social or political interest, such as election sites, Sydney Olympics, the Bali bombing.
The selective approach to archiving enables PANDORA partners to realise some important objectives:
- Each item in the Archive is quality assessed and functional to the fullest extent permitted by current technical and resource capabilities;
- Each item in the Archive can be fully catalogued and therefore can be included in the national bibliography;
- Each item in the Archive can be made accessible owing to the fact that permission to archive and make publications available via the Web has been negotiated with the publishers;
- The 'significant properties' of resources within the archive can be analysed and determined both for individual resources and for classes of them. This enhances our knowledge of preservation requirements and enables strategies for preservation to be put into place;
The PANDORA Archive is now over half a terabyte in size, contains just under 5,000 titles and over 9,000 'instances'. Many of the titles in the Archive are gathered on regular basis to capture new content, and each new gathering is referred to as an 'instance'.
With the permission of publishers, titles are harvested using HTTrack, a freely available Web site offline browser and mirroring tool5 . A small number of titles are not available on the Web but are distributed by email and these are received directly from the publisher. Most titles in the Archive are freely available on the Web, but approximately 100 are commercial publications. To protect the publishers' interests access to these titles is restricted during the period of commercial viability nominated by the publisher. During this period the titles are available for consultation in the reading room of the archiving partner only, and the PANDORA digital archiving system manages this process automatically.
PANDORA Digital Archiving System (PANDAS)
To achieve the greatest possible efficiency in creating a collaborative, selective archive, the National Library of Australia has devoted considerable effort to developing a digital archiving system to manage the process. The original intention was to buy a system off the shelf, as defined in the Information Paper, issued as part of the Digital Services Project6 . The initiative did not find any system that met PANDORA's needs and the Library had no alternative but to develop one itself. In June 2001 the Library implemented the first version of the PANDORA Digital Archiving System (PANDAS) and version 2 followed in August 2002. Subsequent enhancements have brought us to version 2.1.3.
The PANDAS software supports the following archiving functions:
- Manage the metadata about titles that have been both selected and rejected for inclusion in the archive;
- Initiate gathers of titles selected for archiving;
- Manage the quality checking and problem fixing process;
- Prepare items for public display and generate a title entry page;
- Manage access restrictions; and
- Provide management reports.
The software is Web-based and enables all eight partners from their remote locations to carry out all of the tasks necessary to download and store titles in the central archive located on the National Library's server. The software runs on a standard PC and with Internet Explorer 5.5 or above.
The collection managers use PANDAS to store descriptive, administrative and preservation metadata about the Web site being archived, such as:
- Descriptive information (title, URL, persistent identifier, publisher details);
- Permission status (whether permission to archive was granted or denied by the publisher;
- Type of resource;
- Status (selected, rejected, still being monitored;
- Subjects (selected from a list of broad categories);
- Collection name (if the title is being included as part of a collection of sites on a particular event or topic, such as an election campaign);
- Restriction information(e.g., access in-house only for a specified period);
- Gathering schedule.
The gathering schedule can specify that a site should be archived once only, or it can specify a gathering frequency (monthly, quarterly, annually, etc.). It is also possible to enter a date list so that an item is gathered on specific dates.
The PANDAS software allows a variety of Web site copying software to be 'plugged-in' and allows the collection manager to control the scope and timing of the gathering process. As part of the gathering process, technical metadata about the file types captured is recorded by the system. This information is used to generate automatically public notes listing any plug-ins required for viewing the site.
PANDAS currently does not have any automatic quality assurance feature such as a link checking feature. Collection managers must manually check the whole site to identify problems, such as Javascript not working correctly, missing files, or broken links due to coding errors or case sensitivity.
After a site has been gathered and checked for completeness, it is prepared for public access in the Archive. A script is used to disable automatically external links, forms and mailto addresses. When users click on one of these links in the archived copy, a screen will appear giving them the option to leave the archived version of the site and access the link to the live site (if the latter still exists).
A public 'title entry page' is generated for new titles, or is updated when a new instance of a title is gathered. The title entry page indicates whether the title is being gathered regularly, provides a link to the publishers live site if it still exists, lists all the archived instances of the title, provides information about software required to view it, and indicates any access restrictions. It also provides a link to the publisher's copyright statement if there is one, and to the National Library's own generic copyright warning.
Subject and alphabetic title listings are also automatically updated when new titles are added to the Archive.
The PANDAS software manages access restrictions on the titles that require them. Restrictions may be applied for commercial reasons, for privacy or cultural reasons, or as part of a policy decision for certain categories of material. Items can be restricted for a specified time period to use within the Library building only, or can be password protected so that designated users can obtain access. PANDAS manages these restrictions and allows only those researchers in designated locations (based on IP address) or with the required password to access the archived versions. As time periods for restrictions expire, the system automatically updates the title entry pages in indicate the change access conditions.
Future directions
The selective approach to archiving is very labour-intensive. While PANDAS has enabled us to realise some efficiencies and to increase productivity, it still requires manual entry of metadata and processing of titles one by one.
In order to be able to deal with the increasing volume of online publishing, particularly in the government sector, the Library is undertaking a pilot project to explore the possibility of automating or semi-automating the processes of identifying, selecting, describing and archiving Commonwealth government publications. To automate identification, selection and description, it is working with seven government agencies to examine the metadata they produce and to refine workflows for making it available to the National Library for inclusion in the National Bibliographic Database (NBD)7 . The aim is then to distil from this a small number of best practice models which all government agencies could use to contribute metadata to the NBD. In a second phase of the pilot project, metadata would be extracted from the NBD and batch loaded into PANDAS for archiving.
Before this can be done, PANDAS will need to be further developed and this will be undertaken as part of the development of version 3.
The main driving forces for the future development of PANDAS are:
- The need to be able to ingest a much larger volume of online publications and their associated metadata, automatically or semi-automatically;
- The need to improve discovery of and access to items in the burgeoning Archive in a variety of ways;
- The need for PANDORA to be able to interface with data acquired other than by the current selective harvesting method. This may include large amounts of publisher-supplied data, partial or whole domain harvests, and publications and databases from the 'deep web' obtained as a result of the deep web archiving initiative described below;
- The need for a collection manager interface to the preservation metadata, and mechanisms to support preservation processes; and
- The need for better and more flexible reports from the system.
PANDAS Evaluation System
Because libraries and other collecting institutions around the world are beginning to take on the responsibilities of collecting digital publications, and because as yet there are few digital archiving systems available to assist them, the National Library of Australia has received a number of enquiries about use of the PANDAS software. The Library has decided to make the software available on a cost recovery basis, although it will be limited in the number of agencies it can accommodate. To enable interested libraries to assess whether PANDAS does in fact meet their business needs, the PANDAS Evaluation System has been developed and has recently been made available to the first evaluating library. A maximum of three agencies at any one time can evaluate the software.
Archiving the 'deep web'
To date titles archived in PANDORA have mostly been gathered using a robot that follows html links from the root URL submitted as the starting point for gathering.
A very large part of the Web consists of sites not accessible via html links but stored in structured databases, which it is necessary to query for presentation of information. These queries are typically made via selection of options from drop down boxes or by entering terms into a search engine. This requires intelligence that gathering robots and search engine indexers do not have at this stage. The hidden nature of data stored and accessed in this way has led to it being known as the 'deep web'.
The 'deep web' presents special challenges to national libraries that are endeavouring to identify and archive a nation's output in this form, whether they are taking the selective or comprehensive approach. (Both rely on harvesting robots.) The National Library of Australia has embarked on a research project to find ways of taking in, managing and providing access to publications and Web sites structured as databases. It is also participating in the Deep Web Archiving Group of the recently-formed International Internet Preservation Consortium8 and will collaborate with other members in developing solutions and sharing results.
Conclusion
The business of building a digital archive, managing it long-term, and providing access to it is a complex and costly operation. A digital archiving system, which manages workflows, stores the digital objects themselves, as well as descriptive, administrative and preservation metadata, and provides management reports is essential to support it. This system must develop hand in hand with the archive itself, growing to facilitate expanding business requirements and enhancing productivity at every stage. The National Library of Australia is just about to embark on another development cycle for its digital archiving system.
Endnotes
- A summary of the advantages and disadvantages of both the comprehensive and selective approaches is given in sections 3-5 of: Phillips, M.E. (2003) Collecting Australian Online Publications. Online. Available at http://pandora.nla.gov.au/BSC49.doc
- Library of Congress. Election 2002 Web Archive. Online. Available at http://www.loc.gov/minerva/collect/elec2002/
- Masanès, Julien. (2002). 'Towards Continuous Web Archiving: First Results and an Agenda for the Future' in D-Lib Magazine, vol. 8 no. 12, December 2002. Online. Available at http://www.dlib.org/dlib/december02/masanes/12masanes.html
- National Library of Australia. (2003). Online Australian Publications: Selection Guidelines for Archiving and Preservation by the National Library of Australia. Online. Available at http://pandora.nla.gov.au/selectionguidelines.html
- Roche, Xavier. (2003). HTTrack Website Copier: Open Source Offllne Browser. Online. Available at http://www.httrack.com
- National Library of Australia. (1998). Digital Services Project: Information Paper. Online. Available at http://www.nla.gov.au/dsp/infopaper.html
- The National Bibliographic Database is a database to which approximately 1,100 Australian libraries contribute catalogue records and library holdings and it is made available through the Kinetica service.
- The International Internet Preservation Consortium was established in July 2003 under the leadership of the Bibliothèque Nationale de France.
