Staff Papers

Archiving the Web: the National Collection of Australian Online Publications

A paper presented by Margaret Phillips1, Manager, Digital Archiving, National Library of Australia at the International Symposium on Web Archiving, National Diet Library, Tokyo, Japan, 30 January 2002

Recognising online publications as an intrinsic part of the national documentary heritage, the National Library of Australia, together with a number of partners, is building the National Collection of Australian Online Publications. The purpose of the National Collection is to ensure that Australians of the future will be able to access today’s significant Australian online information resources.

This paper describes what we are doing, why we are doing it, and how. It also discusses the lessons we have learned, and aspects of web archiving that we have still to address adequately.

I do not need to tell this audience about the challenges to libraries in collecting, storing and conserving the digital materials for which they are responsible. You know these challenges, are facing them, and that’s why we are all here today, to learn from each other. So I shall go straight on to the situation in Australia and how we have addressed them.

Background - The Australian Situation

In Australia there are three layers of government – National, State and Local - and the libraries of two of these – National and State - are responsible for collecting and preserving the documentary heritage of their respective jurisdictions. At the national level, the National Library is responsible for the published output of the nation as a whole. The State libraries are responsible for building collections of works published in their respective States. Each jurisdiction has legal deposit legislation, some of which includes electronic publications and some of which does not. I shall speak about the national legislation further on.

The print collections of these National and State libraries have necessarily remained quite separate, though resource discovery has been integrated through the National Bibliographic Database, which is a national union catalogue that records the collections and holdings of 1,100 Australian libraries. In the online environment, the National Library quickly saw not only the possibility of, but also the necessity for, collaborating with the State libraries to build one national collection of Australian online publications. Especially in the case of freely available Internet publications, it is possible to collect just one copy of a title in order to provide access to all Australians, indeed, to everyone in the world. Since the management of electronic information resources is such an expensive business, sharing the work and the cost would enable a collection of greater scope and depth to be developed.

Legal Deposit and Building a Collection of the National Imprint

In Australia at the national level, legal deposit provisions are contained within the Copyright Act 1968, which is under review. In a joint submission to the Copyright Law Review Committee by the National Library of Australia and the National Film and Sound Archive, it was recommended

…[t]hat the scope of publications to be covered by the legal deposit provisions of the revised Copyright Act be extended to include microforms, audio-visual materials of all kinds and electronic publications, both networked and artefactual (eg. CD-ROM), and all formats yet to be developed, in addition to the print-based publications currently included.2

This approach is supported by the International Conference on National Bibliographic Services held in Copenhagen in November 1998 which reaffirmed the value of legal deposit and recommended that ‘states should, as a matter or urgency, examine existing deposit legislation and consider its provisions in relation to present and future requirements: and, where necessary, existing legislation should be revised…’3

A key issue for legal deposit legislation and the libraries that administer it is the definition of ‘publication’, a concept that has become blurred in the online environment. Are all Web sites necessarily publications? Which documents on a government or other organisational web site are publications (to be collected and preserved by libraries) and which are organisational records (to be collected and preserved by archives)?

While it will be advantageous to have legal deposit provisions for online publications, it will not be a panacea for all our collecting problems. The Library wants the right to archive any publication it deems to be of national significance, but it does not want to be obliged to archive all Australian online publications and web sites and wants the right to remain selective. It is likely that the Library will find itself called to account for its selection decisions more than has been the case in the print environment and we will have to be able to justify our decisions. A clear collection development policy will be more important than ever.

Legal deposit legislation will not remove the necessity for liaising with many publishers to ensure that publications are complete and accessible. We will continue to need their assistance with archiving certain types of files that are difficult for us to download. In the case of commercial publishers, we will need to negotiate access to publications that are secured in some way, for instance, by password, and we will still need to negotiate periods of restriction so as not to jeopardise their commercial interests.

Origins of the National Collection of Australian Online Publications

At the National Library of Australia, we accepted in the early years of the web that online publications are important social, intellectual and cultural resources and that, with or without legal deposit legislation, our statutory responsibilities in relation to this material were no less than for the material which we traditionally collected.4

Ensuring long-term access can be seen as a two-step process. First the materials have to be identified, collected and made accessible in their current format (the archiving process). Second the materials have to be managed in such a way that they remain accessible as technology changes (the preservation process.)

The National Library has always recognised the importance of both these steps and their interdependence. For practical reasons the first step received most attention when we set up the PANDORA Archive in 1996. The preservation process has, however, been the growing focus of the Library’s efforts as the archive has moved beyond the ‘proof of concept’ stage to the operational National Collection.

The name PANDORA derived from its goal of “Preserving and Accessing Networked DOcumentary Resources of Australia”. For a number of years now the work has been a mainstream, operational activity and we refer to it as the National Collection of Australian Online Publications, although the name, PANDORA5 Archive, is still used interchangeably for it.

Description of the National Collection of Australian Online Publications

The National Collection of Australian Online Publications is a highly selective one containing to date only 2000 web sites. Nevertheless, it already constitutes a strongly representative sample of Australian web publishing by academic, government and commercial publishers, and community organisations. A number of the web sites captured in the archive, including the official web site for the Sydney Olympic Games, have already disappeared from the live Internet. Moreover, about one-third of the sites have been captured on multiple occasions, allowing the gathering of successive issues of serials, and enabling the collection of a sequence of snapshots, which demonstrate how some sites have changed over time.

The Collection now comprises 10,600,000 files using 320 gigabytes of storage; and is growing at about 500 new titles and about 400 re-gathers each year.

The gathering of titles is undertaken by the National Library as well as its partners: the State Libraries of Victoria, South Australia, and New South Wales, the Library and Information Services of Western Australia, the Northern Territory Library and Information Service, and Screen Sound Australia, (the national film and sound archive). In addition, the National Library cooperates closely with the State Library of Tasmania and its independent web archiving venture, Our Digital Island6.

A primary goal is to provide immediate as well as long term access to readers both onsite within the partner agencies and anywhere in Australia, and to this end permission of the publisher is sought and received prior to any site being included in the Collection.

All partners gather web sites and publications for the National Collection using the Digital Archiving System, which was developed by the National Library, and which I will describe in more detail later in this paper. All material, including that gathered by other partners, is currently stored on the National Library’s server and its associated storage systems.

The gathered version of every publication is subjected to quality checking to ensure that all files have been correctly captured.

Every web site/publication is fully catalogued, and the catalogue entries are included in the Library’s own catalogue, as well as the National Bibliographic Database. This policy is based on the Library’s firm view that access to all information resources (digital and traditional) should be integrated. (This approach has been endorsed by the International Federation of Library Associations (IFLA), which supports the treatment of online publications as part of the national imprint, incorporated into the National Bibliography.7)

As an alternative access pathway, titles in the National Collection are accessible via alphabetical and subject lists on the PANDORA home page.

Within the National Library, the work of building the Collection is now a routine part of the selection, acquisition and collection management processes of the Library. Organisationally, it is managed by the Electronic Unit, which forms part of the Collections Management Division of the Library.

Selection Issues for Online Publications

The Library and its partners do not attempt to capture everything published online that relates to Australia. A policy of selective acquisition has been adopted, partly dictated by resource issues but also by the determination that everything in the Collection should be accessible.

To govern the selection of online publications and web sites for which it takes responsibility, the Library has developed a set of guidelines8. These guidelines take account of characteristics such as Australian content and Australian authorship, whether the publication is indexed by a recognised indexing service, and stress the authority of the author, the research value and the quality of the publication.

Up until now the selection guidelines have excluded publications with print equivalents since, in the early years of our work, we needed to limit its scope to manageable proportions. However, we are constantly reviewing the selection guidelines and aim to be as inclusive as limited resources permit.

The policy of excluding those publications with print equivalents has already been relaxed to a certain degree when two years ago we invited indexing and abstracting agencies to enter into partnership with us. These services were already finding that some URLs cited in their references to electronic publications were defunct. The Library now has arrangements with six services, whereby they notify us of publications that they are referring to and we archive them, even if they have print equivalents.

The National Library of Australia has deliberately pursued the selective approach to archiving for its advantages of quality control and permission from publishers to archive and provide access. However, we have remained interested in the work of other agencies such as the Internet Archive and the Royal Library of Sweden, which are aiming to collect and preserve entire domains. We recognise that there is value in a comprehensive approach and have had initial discussions with the Internet Archive to ascertain whether cooperation with it would enable us to extend collecting of the Australian domain.

The Digital Archiving System

To facilitate the building of the National Collection a Digital Archiving System has been developed by the Library with the latest version implemented in June 2001.

Specifications for the system were devised by the Library’s Electronic Unit, and the system was built by contract staff using WebObjects9 as an Application Development Environment. The system is designed to support the following functions:

  • manage the metadata about titles that have been selected for inclusion in the Archive or rejected, for instance, title, URL, publisher details, gathering schedule, whether access conditions apply;
  • initiate gathers of titles to be archived;
  • manage the quality checking and problem fixing process;
  • prepare the item for public display and generate a title entry page;
  • manage access restrictions; and
  • provide management reports.

This digital archiving system is a web-based tool that uses Internet Explorer 5.5 to enable the partners to achieve a more efficient workflow without the need for special desktop software. While currently all titles, including those archived by partners, are stored and delivered from the National Library’s server, the system will support distributed content as an option for the future.

Persistent Identifiers

The publications stored in National Collection are to be preserved for posterity and they may be cited in many places including other web documents, in scholarly articles, in library catalogues and databases, in indexes and abstracts, and in bookmark files within browsers. They therefore need to be accessible indefinitely and require persistent identification.

In 2000, the Library engaged a consultant to provide advice on the direction it should take in relation to persistent identification of digital objects. The report10 produced by the consultant has been valuable in assisting the Library to determine its policy on naming standards and its procedures for managing digital information resources for persistent access.

In the absence of a global, operational system of persistent identifiers meeting the National Library’s needs, the Library has implemented the guidelines for persistent identification developed by the consultant and reproduced in the appendix to this paper. The resulting persistent identifiers are automatically assigned to titles and component files in the Collection. They are not dependent on a domain name to provide unique identification of the resource but could be used as the namespace specific string in the context of any naming system implemented at a later date.

This standard aims to ensure that each file will have a persistent unique identifier, that there will be sufficient granularity of identification to support preservation activities, and that there will be sufficient “intelligence” to enable a measure of grouping and relating of versions.

Since June 2001, the Digital Archiving Software has enabled the persistent identifier to be cited on the title entry page for every title in the PANDORA Archive. While this has been an important step forward in the promulgation of the use of persistent identifiers, it does not assist those researchers or indexing and abstracting agencies that need to refer to part of a publication, for instance, an article in an e journal. The new version of the Digital Archiving Software coming out in March will provide the persistent identifier for every file in the Archive.

Access Control

Access restrictions are applied to a number of titles in the Collection and are managed by the Digital Archiving System. Restrictions may be applied for commercial reasons, for privacy or cultural reasons, or as part of a policy decision for certain categories of material. Items can be restricted for use only within the buildings of partner agencies for a specified period of time, or can be password protected so that only designated researchers can obtain access.

To date there is little commercial publishing on the web in Australia with most of the publications in the Collection being freely available. However, there are a growing number of commercial publications in the Collection.

When a commercial title is selected, the Library negotiates with the publisher to determine access conditions that will not undermine the publisher’s commercial interest. The preferred model is for the publisher to allow use within reading rooms of the partner agencies for an agreed period of time, after which the title would become freely available to external users. Periods of restriction negotiated to date range from three months to five years. Agreements of this type have now been reached with publishers of about 80 commercial ebooks, ejournals and other publications.

The Digital Archiving System manages all of these restrictions and allows only those users in designated locations (based on IP address) or with the required password to gain access to archived versions. As time periods for restrictions expire, the system automatically updates the title entry pages to indicate the changed access conditions.

Preservation

The ultimate purpose of the National Collection is to ensure that Australians of the future will be able to have access to significant Australian information resources currently being published on the web

As mentioned above, a first step in managing an archive of online publications involves creating and managing a safe place where digital resources can be stored. The second and vital step is to ensure that the resources remain viable and accessible in the long term.

Given the diverse collections and formats for which it is responsible, the Library will use a combination of preservation methods for the foreseeable future, including:

  • some technology preservation, including maintenance of software and even some hardware;
  • negotiating with publishers to supply stable source files of some streaming or dynamic formats;
  • migration strategies for those file formats which correspond to compatible new formats, and which are amenable to mass conversion;
  • use of emulators, if they can be found or developed, for some file formats; and
  • simply keeping and refreshing some files, not amenable to migration or emulation, in the hope that a suitable access pathway will emerge.

The Library stores and manages at least two copies of archived files, maintaining the original copy unchanged in its native formats as the preservation copy and creating a second copy, which may have to be manipulated in some way, for access. When preservation action is required, this will take place on yet another copy, which will be created for the purpose. This procedure will safeguard the Collection in the situation where, for example, a series of migrations leads to a dead end. We will always have the original files to go back to.

In 2001 a small trial migration of 127 files was successfully undertaken. In preparation for this, the Library had identified file formats held in the Collection, with a view to establishing an effective preservation path for each format. It had also identified dead and deprecated HTML tags in the Collection, which may not be recognised by later browsers.

This analysis found:

  • 127 tags dead in HTML 4.0;
  • 7 million tags due to be made non-standard in later versions of HTML; and
  • 14 million tags with deprecated attributes.

Clearly, our future migration tasks will be on a much greater scale than the first small trial!

A working group of staff involved in digital archiving is currently engaged in the task of identifying the ‘significant properties’ of various types of digital objects, including those stored in the PANDORA Archive. This group is attempting to answer the question in relation to a given digital object (or class of objects) ‘What exactly is it that we want to preserve?’

The Cedars Project Team in the United Kingdom has also grappled with this matter, and defines ‘significant properties’ as ‘the level of content and functionality retained’ by the archiving and preservation process11. The Cedars Project report notes that the ‘…Significant Properties of a digital object as they have been agreed…may alter the costs associated with preserving those specific properties….The preservation of an object’s full functionality may prove more costly than just a bare-bones preservation of basic intellectual content. The question that has to be asked is whether the object’s long-term value is worth the long-term expense of preserving the ‘bells and whistles’.

Defining ‘significant properties’ of different kinds of digital objects in the Collection will help the Library to

  • understand its preservation task better;
  • set up quality control criteria for judging whether preservation action has been successful;
  • allocate resources to preservation work that is really needed; and
  • determine the level of metadata required to support the management of the digital objects.

Ultimately, decisions about significant properties will be implemented automatically to whole classes of material, in order to manage the Collection cost effectively. Defining the significant properties of classes of materials will be an important step forward.

Preservation Metadata

Significant properties are closely related to the next matter – preservation metadata – as determining the significant properties will dictate the amount of metadata that must be recorded and stored. A successful preservation process relies on an accurate record of the types of files to be preserved, the nature of any preservation activity carried out on them, and the result of such activity.

What information are we likely to need? In addressing this question, the Library in 1999 developed and published a draft set of digital preservation metadata requirements12. In 2000, the Research Libraries Group (RLG) and the Online Computer Library Center (OCLC) invited the Library and a number of others to join an international working group charged with the task of proposing a draft international standard for preservation metadata.

The working group has taken as its starting point the approaches to preservation metadata of the NEDLIB Project in Europe, the Cedars Project in the UK, and Harvard University Library's Data Repository Service, as well as the National Library of Australia’s draft. A White Paper13 issued in January 2001 compared these approaches as a springboard for defining a comprehensive, structured preservation metadata framework, applicable to a broad range of digital object types, archival processes and institutions. The working group’s recommendations for a preservation metadata standard are expected in early 2002.

Cooperation with Publishers

In the online environment, legal deposit legislation will not succeed without cooperation between libraries and publishers. Accordingly, the National Library has been working with the Australian Publishers Association to develop a Code of Practice for archiving, preserving and providing access to commercial publications. It has been important to establish mutual trust and overcome the suspicion that has existed between publishers and libraries in relation to access to electronic publications. Publishers feared that their control over their intellectual property would be lost and their livelihoods undermined. Libraries need to demonstrate to publishers that they can control access in agreed ways and that there are advantages to publishers in legal deposit through wider knowledge of their works and also long-term care and maintenance of them.

The Australian Publishers Association and the National Library have recently reached agreement on the Code and will now ask a small number of publishers to trial it. The Code recognises that safeguarding Australia’s published cultural heritage is a concern shared by publishers and the National Library. It outlines the conditions and responsibilities that each partner agrees to observe in order to ensure Australian online publications remain available for use into the future.

Lessons Learned

In five years of digital archiving the National Library has learned a lot about the technical, legal and organisational aspects of the work and the kind of infrastructure needed to support it. We have reported on what we have learned in conference papers, journal articles, and documents on our web site. There are some other aspects that are not commonly discussed and I would like to share a few of these with you.

We have found that in the digital environment it is advantageous to liaise and cooperate with a wider range of parties than was necessary for print collection development. Active working relationships are required with publishers and indexing and abstracting agencies. Collaborative relationships with other libraries and collecting institutions, both in Australia and overseas, are beneficial for exchange of information and to share the high costs.

Taking a practical approach and ‘learning by doing’ worked well for us in the beginning and continues to do so as we investigate new aspects. Once we start work on a particular task, instead of just considering it theoretically, we sort out many issues as we go along and gain confidence in our ability to tackle the task at hand, no matter how hard. There are still sometimes problems that we can’t solve immediately, but at least we know what they are.

A team-based approach to devising and implementing policy and procedures enables us to draw on the expertise of a wide range of staff within the Library and helps to motivate their commitment to the work.

Starting small also worked well for us. This approach kept the workload manageable while we built up efficiency and expertise. It also enabled us to address issues like quality control in detail and gave us time to learn.

Deciding not to exclude a web site or publication on the basis of its file formats meant that we learn a lot about web publications and what can be done with them, even the very complex ones. That does not mean we always succeed in gathering them, for example, we have not yet managed to archive databases.

Staff undertaking this work need certain qualities – willingness to learn, adaptability, persistence, initiative, ability to tolerate frustration and to think creatively.

Issues the National Library of Australia has still to resolve

Digital archiving is an area of work where the learning and development never stop. There are still a number of issues that the National Library has to solve and, given the nature of the digital environment, as soon as we find solutions for these, others will emerge.

As already mentioned, the National Library still lacks legal deposit for electronic publications and has to resolve many of the issues that go with that.

The Digital Archiving System does not yet deal adequately with a number of significant matters:

  • gathering and storing the software and plug-ins that publications and web sites are dependent on for display;
  • clearly indicating to users that they are viewing an archived object rather than a current version on the publisher’s site; and
  • other authenticity issues.

Although we know in theory what preservation metadata we want to record, at this stage we do not have a mechanism for recording all of it. (The Digital Archiving System keeps some.) This deficiency is being addressed through the development of a Digital Object Management System14, 15 which will contain, among other things, the data to support migration and other preservation activities aimed at providing long-term access to digital objects. The Library has developed a specification for its Digital Object Management System, and is currently developing part of this system, known as a ‘Special Collections Manager’ in-house, after failing to identify a sufficiently affordable and low risk solution in the market place. Completion of the Special Collections Manager, which will provide for preservation metadata, is planned for late 2002.

Another issue still requiring resolution relates to the archiving of databases. I noted earlier our plans to address this problem this year.

In adopting the selective approach to archiving, we are archiving only a very small proportion of the Australian domain. We would like to supplement the selective Collection with periodic snapshots of the entire Australian domain. A short-term consultancy during 2001 identified the associated issues for us.

While we have developed a system of persistent identification for the Library’s digital objects, we see a need for a national system. A proposal for an Australian Digital Resource Identifier has been endorsed in principal by the Council of Australian State Libraries, but this initiative requires further practical development and the implementation of a national agency for allocation of identifiers.

RLG/OCLC are in the process of defining the attributes of a trusted digital repository with their report16 due out early this year. The Library will assess the PANDORA Archive against the attributes listed in this document and, where any deficiency exists in our system, work towards meeting the standard.

Like every other digital archive in the world, we still have a host of preservation issues to resolve. We will keep abreast of research, contribute a little of our own, and work in cooperation with others to find solutions.

Conclusion

As a national deposit library, the National Library of Australia has clear responsibilities for collecting and preserving the documentary heritage of Australia now appearing in digital formats. After five years experience with archiving web publications, building the National Collection of Australian Online Publications has become a routine part of the Library’s collection management activities.

While there area significant challenges ahead, we believe we are establishing a good foundation for meeting them.

Appendix:

PANDORA Persistent Identifier Standard

A recent consultancy recommended the following form of identifier for files stored in PANDORA:

<collection id>-<work identifier>-<archive date>-<publisher’s URI>-<generation code>

where

<collection id>: for the archival collections the collection ID will be “nla.arc”

<work identifier>: a unique number within the digital archival collections assigned to the parent work of which the resource is a component

<archive date> the date the file was archived in the format YYYYMMDD

<publisher’s URI> currently the host name, path name and file name of the resource on the publisher’s site

<generation code>: a two digit code representing the version of a resource which has been migrated from its original format.

1 The author has drawn on the work of colleagues at the National Library of Australia, particularly the following paper:

Cathro, W., Webb, C., & Whiting, J. (2001) Archiving the web: The PANDORA Archive at the National Library of Australia. Paper presented to Preserving the Present for the Future - Strategies for the Internet, held at The Royal Library, Copenhagen 18th -19th of June 2001 Available online http://www.deflink.dk/docalle/692_cathro.doc

2 National Film and Sound Archive, & National Library of Australia. (1995) Submission to the Copyright Law Review Committee on legal deposit. Available online http://www.nla.gov.au/policy/clrcld.html paragraph 6.3.1

3 International Federation of Library Associations [IFLA]. (1998) The final recommendations of the International Conference on National Bibliographic Services. Available online http://www.ifla.org/VI/3/icnbs/fina.htm

4 Fullerton, J. (2000) The information environment in the 21st century: societal issues. Paper presented to the Australian Library and Information Association Biennial Conference, Canberra, 2000.

5 National Library of Australia. PANDORA: preserving and accessing documentary resources of Australia. Available online http://pandora.nla.gov.au/

6 State Library of Tasmania. Our Digital Island Available online http://www.statelibrary.tas.gov.au/odi/

7 International Federation of Library Associations [IFLA]. (1998) The Final Recommendations of the International Conference on National Bibliographic Services. Available online http://www.ifla.org/VI/3/icnbs/fina.htm

8 National Library of Australia. PANDORA Selection Guidelines. Available online http://pandora.nla.gov.au/selectionguidelines.html

9 WebObjects5: the incredibly powerful way to create Java Server applications. Available online http://www.apple.com/webobjects/

10 Dack, D. (2001) Persistent identification systems: Report on a consultancy for the National Library of Australia. Available online http://www.nla.gov.au/initiatives/persistence/PIcontents.html

11 Cedars Project Team. (2001). The Cedars Project report. Page 14. Available online http://www.leeds.ac.uk/cedars/OurPublications/CedarsProjectReportToMar01.pdf

12 National Library of Australia. (1999) Preservation metadata for digital collections: Exposure draft. Available online http://www.nla.gov.au/preserve/pmeta.html

13 OCLC/RLG [Online Computer Library Center/Research Libraries Group] Working Group on Preservation Metadata. (2001) Preservation metadata for digital objects: a review of the state of the art. Available online http://www.oclc.org/digitalpreservation/presmeta_wp.pdf

14 National Library of Australia. (2000). Digital Object Management System request for quotation. Online Available at http://www.nla.gov.au/dsp/doms/

15 The Digital Object Management System forms part of the Library’s digital collection management framework which is set out in:
National Library of Australia. (1998) Digital Services Project information paper. Available online http://www.nla.gov.au/dsp/infopaper.html

16 RLG/OCLC [Research Libraries Group/Online Computer Library Center]. (2001) Attributes of a trusted digital repository: Meeting the needs of research resources: Draft for public comment. Mountain View, CA: Research Libraries Group. Available online http://www.rlg.org/longterm/attributes01.pdf

Back to the top