Staff Papers
Nurturing Our Digital Memory: Digital Archiving and Preservation at the National Library of Australia
A submission for International Preservation News by Dr Hilary Berthon, Preservation Services, National Library of Australia, May 2002
As increasing use is made of digital technologies to create, record, store and transmit our cultural heritage, our global digital memory is born. However, strategies need to be adopted to ensure ongoing access to this memory. The National Library of Australia (NLA) believes that the key to developing successful models of digital preservation lies in cooperation: sharing information as well as sharing responsibility.1 Both the magnitude of the task and the keenness of libraries to define their role in the digital environment have encouraged Australian libraries to work closely together. A long history of collaboration between Australian librariesfor example, in the development of a national bibliographic database, resource sharing and preservationhas laid the foundations for this cooperation. As the challenges of preserving digital information exist across sectors, the NLA has also looked beyond the traditional library boundaries for new partners. A commitment by the NLA to learn by doing, drawing on the experiences of memory institutions and sharing with others the fruits of its experience, has nurtured this collaborative model.2
Archiving and Preservation
As key activities aimed at ensuring the long-term accessibility of our digital heritage, digital archivingthe process of identifying, collecting and making material accessible in its current formatand digital preservationmanaging this material to ensure that it remains accessible as technology changeshave gone hand-in-hand 3. While much early attention was directed towards the archiving end, digital preservation has increasingly been a focus of the NLAs activities.
The PANDORA Archive
PANDORAthe selected National Collection of Australian Online Publications established by the National Library of Australia 4is a collaborative effort incorporating a number of agencies. Broader coverage, avoiding duplication of effort and minimising cost are among the advantages of working together. Established in recognition that online publications are an intrinsic part of Australias national documentary heritage, partners include the State Libraries of Victoria, New South Wales and South Australia, the Library and Information Service of Western Australia, the Northern Territory Library and Information Service and ScreenSound Australia, the national film and sound archive. The NLA also cooperates closely with the State Library of Tasmania which operates its own web archiving project, Our Digital Island 5.
While not all partners currently participate in all aspects of archiving, the PANDORA archiving system, PANDAS, potentially enables collection managers in each of the partner agencies to gather web publications, to record information about publications and to manage access to them. The Collection now comprises over 2000 titles, (approximately 14,000,000 files) using almost 400 gigabytes of storage and is growing at about 500 new titles each year. 6 About one third of these titles have been gathered more than once to illustrate the changing nature of web sites and many are gathered on an ongoing basis to capture new issues of serial publications. Already, many of the captured sites, such as the Sydney 2000 Olympic games website are no longer accessible except through the archive.
Although the NLA maintains a keen interest in comprehensive whole domain harvesting, it intends to maintain its selective approach to archivingchoosing material that most strongly represents Australian social, intellectual and cultural life. In the future, this may possibly be supplemented by occasional snapshots of the broader Australian web publishing output. To assist with the selection of material, PANDORA has entered into partnership with six indexing and abstracting agencies that notify the NLA of publications that they are referring to. That the significance of online material will be determined differently by different stakeholders underlies the NLAs expectation that the national collection it is building with State Libraries and ScreenSound will ideally be a component of a wider national distributed archive with other stakeholders such as the academic sector playing a role. 7
Cooperation with publishers appears to be a natural way forward in developing web archiving solutions. For example, the assistance of publishers to overcome technical and imposed security impediments to archiving certain types of files will be a crucial aspect of ensuring that archived publications are complete and accessible. 8In recognition of their combined responsibility for ensuring ongoing access to electronic publications, the NLA and the Australian Publishers Association have developed a Code of Practice for Providing Long-Term Access to Australian Online Publications. The NLA anticipates that this Code, which outlines the conditions and responsibilities that each partner agrees to observe in order to ensure Australian online publications remain available for use into the future, will be trialled by a small number of publishers in the near future. Restricted access to some archived publications, for example through the reading rooms of partner institutions, ensures that publishers commercial interests are not undermined.
Digital Preservation
As previously mentioned, for the NLA, digital preservation includes the processes involved in maintainingor even recoveringaccess to digital resources. It involves many concepts similar to those for the preservation of traditional materialsthe identification of materials of significance, the identification of risks, damage prevention, stabilisation or repair, the creation of surrogates and condition monitoring. The development of strategies that accomplish these processes for digital materials is among the primary goals of the NLAs Digital Preservation section.
There are a range of digital resources for which the Library takes preservation responsibility. These include the Australian online resources archived within PANDORA; material coming into the Librarys collection on diskette or CD-ROM; digital manuscript, pictorial and audio material as well as digitized copies of analogue materials in the Librarys collection. The Librarys Digitisation projectoperational since August 2001will add significantly to the body of material which needs to be preserved. This project aims to produce nearly 40 000 digital copies of the collection materialphotographs, postcards, cartoon, drawings, transparencies, negatives, maps, sheet music and manuscript materialby the end of June 2002. 9 Some of the challenges in keeping these resources accessible include the pressure of technological change, the volume and diversity of digital resources and the cost and administrative implications of taking action over very long periods of time in response to recurrent threats.
To guide its efforts in this area, the Library has developed a Digital Preservation Policy. 10 There are a variety of elements involved in securing ongoing access to digital material. The NLA has stated its commitment to attend to three: maintaining the byte-stream which constitutes the digital object and the information needed to present it as a meaningful reproduction of the originally presented digital object; maintaining the means of providing access; and maintaining access through citable identifiers with long-term validity. 11 The Digital Preservation Policy provides guidance for addressing these challenges in its own collections.
Recent activities within the Digital Preservation section of the NLA include a trial migration of HTML documents from the PANDORA web archive to a later HTML standard. Another investigation is attempting to define frameworks for identifying significant properties of digital resourcesthose properties that must be retained by any potential digital preservation treatment. This definition will help the NLA to define and allocate resources to its preservation task, measure the success of any preservation treatments and determine the level of metadata required.
As stated in its Digital Preservation Policy, the NLA believes that its digital archiving and preservation objectives will be best achieved by developing and implementing practices that comply with an adequate, coherent and widely understood framework for reliable, accountable and manageable digital archives. 12 In support of this belief, it continues to provide input into the development of standards for digital repositories and preservation metadata through two RLG/OCLC Working Groups. 13
Digital Audio
Changing technologies have brought about immense changes in the way in which audio material is preserved. Until recently, the NLA preserved all its audio material on 10-inch analogue reel-to-reel tape and CD-R. With the delivery of the new HDA Quadriga digital audio workstations, the NLA is now preserving its audio material into a digital mass storage system with public access delivered via CD-R. The Quadriga system allows audio to be converted to BWF (broadcast wave file) format enabling metadata to be stored within the file. This metadata includes both information that will assist the resource discovery and the preservation management of the file.
The implications of this change are hard to underestimatewith 35 000 hours of audio material in the collection which is growing at a rate of 850 hours each year, considerable effort will be put into conversion of existing material and storage of new material into the Digital Objects Storage System (DOSS). Between November 2001 when the system was delivered and the end of March 2002, 380 hours of audio materialcorresponding to 300 Gbyteshave been stored on the DOSS. Speaking of the new Quadriga system, Kevin Bradley, Manager, Digital and Audio Preservation Resources commented the system provides a more reliable preservation process that integrates with our future digital access plans. These access plans include the option of producing derivative copies of audio material which may delivered via the web.
The NLA has already realised the benefits of a loose collaboration with other agencies such as ScreenSound Australia, the National Archives of Australia and the Australian War Memorial who are undertaking similar transitions in their audio preservation programs. A recent conference of the Australasian Sound Recordings Association about Quadriga workstations and archiving proved highly successful in enabling agencies to benefit from each others experience and motivating the Australian audio community to play a role in influencing the future design of sound preservation systems.
PADI
Providing an international gateway to information about digital preservation, the NLAs PADI website 14 has enjoyed the reputation of being one of the very few environments where one can find the latest information as well as digital preservation resources in this fast moving arena. 15 The PADI initiative aims to ensure that information in digital form is managed with appropriate consideration for preservation and future access. It does this by bringing together, describing and providing access to Internet resources to assist organisations and individuals to preserve access to their digital material.
A cooperative approach has shaped the development of PADI. In addition to providing access to information and resources, PADI is also able to offer registered users the facility to participate in building PADI by entering their own new resources directly into the database, making them available to all PADI users. 16 PADI has recently signed a Memorandum of Understanding with the (UK) Digital Preservation Coalition and is actively seeking further partnerships that will assist it to maintain its international standing as an accessible, comprehensive and trustworthy one-stop service in digital preservation.
Conclusion
In its approach to the task of preserving our digital memory, the NLA has recognised the value of sharing information and learning from others. Its participation in partnerships, commitment to communication, and its development of mechanisms, such as the PADI service, for comparing approaches and reviewing developments, are testament to its view that our digital memory will be best nourished by working together.
NB : special thanks to Gerard Clifton and Shelly Grant, National Library of Australia, who provided information for sections of this article.
Footnotes
1 Towards a national infrastructure for access to Australias documentary information resources in electronic formats: strategic developments at the National Library of Australia; a paper presented by David Toll, Deputy Director General, National Library of Australia at the VALA conference, Melbourne, February 2002; http://www.nla.gov.au/nla/staffpaper/2002/toll1.html
2 The National Library of Australias Digital Preservation Agenda, an Interview with Colin Webb; RLG DigiNews Vol. 5, No. 1; http://www.rlg.org/preserv/diginews/diginews5-1.html
3 Archiving the web: The national collection of Australian online publications; a paper presented by Margaret Phillips, Manager, Digital Archiving, National Library of Australia at the International Symposium on Web Archiving, National Diet Library, Tokyo, Japan, 30 January 2002; http://www.nla.gov.au/nla/staffpaper/2002/phillips1.html
4 PANDORA Archive: Preserving and Accessing Networked Documentary Resources of Australia; http://pandora.nla.gov.au/index.html
5 Our Digital Island; http://odi.statelibrary.tas.gov.au/
7 Digital Continuity: the role of the National Library of Australia; a paper presented by Pam Gatenby, Assistant Director General, Collections Management Division, National Library of Australia, at Digital Continuity: a Forum for Australian Universities Swinburne University of Technology, 19 November 2001; http://www.nla.gov.au/nla/staffpaper/2001/pgatenby4.html
9 National Library of Australia Digitisation Policy 2000 2004; http://www.nla.gov.au/policy/digitisation.html
10 A Digital Preservation Policy for the National Library of Australia; http://www.nla.gov.au/policy/digpres.html
11 A Digital Preservation Policy
12 A Digital Preservation Policy
13 RLG Long-Term Retention Initiative: Digital Archives Attributes Working Group: A Joint Effort with OCLC; http://www.rlg.org/longterm/attribswg.html OCLC/RLG Preservation Metadata Working Group; http://www.oclc.org/research/pmwg/
14 PADI: Preserving Access to Digital Information; http://www.nla.gov.au/padi
15 Response collected in PADI user survey, User Insite Pty Ltd, July, 2000
16 PadiUpdate; http://www.nla.gov.au/padiupdate/
