Staff Papers

Digital Preservation: The Australian Experience

Presented by Deborah Woodyard, Digital Preservation, at the 3rd Digital Library Conference: Positioning the Fountain of Knowledge, State Library of Sarawak, Malaysia, 2nd - 4th of October, 2000.


CONTENTS


Abstract:

Digital preservation is increasingly recognised as a vital part of managing digital resources. Australian institutions, particularly libraries and archives, are making good progress in this challenging new area, including a number of projects at the National Library of Australia. The new Digital Services Project, including the PANDORA archive, and the PADI web site are prominent features. There is a strong need for national cooperation, international collaboration and knowledge sharing for further development.


Introduction

Australia has been seriously examining digital preservation issues since at least 1994 (1). Early work in the area produced papers discussing the issues (2), a survey of the preservation intentions in a variety of institutions and a statement of principles (3) for ensuring continuing access to all categories of significant digital objects. Now work in the field has advanced considerably producing several projects of international acclaim.

A majority of the progress in the preservation of digital resources in Australia has been made in the Library and Archive sectors. Both fields already have solid backgrounds in the long-term preservation of traditional documentary materials, which provides a strong foundation for continuing this work with digital resources.

At the National Library of Australia, we believed we needed to get involved with digital preservation because we recognised that:

This paper looks at digital preservation in Australia by focusing on the projects that are being conducted in various areas. These projects have been categorised under the following headings:

What we are preserving -

How we intend to preserve it -

What we are preserving

Digital materials of long-term significance come in all different shapes and sizes ranging from plain text documents to interactive Internet publications, and they are archived by many different institutions from Universities to Government. The purpose of each collection will dictate different archiving and access strategies, and therefore the projects that have been established also vary considerably.

This paper will focus on three significant categories of material, electronic records, digital audio material and online publications, which are being archived in major Australian projects. Additional projects are aiming to preserve other digital materials, such as theses and cartographic materials.

Electronic record-keeping

Records are generally managed by archives and are often in large collections of similar formats, such as in a database. Maintaining their authenticity is an important feature of their long-term value. These characteristics strongly influence the type of preservation path that has been chosen in the following projects.

There are a number of initiatives to include electronic record-keeping in the development of standards and guidelines for best practice in the archival community in Australia.

The most detailed and advanced initiative is the Victorian Electronic Records Strategy (VERS) (4), which was initiated by the Public Record Office Victoria (PROV). The Strategy will assist agencies with developing electronic records management, archiving systems and policies, yet fit with their existing business processes and records management structures. It will also enable the PROV to satisfy its own requirements to manage state government records and preserve the state's heritage.

The VERS project produced a standard, PROS 99/007 Standard for the Management of Electronic Records (5), which recommends a standard record format incorporating the following features:

Work on this project included a prototype of a “future state” system developed using this strategy to test methods for electronic record creation, management and archiving. This enabled the PROV to produce system designs and costs analysis to use as a basis for advice to government and industry on essential archiving requirements.

A number of other initiatives have also produced guidelines that effect the long-term management of electronic records:

Audio archiving

Audio archives have been amongst the earliest of the collecting institutions to embrace digital formats as collecting and preservation media. Magnetic tape recordings in analogue form have been collected and stored in Australia for over 50 years, however, format obsolescence and media degradation affecting analogue tape have forced the evaluation of digital carriers as a preservation strategy. Australia has a very active sound archiving community and most sound collecting institutions have embraced digital in one form or another for some years.

The Australian Broadcasting Corporation’s radio network instituted a computer-based digital on-air system that had major implications for its archives. They have now dispensed with all analogue copies of new material and archive on recordable compact disc (CD-R).

The National Library of Australia was the first to adopt CD-R as an archival carrier and believes that the technical limitations of CD-R can be managed through a comprehensive testing and analysis program, and careful technical management of equipment and facilities. We now have a database of CD-R data built up over five years.

The Australian Institute of Aboriginal and Torres Strait Islanders have been archiving their audio collection on CD-R in data format, and Screen Sound Australia, the National Film and Sound Archives, have been using CD-R and DAT (digital audio tape).

Sound files take up significant storage space, and large collections of audio material require large numbers of CD-Rs. Consequently, when storage media exist in large numbers such as this, matters of media stability are highly significant. The NLA and SSA, have instituted many evaluation and life testing trials on CD-R and DAT tapes. Results show CD-R is potentially a useful and functional part of the audio archive, providing that tight tolerances are applied and comprehensive error measuring is undertaken. The use of these storage media in archives is about technical management, and the National Library of Australia’s experience in the use of CD-Rs over the past five years has shown that with due attention to detail, such an approach can be successful (9).

However, the ever-expanding storage capabilities of mass storage systems now make them viable for the storage and preservation of large amounts of audio data. Many Australian collecting institutions are establishing standards and best practices covering formats, technical specifications and metadata, establishing parallels with preservation of other digital materials.

Archiving online publications

The Internet is now undoubtedly a rich source of information and publications documenting our current place in history. Therefore it has become a strong focus for many digital preservation efforts around the world.

At the National Library of Australia we have been developing the PANDORA archive, a national collection of electronic publications built on shared responsibility for archiving in conjunction with the State libraries. The configuration of this project is still evolving, and while some State libraries (Victoria and South Australia) have decided to participate in a common project architecture, other State libraries (in particular Tasmania (10)) are pursuing their own systems.

The primary objective of PANDORA is to capture, archive and provide long term access to significant online publications selected for national preservation. To accomplish this we need to address both archiving and preservation processes (12). Archiving processes refer to the collection and provision of immediate access to the publications; preservation processes involve managing the materials and applying appropriate strategies (e.g. migration etc) to ensure long-term access can be provided.

The National Library has more thoroughly developed the archiving processes in its work so far, which include:

These procedures can be very time consuming and are often challenging. Difficulties occur at many stages such as negotiating and obtaining permission from publishers, collecting certain types of files (e.g. databases, Java-based), attempting to collect disappearing or re-located titles (14).

Five staff are currently working full-time archiving titles on PANDORA. There are also a number of staff who provide support from the IT and Preservation areas. There are now approximately 700 titles archived and roughly a third of these are regularly collected at set intervals.

Techniques for managing long-term access to these digital resources is still dveeloping, and it cannot be accurately predicted yet exactly what the ideal processes will entail. They will surely require the use of standards and appropriate infrastructure, such as the application of persistent identifiers (15) and preservation metadata, in addition to familiar methods such as emergency back-ups, and strategies such as migration and emulation.

To manage these demands the National Library is now developing the Digital Services Project (DSP) (16). The DSP is a technical infrastructure that will support all current requirements of PANDORA as well as provide an integrated collection management environment for:

We have discovered through a ‘Request for tender’ process that no ready-made, off-the-shelf system exists to cover all of our requirements, so the project has been divided into three smaller development and procurement projects with a view to their integration. These projects are:

The National Library aims to complete this project in 2001.

How we intend to preserve it

With such a variety of materials it is apparent that no single preservation solution will be appropriate for all digital resources, but various combinations of solutions will need to be employed. Some strategies that look promising and are receiving a lot of attention and development are metadata for long term preservation, and migration and emulation strategies. It is very important that we not only further research on these topics, but we must also share information with like-minded projects throughout the country and internationally.

Metadata for long-term access

Metadata is an important and high profile tool for assisting long-term access to digital material. There are a number of projects in Australia developing the use of metadata specifically with this intention.

Electronic records projects in particular see metadata as the answer to long term preservation. For example, the VERS project mentioned earlier specifies the importance of creating metadata to provide long term access. They recommend the creation of a metadata record in plain text and then linking this to the electronic record in the archive to ensure software and format information is available to interpret the record in the future (17).

The National Archives of Australia also encourages the adoption of their Recordkeeping Metadata Standard for Commonwealth Agencies (18) as a common descriptive standard across government. They believe it will help to “fulfil a range of records management responsibilities and aid planning for data migration and other preservation needs by identifying, in standardised and accessible ways, the software and hardware dependencies of records”.

The National Library of Australia has also been developing a set of metadata specifically for preservation (19). It aims to be applicable to any type of simple or complex digital object and recommends elements to describe file formats and structures, system requirements, responsibility information, etc. An exceptionally complex task, this has not been directly implemented as a full set and will now feed into further international work developing preservation metadata with American and European groups led by the Research Libraries Group (RLG) and the Online Computer Library Center (OCLC) (20).

Researching digital preservation strategies

Strategies for digital preservation such as transfer, migration and emulation are at the core of management for long-term access. Storage and metadata systems essentially aim to make emulation and migration possible. Research into these techniques is progressing but still in early stages.

In Australia, the National Archives asserts they are undertaking a project during 2000 to develop advice for Commonwealth agencies on using migration as an electronic records preservation strategy (21).

The National Library is also conducting several trials of various types of transfer, migration and emulation, each are described below.

A few years ago digital content on physical formats such as floppy disk was identified as being at high risk of loss due to media failure. This prompted a trial of media transfer (22), and now all Australian floppy disks in the collection are routinely transferred to a more stable carrier, currently CD-R. This involves selection of high priority materials, transfer, quality checking, checking cataloguing details, documenting access procedures and packaging the disk to redirect use to the new surrogate.

Migration trials are focusing on text and HTML files in the National Library’s Manuscripts and PANDORA collections respectively.

A number of digital text files have entered our Manuscripts collections on floppy disks of various sorts, containing a wide variety of file formats. The National Library has purchased software that can interpret a very large number of obsolete word processing formats and then aid the migration of these text files to a current and hopefully more robust file format. As long as you have a disk drive that can accept and read the data off the disk, this software has a very good chance of being able to interpret your text into something comprehensible. Formal procedures for these translations are still being developed.

We have also embarked on migration experiments with some HTML pages in the PANDORA archive. The format of these pages is not yet obsolete, but the HTML 4.0 specification has declared a number of mark-up tags as "dead", or not to be supported in this or future versions. The aim of these trials is to make changes in the HTML source code to remove tags declared as dead and replace them with current tags, effectively migrating the source code to a different version to reduce problems of future compatibility with web browsers. (Of course the HTML source code of most pages in the archive do not strictly conform to any one version of HTML, such is the nature of the language and tolerance of the Internet.)

We consider this a useful migration exercise to test the effect of making changes in the code; though small, it will test the effects of making changes on a scale we can easily monitor closely. At the same time it has potential to show up many difficulties of batch processing changes in an archive.

For long-term access to more complex digital resources, such as those that contain executable files, we can see that migration may have limitations and emulation could be a more suitable solution.

Emulation trials reflecting the work done by Jeff Rothenberg (23) in Europe are also currently underway, looking at emulation of early DOS operating systems in order to run a number of early Australian digital publications. The aims are to evaluate the use of software for emulation, examine the results and viability of emulation as a reliable technique for providing long term access, and determine the requirements for managing material with the aim of enabling emulation in the future.

Sharing information

To develop effective solutions we must work together and combine our knowledge and experiences, both nationally and internationally.

In PANDORA for example, we value the opportunity to develop a national model where the State and National Libraries are aiming for a common goal, a national collection of electronic publications. We also value the chance to work with a diversity of approaches, such as the different system being developed to achieve this goal by the State Library of Tasmania, the archiving of electronic theses being pursued by university libraries through the Australian Digital Theses Program (24) and other international projects (25).

The National Library also provides a forum for the exchange of digital preservation information via its internationally acclaimed PADI (Preserving Access to Digital Information) web site (26).

The PADI initiative aims to help ensure that information in digital form is managed with appropriate consideration for preservation and future access.

Its objectives are to:

The PADI web site is a subject gateway to digital preservation resources, and it has an associated discussion list, padiforum-l, for the exchange of news and ideas about digital preservation issues.

Information is arranged on the web site in two streams: types of resource and subject headings. Ths site also includes standard features for searching and browsing. There are roughly 850 resources categorised by 35 subject headings such as national strategies, digital libraries, formats and media, metadata, and digital preservation strategies.

We encourage anyone to recommend resources to add to the web site and welcome feedback on how to make this a more useful tool for everyone to use.

Conclusion

In Australia, and particularly at the National Library, we have contributed a very practical approach to the development of digital preservation in the world. We have analysed the situation, recognised problems and made a good start on implementing some solutions. We have not waited until they are all thoroughly tested and proven because we believe there is limited time to find solutions due to the rapid rate of technological change.

One thing that characterises the digital preservation experience in every country, not only in Australia, is that it requires an enormous cooperative effort and information sharing. In Australia we are developing our own projects and policies, involving original work but feeding into and from a global information base, without which this would all be close to impossible.


Acknowledgments:

I would like to thank Kevin Bradley and Colin Webb for their input and assistance with preparing this paper.


References:

(1) Webb, C. and Jones, M. (1998) Electronic Publications and the Survival of Information (http://www.nla.gov.au/nla/staffpaper/mjcwvala.html) Paper presented at VALA '98 Conference (Melbourne)

(2) (1996) Key Issues in Australian Electronic Publishing: Collected Reports (http://www.adfa.oz.au/EPub/key/Opening.html) Electronic Publishing Working Group, Australian Vice-Chancellor's Committee

(3) (1996) Statement of Principles for the Preservation of and Long-term Access to Australian Digital Objects (http://www.nla.gov.au/preserve/digital/princ.html) National Library of Australia

(4) (1999) Victorian Electronic Records Strategy Final Report (http://www.prov.vic.gov.au/vers/final.htm) Public Records Office Victoria

(5) (1999) PROS 99/007 Standard for the Management of Electronic Records (http://www.prov.vic.gov.au/gservice/standard/pros9907.htm) Public Record Office Victoria

(6) (viewed Sep. 2000) Commonwealth Recordkeeping (http://www.naa.gov.au/recordkeeping/default.html) National Archives of Australia

(7) (1999) Keeping Electronic Records (http://www.naa.gov.au/recordkeeping/er/keeping_er/contents.html) National Archives of Australia

(8) (viewed Sep. 2000) Government Recordkeeping Manual (http://www.records.nsw.gov.au/publicsector/rk/manual.htm) State Records Authority of New South Wales

(9) Bradley, K. (2000) CD-R Archiving: Case study of an interim media. Paper presented at A Future for the Past: IASA-SEAPAVAA Conference 2000 (Singapore)

(10) (viewed Sep. 2000) Our Digital Island (http://www.tased.edu.au/tasonline/history/) State Library of Tasmania

(12) Phillips, M.E. (1999) Ensuring Long-Term Access to Online Publications (http://www.press.umich.edu/jep/04-04/phillips.html) Journal of Electronic Publishing, volume 4, June 1999

(13) (1996) Guidelines for the Selection of Online Australian Publications Intended for Preservation by the National Library of Australia (http://www.nla.gov.au/scoap/guidelines.html)

(14) Ash Relf, F. (1999) PANDORA - Towards a national collection of Australian electronic publications (http://www.nla.gov.au/nla/staffpaper/ashrelf1.html) Paper presented at 10th national Library Technicians conference (Fremantle)

(15) (2000) National Library Standards Activities: Item Identifiers (http://www.nla.gov.au/services/standards.html#ItemIDs) National Library of Australia

(16) (2000) Digital Services Project (http://www.nla.gov.au/dsp/) National Library of Australia

(17) See note 5 above

(18) (viewed Sep. 2000) Recordkeeping Metadata Standard for Commonwealth Agencies (http://www.naa.gov.au/recordkeeping/control/rkms/summary.htm) National Archives of Australia

(19) (1999) Preservation Metadata for Digital Collections (http://www.nla.gov.au/preserve/pmeta.html) National Library of Australia

(20) (viewed Sep 2000) RLG and OCLC Explore Digital Archiving (http://www.rlg.org/pr/pr2000-oclc.html) Research Libraries Group

(21) (viewed Sep. 2000) Electronic Records: Preservation and migration of electronic records (http://www.naa.gov.au/recordkeeping/er/summary.html#preservation) National Archives of Australia

(22) Woodyard, D. (1998) Farewell my Floppy: a strategy for migration of digital information (http://www.nla.gov.au/nla/staffpaper/valadw.html) presented at the VALA Conference (Melbourne)

(23) Rothenberg, J. (2000) An Experiment in Using Emulation to Preserve Digital Publications (http://www.kb.nl/nedlib/results/emulationpreservationreport.pdf) National Library of the Netherlands

(24) (viewed Sep. 2000) Australian Digital Theses Program (http://adt.caul.edu.au/) University of New South Wales.

(25) Berton, H. and Webb, C. (2000) The Moving Frontier: Archiving, Preservation and Tomorrow’s Digital Heritage (http://www.nla.gov.au/nla/staffpaper/hberthon2.html) Paper presented at VALA 2000 conference (Melbourne)

(26) (viewed Sep 2000) PADI (http://www.nla.gov.au/padi/)

Top

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us