A paper presented by Colin Webb, Director, Preservation Services Branch, National Library of Australia, at the Preservation 2000 Conference, York, UK, December 2000
Recently I received an email from a friend with a new signature block: "If you intend to live forever, so far, so good," it said. The more I thought about it, the more appropriate it seemed to the subject of digital preservation. We hope that "so far, so good" for our digital collections is a little more promising than this one-liner really is, but we dont really know.
This paper discusses the National Library of Australias (NLA) perspective on managing a national digital collection. It presents a brief overview of what we are dealing with, who is involved, and NLAs approach over the past few years, especially in building an archive of online publications, and in developing a national model for archiving. Finally, the paper looks at our current perspective on maintaining long-term accessibility.
The National Library of Australia aims as part of its statutory obligations, to retain in perpetuity a copy of all material published in Australia In line with this objective, we are also committed to preserving and making accessible a national collection of Australian digital publications. We also have an interest in building and maintaining a collection of other Australian documentary resources in digital form.
It will help in understanding our experience and our approaches to also understand the context in which we operate. The library system in Australia has a history of excellent cooperation between a relatively small number of libraries with a significant collection-building and collection-preserving role. The National Library is expected to provide leadership but it has only a limited mandate to insist: mostly it is expected to identify needs, propose solutions, negotiate with others, and seek agreements that may or may not include collaborative action.
In the areas of collection building and maintenance, the "others" are the other deposit libraries, basically State libraries set up by each State and Territory government, and a small number of the older and larger university libraries. While there is a strong public library network, its main business is generally delivery of information, without a noticeable role in digital collection building.
There is a tradition of legal deposit, and it will be interesting to see how publishers and libraries cope with it in the digital environment. At present we operate with varied arrangements, some States having introduced legislation extending legal deposit to digital publications of various kinds, while the other States and the Commonwealth have not as yet, although that is likely to change within the next year or so.
These two factors - institutional relationships based on cooperation, and the absence of an explicit legislative basis for digital collection building, at least at the federal level - have produced a situation where organisational issues have been as important to us as technical ones.
There are other significant players as well, of course. Other cultural collecting institutions are building digital collections (largely made up of digital copies of their own non-digital collections), and the national film and sound archive, ScreenSound Australia, is seeking a deposit role with "published" digital audio and video material.
All Australian governments have been using electronic record keeping systems for a long time, and their respective archival bodies have been actively developing guidelines and approaches for managing electronic records for at least the past 15 years - somewhat longer than have our libraries. They have produced many worthwhile results, such as the National Archives' approach to Record Keeping Metadata1, and the Public Record Office of Victoria's VERS (Victorian Electronic Records Strategy) project and subsequent standards2, to name two notable examples. These and other agencies involved in archiving digital information are potentially important parts of a national digital collection, but poor coordination with the library sector makes it hard to fit them into this discussion.
If these are the main players, what is going into our national digital collection? So far as libraries are concerned, we have identified four groups of material. Although we presently manage these four categories through different workflows, we intend to bring their management together where it seems useful to do so as our infrastructure develops.
We manage digital publications that are a) available online as well as what I will call here b) physical format digital publications (PFDP) released to the market on diskette, CD-ROM or other carrier. We also have a business interest in a broad range of other digital information documenting Australia and its place in the world. This includes c) diskettes and tapes serendipitously appearing in transfers of paper-based collections of manuscripts, and d) unpublished but intentionally produced materials such as oral history recordings and original digital art, as well as the results of digitisation programs. (Although digital images on websites can be considered as publications, behind them, usually unpublished, there may be higher resolution "master" versions intended for long-term maintenance that are stored and managed separately.)
It is certainly true that for libraries (at least for NLA), online publishing is the main digital archiving game, and the following discussion will reflect that. However I believe it is important not to ignore the other forms of digital information we are likely to encounter, as they present their own preservation challenges.
From the early 1990s the National Library of Australia recognised that it needed to extend its collection building and collection maintenance roles to incorporate digital publications. The necessity of capturing and preserving publications in digital forms always appeared to be a more critical priority than showcasing our collections by digitising them.
From the beginning we have been determine to build a very close relationship between what we want to achieve, our conceptual framework for doing it, and our practical experience. Our approach has been developed through a series of modelling exercises, starting in 1995 as an attempt to define a very modest pilot project, leading to the PANDORA 3proof-of-concept project which was launched in 1996. With PANDORA the Library established a widely based group charged with defining selection guidelines, and dedicated resources to the management of online publications and PFDP.
From 1997 the Library invested significantly in modelling business principles and a logical data model4 for the growing archive. These and all other important planning documents for PANDORA have been made available for comment to a wide group of individuals and institutions involved in similar work internationally.
Through the following two years the Library recognised that it needed to ramp up its technical capability to secure and progress PANDORA, and refine its underlying understanding of how PANDORA fits into the Library's business directions and systems. In late 1998 the Library published its Digital Services Project (DSP) Information Paper5 which sets out requirements for a technical infrastructure to collect, store, provide access to, and manage the PANDORA Archive, as well as to support the management of other digital and analogue collections. Over the past two years these requirements and understandings have been refined through a series of public procurement processes aimed at delivering greatly improved collecting tools, metadata searching, archive management, and digital object storage systems.
The National Library of Australia has taken a keen interest in the Open Archival Information System (OAIS) Reference Model developed by the Consultative Committee for Space Data Systems. The model has been helpful in evaluating the structure of the PANDORA archive. However, our focus on building and managing an existing archive, rather than developing a universal model for archiving, has produced some differences in language as well as approach (although we believe the differences in terminology to be greater than the differences in understanding).
Because we have taken and continue to take a holistic view of what is required to secure the ongoing accessibility of digital collections, we have found it useful to distinguish between:
In our model, preservation of accessibility depends on both. However imprecise and unsatisfactory these terms might be, we have found them useful in linking collection building with collection maintenance, and reminding us that we need to energetically pursue both. 6
In the discussion that follows I have used this dichotomy mainly because I want to emphasise the progress that has been made in archiving and the relatively undeveloped state of our response to long-term preservation challenges.
Most Australian deposit libraries have been collecting PFDP since the early- to mid-1980s, but until the mid-1990s there was little attention given to aspects other than controlling access. In 1996 we surveyed the PFDP in NLA's collections, and concluded that they were inadequately identified and described for long-term management or even for locating them reliably by format. We also found that significant numbers were already affected by technological obsolescence, although in many cases the problem was as basic as the Library having disposed of all PCs with appropriate disk drives. Steps we have taken include improving control of these items as they enter our collections, identifying and describing their software dependencies, and copying publications from diskette to CD-R carriers, which are stored securely. As a result, we have a much better idea of what we have and of the suite of formats and software and hardware dependencies that we will have to manage. We are pleased to have made this progress, but recognise that long-term preservation will require other strategies.
Recently we published the selection guidelines we use for prioritising PFDP for preservation action.7 While we would like to preserve everything, the complexity of many PFDP items means that we almost certainly will not have the resources to do so. 8
The number of "digital manuscripts" is probably small but may well include material of research value such as early drafts of significant literary works. Our approach in the face of unidentified diskettes containing potentially important information in unknown formats has been a "data archaeology" one, using data recognition and format conversion software to recover at least basic information that can be used.
We agree with those who say that rescue is a poor kind of preservation strategy, but we are pleased to have been able to use it to good effect. 9
PANDORA has been well documented through the NLA's website and numerous papers in recent years, and there is not space to describe it fully here.10 (Some current technical details are included in Appendix A.) However, I do want to discuss some of the business principles - most of them controversial and all subject to review - on which PANDORA operates.
It was agreed at a very early stage that we should strive to capture not just the content but the look and feel of digital publications. Many contain software plug-ins or dynamic features that make this one of the most challenging aspects of our work. In some cases we struggle even to capture such material adequately.
We currently do not attempt to capture everything published online that relates to Australia. While we argue that a similar situation exists even with print publications despite our legal deposit and claiming programs, we are indeed much more selective about digital publications. This position is driven by resource issues, both now and in the future as we expect it to be, in the context of the Library's insistence that what it collects and preserves must be accessible. Our experience is that achieving this with digital materials is much more resource intensive than with print materials.
If we wanted an approach allowing us to:
then we believed we could only afford to collect selectively. As we think these are benefits worth having, we have pursued such a selective approach.
We are reasonably comfortable with this, in that a large number of Australian sites are .com sites largely made up of promotional information unlikely to justify the commitment of resources required for long-term preservation. By applying professional judgement to the selection process we are able to live within our resources while focusing on what seems to be most important.
Others have taken a quite different approach, aiming to collect and preserve everything on the Web relevant to their collecting focus, such as their entire national domain. (In the case of the US-based Internet Archive, that amounts to snapshots of everything, everywhere, although the Archive management acknowledges that the capture process is neither exhaustive nor guaranteed to work for everything.11)
Arguments in favour of a comprehensive capturing approach include the following:
The National Library of Australia does not reject any of these arguments individually, nor the conclusion that for some national libraries comprehensive collecting is viable - ultimately it is their own business. However, on balance we still believe that we can only deliver the kind of quality-controlled, well-managed archive we believe PANDORA should be, by continuing to collect selectively.
Having said that, we also recognise that it will be important to capture a picture of the entire Australian domain periodically. We are currently looking at ways of doing that, possibly in conjunction with existing services which "vacuum" the Web, including its Australian component. We see these as two related but different exercises aimed at different objectives, the first providing comprehensive accessibility to a body of selected, high quality research resources, the second providing more of a large picture worth studying in itself without necessarily being able to zoom in on some of the detail.
The selection process
While a selective approach implies a selection process, we don't believe there is anything mysterious about this. Selection should be based on the collection development policy or whatever business needs drive the institution's collecting. In a networked digital environment we have found that we need to take account of some additional considerations in applying our CDP, including:
The PANDORA selection guidelines12 is a living document that is subject to regular review. The key elements of the guidelines are to do with the Australianness of material, and its relevance for future research use.
This business principle is based on a purely pragmatic issue: how to limit the number of publications eligible for inclusion in the archive. As printed Australian publications are already collected comprehensively between the national and state libraries, it seemed sensible to concentrate resources on collecting and managing those publications which exist only in digital format. To date the application of this principle has eliminated about seventy percent of titles that would otherwise fall within the PANDORA selection guidelines.
This principle has been challenged from the beginning by the argument that online access offers functionality which is different from print even if the content is substantially the same. Already we have decided to archive e-journals being indexed by indexing and abstracting services regardless of print equivalence, and it seems likely that this principle may eventually be abandoned in the face of user expectations.
The decision to catalogue online publications onto the (Australian) National Bibliographic Database is an acknowledgment that we have a responsibility to record digital publications as part of the national imprint. Creating catalogue records also facilitates integrated access to Australian publications regardless of format.
Permanent naming
Files in our digital collections need to be reliably found. The location-based nature of the URL is a major weakness, and NLA, along with a number of other national libraries, has been exploring options for persistent identification of files. A paper on managing online resources for persistent access is expected to be released shortly.
From the inception of the PANDORA project, the National Library envisaged that the collection, provision of access to and preservation of Australian digital publications would be a co-operative activity involving itself and the other deposit institutions, such as the State Libraries and ScreenSound Australia. Essentially the National Library of Australia is requesting the State Libraries to take responsibility for archiving significant online publications relating to their own State, such as State government publications, and other categories to be develop in agreement with NLA. Four of the six State Libraries are now archiving online publications to varying degrees. Two have developed their own internal processes and two are using the processes and procedures set up by the PANDORA project.
Earlier discussion of possible national models for archiving suggested some important conclusions:
From the National Librarys point of view it does not matter whether the deposit institutions develop their own archiving infrastructure or join the PANDORA Archive as a contributing partner. What matters is that the institutions work together to ensure that Australias significant digital publications are collected and preserved for future access cost-effectively.
The key undertakings that NLA sees as critical to achieving such an outcome include:
Given the diverse collections for which NLA is responsible, we are most likely to use a combination of strategies for the foreseeable future. Although a single solution is theoretically attractive and worth pursuing, our thinking and experience has not developed to the stage where we can rely on a single approach to deal with what are reasonably urgent needs. Even given the considerable potential benefits and elegance- of an emulation-based approach we are very likely to need a program that includes:
We are currently in the process of working out how to distinguish which strategies will work best for which materials. This will probably still leave gaps, either because we lack the technology to deal with some formats, or more likely because they will not fit into the workflows we adopt to manage the strategies cost-effectively.
Because we are not satisfied that any single approach will give us the security of access that we require, we expect to store and manage multiple copies of archived files, maintaining both currently accessible versions and versions of files in their native formats.
We are aware that whatever strategies we use, we will need excellent technical and management systems, and excellent metadata to support them. Even with those in place, we expect quality control of our access paths to be an ongoing commitment, raising questions of "how close to the original intended functionality is close enough" for every file being accessed.
We are engaged in some modest but practical work to bring theory closer to our immediate needs. Our Migration Strategy Group (MSG) project is engaged in three exercises:
We do not yet have a suitable set of mechanisms in place to manage even these reasonably short-term responses to the problems imposed by software dependency. While the systems being procured through the Digital Services Project will help, we will have to assess the contribution they can make, and how much human intervention will still be needed. Given the nature of our digital collections and the business principles we are trying to support it is likely that significant human intervention and decision-making will be required for some time to come.
"So far, so good", but a lot of work is still needed if our national digital collection is to live forever.
Technical Details: PANDORA's Dimensions
At the last count, there were over 900 titles in the PANDORA Archive, occupying approximately 40GB. Approximately two-thirds of the titles have been archived on a one-off basis. This is either because the title is static, for example a completed report or project, or because the site has been archived as a "snapshot" example of how the Internet is being used by Australians.
The remaining one-third of items in PANDORA are being archived on a regular basis, ranging from weekly to annually. These items may be electronic journals, or sites of ongoing research interest that are being updated regularly.
On average, around 35 new titles are selected and archived each month. In addition, an average of around 30 titles that are being archived on a regular basis are "regathered" each month.
The data is stored in a Unix file system. Currently, the production area of the PANDORA archive is around 15GB, and the working file area (new gatherings being assessed and fixed prior to moving them into production) is around 7GB.
For particularly complex or large publications, the publisher transfers the files over the Internet or sends them on CD-ROM. However, for the majority of titles we use a robot to gather the desired files directly from the Internet. We are in the process of changing the software used for these gatherings to two main tools: HTTrack and Teleport Executive.
We are currently developing in-house at the Library a new gathering system. We are moving away from a rather clumsy system of an Access database containing management information combined with a customised communications interface for submission of requests and problem logging. Our new collection management system will be an integrated Web-based interface. It will provide for the entry and maintenance of management information describing the title and interactions with the publisher, as well as the assigning of persistent identifiers, initiation and monitoring of gathering requests, tracking of and repairing problems, and automatic generation or amendment of title entry pages.
1 Available at <http://www.naa.gov.au/recordkeeping/control/rkms/summary.htm>
2 Available at <www.prov.vic.gov.au/vers/>
3
PANDORA stands for Preserving and Accessing Networked
Documentary Resources of Australia
4 Available at <http://www.nla.gov.au/pandora/bpm.html>
and
<http://www..nla.gov.au/pandora/ldmv2.html>
respectively
5 Available at <http://www.nla.gov.au/dsp/>
6
Preservation specialists may recognise this as a significant
advance and one of the more positive aspect of the growth in digital
collections
7 Available at <http://www.nla.gov.au/policy/selectgl.html>
8
Our intentions are not well conveyed by a recent headline
(Information Retrieval and Library Automation, 36(5), p9: "National
Library of Australia: Deciding Which Disks to Ditch", although this may be the
effective outcome.
9 Both data rescue and
technology preservation have received a bad press as long-term strategies, but
we regret that their quite reasonable rejection has lost us some years'
experience in exploring when and how we should use them as a bridge to other
strategies.
10 The best entry point to both the
archive and to information about it is
<http://www.nla.gov.au/pandora/>
11
Personal comments by Brewster Kahle, Australian Library and
Information Association conference, Canberra, 24 October 2000
12
National Library of Australia. (1999).
Guidelines for the selection of online
Australian publications intended for preservation by the National Library of
Australia
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |