National Library of Australia Staff Papers, 1998

Font Size:  Small  Medium  Large

Preservation of electronic information: what we should be thinking about now

A paper presented by Colin Webb, Manager, Information Preservation, National Library of Australia, as a keynote address to the 9th Annual Symposium on Preservation, National Diet Library, Japan 24 November 1998

Chairperson, fellow participants in this symposium: let me begin by saying that it is an honour to be invited to address this symposium. I am very grateful to the National Diet Library for the opportunity to be part of this gathering, and to meet with you all.

What I want to do this afternoon is to talk about the way we have approached the preservation of digital information in Australia, particularly at our National Library where I work. I will also refer to some developments in Europe and North America. During the course of my visit I will be very interested to hear how other participants are approaching similar problems.

I will start by being specific about how I use the terms "electronic information" and "digital information". It is necessary to make this distinction because I am really going to focus on the latter, and I need to warn you in case some people are disappointed when I do not mention things like analogue sound and video recordings. These are legitimately considered to be electronic information, but they are not digital. While there is overlap in the preservation of digital information and electronic information in general, I believe they are best treated as separate fields that can learn from each other. At the National Library of Australia I am responsible for both areas, but today I want to talk only about digital information and how we preserve it.

I also want to distinguish between three types of digital information of interest to libraries: firstly, online publications generally made available over the Internet; secondly, what we might call physical format digital publications issued on tangible media such as CD-ROM and floppy disk; and thirdly, other digital collections such as the files resulting from digital imaging programs, digital sound files, computer-based record-keeping systems, datasets, and so on. I will be mainly talking about the first two kinds – online and physical format digital publications, because I think that is where the real preservation problems lie for us.

Almost 10 years ago I was working for another national institution in Australia, the National Archives, and I was speaking at a workshop on managing electronic records. The highlight of a rather dull workshop turned out to be a dispute between our IT manager and myself. As the preservation manager, I said the best we could do was to print out the record onto paper or microfilm, while our IT specialist said, in effect: "Don’t be stupid! We will manage this like any other computer-based data." I spent the next five years trying to work out which one of us was right. I eventually decided that we were both wrong, but that I was more wrong than he was. Preservation for digital information will require more than normal IT procedures, but it will have to be built around the technology itself. In most cases, a piece of paper just won’t do the job!

In some ways we are still caught up in the same kind of debate. We have made great progress over 10 years, and the steps we are taking now will make preservation possible in the future. What we have done so far has been somewhat like building a whole new library system, so it shouldn’t dismay us if there are some frustrations. But the next great challenge will be to ensure the digital information we are archiving now remains accessible in the long term. That is a challenge that still contains many uncertainties.

I take it as a sign of genuine progress that we have moved from a general, rather blurred focus on the difficulties to a much sharper focus on how we might deal with quite specific problems. The simple facts are these: we are still trying to deal with many of the problems we already know about, and we are trying to recognise the critical problems ahead of us.

This experience has been reflected in Australia, where many librarians and preservation professionals said even just three or four years ago that digital resources were not worth preserving, and that the task was impossible anyway: where could we even begin? From this we have moved through a range of small beginnings – sometimes deciding to try something even if we couldn’t see how to do everything - to a more systematic program of building infrastructure that could support long term preservation as well as current access.

Most of the rest of my talk will be about the various kinds of infrastructure we have been building, and the issues still to be resolved before we can be confident that we will meet our preservation responsibilities.

In 1995 the National Library of Australia set up a small committee to establish guidelines for selecting online publications we would seek to preserve. In 1996, the Library took the next logical step, embarking on a pilot which was later to become known as the PANDORA project. In PANDORA, (which stands for Preserving and Accessing Networked DOcumentary Resources in Australia), we try to archive the online publications chosen using our selection guidelines.

It is important that I clarify another terminology issue at this point. At the start of PANDORA we talked about preservation, and of course preservation is still part of the PANDORA name, but we have come to recognise that PANDORA is archiving these titles. It is a series of early steps in the preservation process, but it is similar to putting a book in a library building – it is not the end of the preservation story, and there is much more to be done.

PANDORA has been through a number of phases, growing from a small internal team looking at possibilities, to a proof of concept project, and then to a national archiving model that we are now implementing. The project has been widely reported, and there is only time here to give a brief overview. A much more detailed description, and the PANDORA archive itself can be found on the NLA web site.

Why did the Library get involved with this kind of material? We believed - and still believe - that Australian digital publications should be seen as another information source to be integrated with existing sources. But they are at great risk of being lost if action is not taken. So we believe that we have to provide the pathways for long term access. We also believe libraries are well placed to be the main crystallisation points for digital archiving.

We do not aim to capture all of our nation’s digital publications, online or otherwise, unlike some other archiving projects. We believe we can do a better job by being selective, although we respect the views of others who do try to archive a nation’s entire online publishing output, such as the National Library of Sweden’s Kulturarw project, or the entire global output, such as the Internet Archive.

There are many lessons to be learnt in such an exercise. One lesson we have learned is that things are both easier and harder than they first appear.

At every stage we have had to find concepts and models that simplify what looks like an immensely complex landscape. In PANDORA we have recognised many useful models familiar to us from the print world: models which don't explain everything but do provide us with some help. Collection development policies; classifying and describing information resources; providing access that takes account of copyright; managing risks of loss – these are everyday working concepts from a non-digital context for most of us. In all of these areas, we have been able to say: "What we know from the print world is a good starting point". On a journey of a thousand miles, formulating basic principles has helped us see where we need to go, and given us to the will and the courage to go through the front door and to head out towards the horizon.

But PANDORA is made up of many boxes, each one full of decisions and dilemmas that have been with us every step of the way. The bright side of this is that every difficulty addressed is a lesson learnt, helping us to build a workable digital archive based on sound and understandable principles deal well with the required level of complexity.

In establishing the required concepts and in working with the details, we have relied heavily on modelling - trying to map all the relationships between the processes, the data, database structures, and metadata. For most of 1997 this was our main focus, trying to get it right in a way that would work for a larger archive than the 30 titles that had been taken in already.

In 1998, while continuing to build the archive, (which now holds almost 100 titles), we have focused most of our management attention on two areas that will take PANDORA forward, and will strengthen our ability to handle digital information generally. We want to build a better framework in which we can manage our archive effectively and efficiently; and we want to identify the issues where we need to take action and enlist partners who might be interested in taking action with us.

Our technical framework for PANDORA has consisted of a mixture of proprietary software and hardware, open standard tools, and purpose built applications we have developed ourselves. The basic components include:

  • a digital object gatherer called Harvest, which only partly meets our needs;
  • a Web server ;
  • our National Library catalogue using metadata to describe archived objects at title level. Our Web-based OPAC is also the Web interface to the archive;
  • we store the archived objects as a hierarchical set of files on an IBM RISC 6000 computer;
  • and we use some purpose built programs that give us limited management of the archive.

It has been obvious for some time that we need to improve some of this to cope with the expected size of the archive. We also recognise that digital libraries and archives cannot afford to have unintegrated systems and pockets of storage when their core business is increasingly to provide linked, rapid, and secure access and retrieval. The need to improve the technical infrastructure of PANDORA itself, and the need to improve the way we manage it and our other collections, have both forced us in the direction of a major systems integration project. We call this our Digital Services Project, and we will use it to collect, store, manage, and deliver all kinds of digital objects the Library holds, including sound recordings, pictorial images and text files, as well as the archived online publications. We also want to use it to help us manage physical collections of traditional library materials, as well as the physical format digital publications such as CD-ROMs that we are unable to store within the PANDORA project. Our Digital Services Project will not provide storage for everything, but we hope it can give us management systems for most of our library collections, both digital and non-digital.

We will be taking our documentation for this project to the market place in the next few weeks, looking for proposals that address our needs. We are very committed to sharing information on what we are doing, so our documentation will be made available on our Web site once the Request for Tender has been released.

This provides a good link to our second main preoccupation over the past year. We see information sharing as critically important – we cannot afford to waste effort, so we invest some of our limited time and energy in trying to establish mutually beneficial relationships with other people involved in digital archiving. We have established links with many other libraries and projects overseas. To date we have mainly looked to partners in Europe and North America, which may reflect our poorly developed knowledge of what is happening elsewhere.

What are the issues we should be thinking about now? I want to look at this from two perspectives: the issues involved in building national archives of digital publications, and the issues specifically involved in long term preservation.

Issues in building national archives of digital publications

We have been discussing issues with a number of other national libraries and key projects that we identified as particularly relevant. The issues we have been discussing fall into six areas:

  • Collection building
  • Description and control
  • Managing the collection
  • Metadata
  • Organisation and cooperation, and
  • Preservation

I want to reflect briefly on each of these, from a preservation perspective.

Under collection building, there are some important issues about how best to get publications into the archive. But there is an even more fundamental question: what is the archive? In an online environment, do we have to move bits and bytes to a particular location, or can we simply leave things where they are and point to them?

While we believe there is potential to have de-centralised archives – a model we are in fact exploring in Australia, as I will describe shortly - we are convinced that most digital objects will have to be moved to archiving sites offering greater ongoing security than the publisher’s Web site can usually offer.

So "collecting" online publications becomes a process of discovering what is being published, registering it in some way, and moving a copy to where it can be preserved. This raises a number of technical, organisational and legal or ethical questions.

What technology can we use to capture online publications residing on remote servers? We are using indexing software as a robot to search the Web and make copies in accordance with specific instructions. It is a difficult process with imperfect tools, and we would like to improve the tools we use for it.

Of course, another approach would be to make arrangements with publishers to "push" their publications to the archive, rather than us having to "pull" the publication from their site. This approach is gaining favour in Europe, with agreements between various libraries and major academic publishers, such as in the NEDLIB project – the Networked European Deposit Library, a consortium of nine European national libraries, a national archive, and three large publishers, all coordinated by the Royal Library of the Netherlands. That approach may or may not work with the myriad of smaller publishers populating the Web, although it is conceivable that legal deposit legislation could require publishers of all kinds and sizes to at least register their publications. In Australia we are hoping to have deposit arrangements for digital publications sorted out soon.

At the organisational level, there is a major question about what one should try to collect and preserve. As I have already mentioned, in Australia we have decided to be selective, while some other national libraries, especially in Scandinavia, are trying to archive everything. It requires some hard thinking about what one is trying to achieve, and whether it is achievable.

Of course, copying someone else’s publication from their site does raise legal and ethical issues. We have to negotiate the right to do what we have to do; we also have to ensure that the rights of creators, publishers, and so on are not jeopardised by our archiving and access activities.

However the publications become part of the archive, they have to be described and controlled. From a National Library perspective, this raises important issues about the level of control we should apply. Is it good enough to simply make the archive content available to external search engines? We have concluded that we need to include digital publications in our national bibliography, integrated as research resources with the rest of our collections.

Describing digital objects now presents few problems within our cataloguing processes. Cataloguing rules and widely used cataloguing structures such as MARC have evolved, sometimes a little slowly, in response to new requirements.

For online publications we have a pressing need for a reliable system of permanent naming in place of the notoriously unreliable URL. Without a more persistent naming system we will continue to lose hypertext access whenever file locations change. In Australia we have been using a PURL Resolver Service to give permanent names to our own archived copies, but we need a solution that has wider acceptance. That might well be the URN (or Universal Resource Names) being developed and promoted by the Finnish National Library.

I must say it is easier to talk about managing a collection of digital publications than it is to actually do it!

For national libraries seeking to maintain an accurate record of a nation’s documentary heritage there are issues of version control and authentication. Because techniques such as encryption, time stamping, watermarking and digital signatures are also of vital concern to business interests, it may be that solutions for digital archives will emerge from the commercial sector.

As I have already indicated, we give a high priority to getting the technical framework in place to manage our archive in support our business objectives.

When we seek to manage distributed archives there are further complications. We need to think hard about the trade-offs involved. The task of building and maintaining national collections of digital publications may well be more than single institutions can sustain; on the other hand, cooperative models introduce another layer of complexity and costs. This is an important practical issue that we are exploring in Australia. Whatever one’s view on the advantages and disadvantages of decentralising archiving responsibilities, we have to build something we can manage. Good intentions and fine theories are not enough: preservation will depend on something that works, not on something that should have worked but didn’t.

Metadata becomes a vital part of managing digital collections. We have to be able to manage them actively. Somehow we will have to integrate the information we need to preserve these publications with the metadata used to streamline other processes of management and access.

There has been much work on metadata for digital publications, most prominently through the Dublin Core series of meetings, which has been developing a set of core metadata elements principally aimed at facilitating resource discovery – connecting users with useful digital resources.

In the preservation community we have different metadata needs, which have been partially addressed by a Research Libraries Group working party on Preservation Issues of Metadata. Earlier this year the working party recommended a set of 16 preservation data elements, which we really need to test over time.

Finally in this broad sweep of national digital archiving issues, there are questions about relationships – whom do we need to bring together? How do we build on the work that is going on in various countries and various sectors?

I have already referred to our decentralised national archiving model in Australia, and I would like to expand on it a little more now. We have decided that someone needs to take a lead, and as the National Library we have taken that role. We also want other players in Australia to take on archiving roles. They will probably include some, although not necessarily all, of the state libraries, some of the university libraries, other collecting institutions, and a few publishers. We don’t expect everyone to take the same role, and we don’t expect everyone’s role to stay the same. There are certain to be changes over time, and some of our partners will only be able to commit themselves to archiving small amounts of information for a few years, not forever. But we still need them to do it, and to do it well. We also need to negotiate agreements with them regarding what happens to their archive when they can maintain it no longer. Despite some early concerns, we are now comfortable with this as a working model, because we think it will reflect reality. Our current national model could be described as a centrally led, but distributed, network of carefully managed but dynamic archiving relationships.

Of course many other models exist for national and transnational action, including the Digital Library Federation of the US, the Canadian Initiative on Digital Libraries (CIDL), the EVA project in Finland, and the NEDLIB project I have already mentioned. There are also some outstanding models for cooperative action within specific sectors, such as the framework for creating and preserving digital archives of the Arts and Humanities Data Service of the higher education sector in the UK.

Libraries particularly need to develop relationships with digital publishers. This relationship can help in promoting intelligent publishing conventions that might make preservation easier, through the use of standard supportable formats, or the transfer of expertise in how the publications are put together. Ultimately, we will also rely on publishers accepting the work we do, because we cannot preserve without copying, which tends to worry publishers.

In digital preservation one can easily get the feeling that development is so rapid one can hardly keep track of it – it was described a few years ago as similar to drinking from a fire hose. And yet at the same time many of the most important issues remain unresolved. This is partly because the material we are trying to deal with keeps changing, so our partially conceived solutions are overtaken before we even get the chance to test them. But it also reflects our failure to learn from each other.

The preservation community has had to step out into a wider research community. In Australia, we had come to think that we knew where to look for answers to our preservation difficulties. In this field of digital preservation it is harder to know who has the expertise to help us. It calls for teams of expertise, good communications, a willingness to try things and a willingness to share honest information about the results. It probably also requires a readiness to look in unlikely places for ideas that might work. A good example of this is the work of the Consultative Committee for Space Data Systems, which has developed areference model for an Open Archival Information System, which has informed our approach to our Digital Services Project and PANDORA.

An especially important relationship could be one with higher education. Whenever we have asked, we have found a deep level of interest among Australian universities, and on a recent visit to the UK I found a very active research effort under way in many areas relevant to digital preservation. In Australia, if not elsewhere, we have a problem finding the time to establish and maintain relationships with the people doing the kind of research we need.

This is especially so in the area I want to separate out, perhaps artificially, as "preservation".

As I have already suggested, the steps involved in identifying, capturing, controlling and managing digital publications all play a critical role in making preservation possible, but of themselves they are insufficient to ensure long term accessibility. There will be a range of threats to digital archives, needing to be addressed routinely and at recurring intervals. If we are going to maintain access we will need to develop and commit to realistic pathways that address the threats.

Many preservation measures will be included in standard IT practice such as the use of secure backups and the development of counter-disaster plans. However, the certainty of future changes in technology demands other responses that we still need to develop and test.

I want to finish this presentation by focusing on the specific preservation issues we are trying to explore. The emphasis here is on physical format publications. We have given them some priority because they are less amenable to the normal IT management processes that will help us with online publications. However, many of the needs I am going to mention here apply to archived online resources as well.

In an ideal world deposit libraries might preserve everything, but operating in an environment of technical and resource constraints means that priorities must be set. Already there are enough indications of difficulty for us to recognise that it may not be possible to preserve access to everything, and that some options will be costly or ineffective for some materials. Libraries in Australia and elsewhere need more information on costs and options to help them decide what is feasible to preserve and how best to do it.

We are currently developing a research agenda in Australia to help us answer those questions. For many issues our main task is to find the people with the answers and listen to them. On the other hand, we also expect that even when we know what is possible we will still have to decide what works best in our own operating environment.

Our agenda is framed around three steps:

  • Identifying what we already know or believe
  • Identifying the issues needing to be resolved before we can formulate strategies for preservation, and
  • Using the research in formulating a national preservation approach.

What do we "know" already?

I have tried to list the options people turn to when confronted with the digital preservation challenge. We don’t have time here to go through the full list of what we know and believe.

Options

If we are going to preserve a physical-format digital publication, what seem to be our options?

  • documenting the item and storing it with the hope that some "rescue" technology will make it accessible in the future – this doesn’t look very hopeful!
  • "refreshing" the item by transferring it to another copy of the same kind of carrier – this can only be a short term answer
  • "transferring" the item to a more stable carrier such a CD-R or a more highly maintained system such as a backed-up tape archive – this is also only an interim solution
  • "freezing" the item by transferring it to a very stable, probably human readable, form such as print, microfilm, or something like the HD-Rosetta being developed by the Norsam Technologies people at Los Alamos in USA – this will only work for non-interactive items
  • "migrating" the item by copying the data and its supporting software to a currently accessible computer format, (with or without trying to maintain the "look and feel" of the original)
  • finding software that will "emulate" the original operating software and make data accessible in a different operating environment
  • migrating the item to a current operating environment, but maintaining a preservation master copy of the data in its original format so it can be emulated when losses through migration become unacceptable.

How to achieve these, how to choose the most cost effective and appropriate combination of them, and whether there are other viable options, are the basic tasks before us.

What we need – issues to research before we can formulate strategies

These are the things we need before we can formulate preservation strategies – the issues we need to be thinking about.

  1. We need agreed arrangements for recording metadata that will help us preserve publications
  2. We need cost effective ways of predicting and measuring deterioration rates across collections
  3. We have to find cost effective ways to predict when changes in technology are likely to make collection material inaccessible
  4. We need more information on the feasibility of "rescuing" currently inaccessible materials
  5. Most particularly we need more information on ways of keeping materials accessible:
    i. Either by keeping their original access pathways available and operational (what we might call "technology preservation") In the long-term there is very little merit in a proposition to maintain an archive of hardware and software, but it could be the only strategy available – a bridge - pending the development of more sustainable preservation pathways.
    ii. Or, by finding or developing software that will emulate the original operating software. This is known to be a major research focus within Europe and North America. The main difficulties with emulation are likely to be the costs, which may mean that some less widely used formats are judged too expensive to emulate, and the need to develop emulators for the emulators. It is an approach that could be adopted to shift part of the cost burden to the user, which might be a good or bad thing.
  6. iii. Or, by migrating publications and their supporting software to each new generation of an operating platform.
    This is the strategy the NLA intends to focus its own research efforts on in the near future. There is still much work to be done to test reliable, cost-effective migration procedures. We are particularly interested in liaising closely with the CEDARS project in the UK. CEDARS, running out of the Universities of Leeds, Oxford and Cambridge, aims to test emulation, migration and other approaches. The US-based Council on Library and Information Resources is also playing a very important role. They have commissioned comparative studies on digital preservation strategies by Jeff Rothenberg of the Rand Corporation, and by Cornell University Library. We will be watching with keen interest.

  7. Finally, we need more information on issues in managing preservation. In addition to the questions already posed in this paper, we need to know:
  • How can we identify the essential elements of an item that must remain accessible?
  • How do we decide the most appropriate preservation path for a particular item – what indicators can we use to guide our decisions?
  • How can we test our ability to preserve?

Conclusion

As I said at the start, I believe we have made great progress, but that we are still presented with some unresolved issues. We have solved the short term hardware obsolescence problem by keeping a track of the hardware we need to keep for a time after it has passed out of general use, and by "in-time transfer" of data to currently supported hardware. We have largely solved the media instability problem by transferring data to more stable carriers, which we can do without affecting the data content at all. We have solved the problem of Internet instability by archiving publications in projects such as PANDORA. I believe we have also solved a lot of the organisational commitment problem by strongly asserting the role of libraries, especially deposit libraries which should have a much better chance of maintaining commitment and resources than would other players. And we are even coming to grips with the requirements for handling "upgradable" formats, where we can create homogenous bodies of data that can be migrated in bulk, or efficiently accessed by emulation programs. I am not saying that all the problems have been solved on those issues – as I have described in this paper, we have some way to go in proving we can manage all of them - but at least the basic concepts are in place.

What we are still grappling with is the issue of software dependency, especially where publications use customised version of software, or software that may be less widespread. For this we need good solid research that will show us what is possible.

But I am hopeful. At least we no longer believe we can leave it to someone else, saying it's not our business, or hoping it will just happen.

Thank you.