Long-Term Management and Preservation of Publications on CD-ROMs and Floppy Disks: Technical Issues
Presentation by Colin Webb, Manager, Information Preservation, National Library of Australia at the National Consultative Meeting on the Management of Australian Physical-Format Digital Publications, National Library of Australia, October 27-28 1997
- Introduction
- Range of Technical Issues from Surveys
- Other Concerns
- Where we Are and What we Have Been Doing
- What's Happening Elsewhere
- Potential Approaches?
- Discussion points for Groups - Technical Issues Session
Introduction
In this session I am going to try to air some issues as a way into our group discussion of technical barriers and possibilities. My presentation will not go into technical detail, but talk more at the level of how we can manage problems. Everything I will say you almost certainly have heard before. I hope that the necessary technical detail will come out in our group discussions - I am just trying to set the scene. This suits me fine as I am not a computing professional, but a preservation professional, trying to bring together the required expertise to focus on what are multi-faceted challenges.
To bring out the most important issues, I want to look firstly at the feedback we got from publishers and libraries in our recent surveys, then talk about some other issues that affect our ability to manage the preservation task. I will then very briefly look at our attempts so far to provide a preservation response, and finally report on potential solutions raised in our surveys, especially our survey of publishers. This should lead into our group discussions, which I hope will focus on questions of what we can do to try to find some answers to the challenges.
Range of Technical Issues from Surveys
When we surveyed publishers' and library views of the challenges, we came up with a range of hardware issues, software issues, media instability issues, and what might be called feasibility issues:
We all talk about obsolescence - superseded hardware, superseded software - changes in operating systems, platform programs, delivery systems and access technologies, rendering programs, and other software dependencies. A British Library R&D report in 1996 put it this way:
To access the data we need the originating software. To run the software we need the operating system it was designed to run on. To run this operating software we need the hardware it was designed to run on.
While the picture is not quite that gloomy - we don't necessarily need the original software or hardware to access data - in some ways it is more complex than the quote suggests, because there are many different kinds of software in most multimedia products. The bottom line is that we are dealing with many layers of dependence on machines and programs that we simply discard after a few years.
Along with the obsolete equipment comes a knowledge / expertise problem - even if we can find the program, we can't find anyone here who knows how to operate the darn thing.
Even if we can migrate these publications, there is a challenge in maintaining the integrity of the information. Somehow we have to preserve the essence of the publication, when in fact moving to a new platform is most likely to change the way it works and how it looks and feels to the user. We have to start thinking in terms of "lossy" and "lossless" migration, just as we talk about lossy and lossless compression.
We also have to be able to verify or authenticate the material we copy so that users know what they are looking at.
There are a lot of issues about copying restrictions - in many cases these are legal and ethical constraints where technology may help us find a solution. In some cases, however, the copying restrictions themselves are part of the logic of the product - security devices to prevent copying or to close off access.
We are starting to encounter hybrid disks - physical format publications that provide access to an Internet site, for example - with all the challenges of trying to preserve both parts and maintaining the relationship between them.
And costs - will the cost to find and apply solutions be prohibitive? Even if we can find ways that may be acceptable one-off, will the community, or users, or whoever has to pay for it, be willing and able to do so? Whereas we can give a lengthy useable life to most traditional library materials by storing them well, electronic materials will all require expensive intervention if they are to survive with anything like their intended functionality. The 1996 British Library R&D report I have already quoted gave some tentative estimates for the total costs of managing and preserving publications in print and electronic formats, for the first 25 years after publication:
- $A245 for a print publication
- $A4,717 for a digital publication stored offline with a manual retrieval system
- $A10,338 for a digital publication stored online or nearline in an
automated system
(monetary figures converted at current exchange rates)
These figures are problematic (as the report admits). In fact, we don't have any firm idea of how much it is going to cost, because we don't know what is going to be required to do it. When we do have a better idea, one of our greatest needs will be for believable costings and cost models. However, the relative scale of these estimates feels about right.
Other Concerns
These challenges you told us about map well with issues that we and other libraries had identified as concerns. There are other technical issues that also engage our attention, because they profoundly affect our ability to manage these materials:
We need quite a lot of information about the publications so that we can manage them efficiently and effectively. We have to be able to find them in our collections and group them for attention; we have to be able to make decisions about risks and priorities.
The rate of deterioration is unpredictable, but we assume there is a sharp deterioration curve and only a short lead time. This unpredictability comes from the diverse range of products that we lump together under the name "physical format electronic publications", the range of qualities used by manufacturers, and the nature of the materials. It also relates to the volatile nature of the market - we see "killer technologies" sweeping through, often led by give-aways that aim for maximum market coverage in a very short time.
We require ways of monitoring the risk levels that will indicate when action is needed, such as deterioration of the data, changes in technology that mean it is time to move on to another carrier, another format.
We have talked about hardware and software obsolescence - we find we are struggling with what that means. In fact we work in a context of "effective obsolescence": even if the hardware and software is available somewhere, it may not be available where we need it. Getting hold of it may be so difficult that we decide the publication is not worth the effort required to preserve it. In this way, quite a lot of middle ranking material - not national treasures, but not junk either - may well be lost because it will be difficult - not impossible, but difficult - to retrieve it.
We talk about migration, but most successful migration experience has been with large quantities of homogenous data. Because of the diversity of multimedia publications, right down to their internal diversity, migration looks like a one-by-one thing. The migration pathways we have to follow are uncertain and unclear. Some material just may not be migratable, and most publications may lose some of their look and feel over successive migrations. We have heard conflicting stories about CDROMs - that they will be easier to migrate than floppy disks; that they will be impossible to migrate.
For some "flat files", migration to paper is probably the most cost effective preservation path, but there is so little material that fits into that category that it is probably not worth talking about. For anything other than flat files, printing out to paper is like taking a photocopy of a black and white Polaroid of a TV screen showing a colour movie of a novel, as a way of capturing the richness of the novel.
We also assume that migration is an ongoing process. We operate in an era of archival processes, not archival products, so we won't be moving to a standard format that will work forever. To what extent will we be able to batch and automate that ongoing migration? Will we have to deal with diversity every time we reformat?
We know we probably won't be able to preserve every electronic publication produced in Australia, so we will also have to work out our priorities. Technical issues will tend to distort those priorities - publications that are difficult to preserve may not be preserved; publications that are most at risk of deterioration or of being overtaken by technological change may be given a higher priority than their value warrants.
We have to find suitable security, storage, packaging and handling arrangements to ensure the carrier survives long enough for us to migrate the information.
And we need to refine our expertise and even our terminology. There are layers of processes in managing this stuff that we aren't yet describing adequately, eg different kinds of metadata, different degrees of migration.
Where We Are and What We Have Been Doing
The point I want to make is that libraries are committed to finding cost effective ways of dealing with these issues, so that this part of our cultural heritage is accessible in the future. The National Library of Australia and others are willing to take a leading role in looking for solutions and in building viable preservation pathways. I am going to talk about what we have been doing at the National Library, and others may well have their own stories to tell, but it will be clear that we haven't worked out how to do it - we are in the taxi looking for the airport; we have taken the first steps in a journey of a thousand miles, but there is a long way to go, and we aren't sure what the next steps should be - that's really why we are holding this meeting.
From what we have been doing already:
- we believe we know how to store and handle publications in these formats
- we believe we know how to control access and use
- as we talked this morning, we aren't completely on top of how to get this material into our collections, but we think we are on the way
- we believe we know what information we need to manage these publications, (though we are still sorting out where to put it)
- we believe we know what is in our collection, both in terms of numbers and titles, but also operating requirements. When we conducted a major survey of our holdings we found a great range of operating systems, hardware requirements, and software requirements, some of which we couldn't supply. We found we were right out of Apricot, Microbee, Commodore, BBC and Apple II, IIe, IIc, IIgs and II+ support facilities, if we ever had them
- we believe we can migrate some floppy disks to a new, more stable carrier and still use them (see also the report on the migration trial). At the National Library we chose to start with floppy disks because they seemed to be more at risk of loss than CDROMs; it's good that other people have approached the question of where to start differently. And we chose to migrate to CD-R because we believe it will be supported in the Library for a few years to come.
Although we were reasonably happy with the results of our trial, we know it is only a starting point, and doesn't tell us very much about later migration across technologies - we have a bridge across media instability, not a parting of the sea with a prospect of the promised land on the other side.
What's Happening Elsewhere?
We have also looked at what people elsewhere have been doing. Our survey of State libraries in Australia, and a survey of overseas activity, both suggest to us that no-one has found a magic wand to wave over these problems. The most hopeful sign is the interest expressed in working together to find feasible options. We know that some State libraries have taken some steps; we know that the German and Dutch National libraries are embarking on deposit and research programs that they hope will lead to preservation; and in the past few weeks the Society for Motion Picture and Television Engineers announced a working group to look at a Universal Preservation Format for electronic media. We want to be in touch with all of these developments and others.
Potential Approaches?
In the context of that level of interest and commitment, it was tremendously useful to ask publishers what options they could see. I want to finish - and lead into our discussion session - by reporting and reflecting briefly on the kinds of potential solutions, or building blocks for solutions, that came from our survey:
- one perspective is that there is no problem - CDs will last for at least 50 years, so they look after their own preservation
- another perspective is that the information is not worth preserving for more than its period of commercial availability - again, there is no problem
- there is no solution - they can't be preserved, so just accept that fact and don't waste time and money trying to preserve them
- some people said libraries should migrate publications to new platforms
- some people said publishers should migrate to new platforms, redesigning and republishing in new formats such as DVD or the Internet
- publishers should publish to standards
- equipment and software producers should build in backwards compatibility so older versions can be operated on new platforms
- publishers should use such common open formats that it gives forwards compatibility - their publications will be easy to operate with expected future systems
- libraries should negotiate with software developers, manufacturers, image banks, and so on to see what they could contribute
- publishers could provide libraries with customised copies that don't have copy protection
- publishers should not hard code access pathways to their products
- publishers could provide libraries with copies in a more preservable format
- publishers should provide sufficient technical information with their publications for libraries to know what they are dealing with
- publishers could go beyond that and provide detailed specifications of how their publication is put together and how it operates
- publishers could provide long term access to underlying data in a migratable form
- publishers should provide copyright clearance for preservation copying
- libraries should ensure that publications so deposited are kept secure and publishers' interests are protected
- libraries should use emulation software to emulate the required operating environment for obsolete material
We have gone some way with some of these. It seems obvious that solutions aren't going to fall into our lap - they will require research, probably at both the theoretical and practical levels - what might work, and how could we make it work? Research on things like human / computer interfaces is beyond most libraries; but there is a lot of more applied research that we could do that would help us move towards practical preservation action. We have limited resources, but a lot of commitment to find solutions. Are there ways of bringing our limited resources together to make whatever progress we can? Are there ways of bringing more resources to the task? what do we need to make it happen?
When we break to discuss these technical issues, I hope we can come up with a menu of matters requiring further work, and suggestions for how we could progress them - how we could enable some of these potential solutions to be developed, tested and put in place.
Discussion Points for Groups - Technical Issues Session
What technical issues look like they will need more action that we could identify/ take action on, either together or as sectors?
A. What is being created?
- A1. What is the likely history of technical issues we have to deal with over the next 10 years? Can you plot some likely scenarios in terms of publishing formats, preservation challenges, and how they might be met?
- A2. How can we work with software developers and others who are not publishers but who work at a deeper level? How do they influence our ability to preserve? What could they do to help?
- A3. What does publishing to standards mean in practical terms, and how will it help?
- A4. In practical terms, what achievable and useful contribution can backwards and forwards compatibility make?
B. Migration by libraries
- B1. How can libraries deal with the challenges in migrating publications to new platforms?
- B2. What impediments will there be to migrating CD-ROMs?
- B3. How could publishers provide deposit libraries with copies in a more preservable format?
- B4. Is it feasible for publishers to provide libraries with customised copies that don't have copy protection, if libraries manage them securely?
- B5. Would it be helpful to standardise the technical information supplied by publishers?
- B6. Do we need to keep an archive of software?
C. Migration by publishers
- C1. Are publishers likely to go back to their current publications and re-engineer them for other platforms? How hard is that likely to be?
- C2. What are the preservation implications of publications being migrated to the Internet?
D. Emulation
- D1. Would publishers be willing to provide detailed technical specifications of how their publications are put together and how they work?
- D2. Will the emulation path work as a widespread solution, long term?
E. Research
- E1. Are there ways of bringing our limited resources together to make progress with research?
- E2. What technical issues should be a priority for research, and how could they be pursued?
For further information contact:
- Colin Webb, Manager, Information Preservation
National Library of Australia
E-mail: cwebb@nla.gov.au
Telephone: +61 2 6262 1381
Facseimile: +61 2 6273 4535
Return to the National Consultative MeetingLast updated 6 January 1998
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |