Paper by Deborah Woodyard, Electronic Preservation, National Library of Australia. Presented at Information Online and On Disc '99, Darling Harbour, Sydney, 21 January 1999.
What must you do to preserve your electronic publications on disk? Act now. The volatile nature of computer disks means time is running out to safeguard these publications against extinction. The purpose of this paper is to provide guidance based on our practical experience to those who wish to manage their collections on computer disk. Appropriate materials, identification, metadata, copyright, preservation and access considerations are dealt with in practical terms to help you understand how we manage our collections, and how this can be applied to your own materials.
A few years ago the National Library recognised the need to address the preservation of the growing number of publications on computer disk in the collections. These items, which can be precisely labelled as physical format digital publications, presented many new issues and challenges which we are still in the process of analysing and researching. Our conclusions so far are based on a combination of experiments, experience and examination of international best practices. We are far from any ultimate solutions, if they exist at all, but now we feel we have learned enough to pass on some useful advice.
There are a number of reasons why we believe publications on disk deserve preservation attention despite the apparent difficulties that tempt many people to put it in the too hard basket. Firstly, they are just another type of publication, and we do not discriminate against format or we would be denying our duty to preserve documentary heritage. These publications are important as an example of the use of new technology for publishing, so they have added historical significance. These materials also provide added value to publications because they have special functionality not available through the use of paper, for example, key-word searching of the King James Bible, or the interactive multimedia educational publications such as Beefup published by the Queensland Department of Primary Industries.
Unfortunately it can be guaranteed that the computer disks in our collections will deteriorate and the hardware and software required to access the data on them will go out of date in a very short period compared to the life expectancy of paper publications. There is an added difficulty when our resources for dealing with these issues are obviously constrained. The result is that we may not be able to retrieve some information stored in electronic format in 5 years, and this will certainly be a problem in 50 years. So action needs to be taken sooner rather than later to reduce the impact of these changes.
We dont claim that the recommendations being presented here are the only possible answers, but we do believe our experiences may help many people to get a start on managing and preserving their collections. These are not long-term answers either. They should be considered first steps in what will need to be a continuing process.
We also make the assurance that the Library has committed to pursue long-term solutions as well. Strategies for this are set out in the proposed research agenda [1], which has initially been sent to State Libraries, but we welcome contributions from anyone interested.
Have you ever noticed that the definition of archiving or long term use varies between industries? Employees in cultural heritage industries would define these things as hundreds of years, but the Information Technology sector may believe 3 years is a very long time.
When some people are initially presented with the challenge of preserving digital information they say whats the problem? Its easy. Just copy it. This simple solution is fine while the hardware and software are current, but when the operating system is outdated, what good will an exact replica be, and how will it be copied again? The issue is technological obsolescence. Obviously we will have to look for other solutions.
Another common misunderstanding that is emerging is that we dont need to worry because these materials will be migrated to the Internet. Perhaps new publications will be more likely to be produced online rather than on disk, but it is not true that we can just forget material produced in off-line formats or the material already in our collections. It is important to note that these types of materials are essentially different. Apart from the common factor of needing a computer to use them, there are many different kinds and uses of computers that work on different programming languages. The Internet generally applies codes such as HTML, XML, SGML, CGI scripting, etc, however, many publications on disk are not created in this way. They use other languages that are designed for other purposes, such as C, C++, Visual Basic, and older publications that may be in COBOL and PASCAL.
It is more likely that a combination of the following options will eventually be employed:
Before we reach the stage where we can implement any of these complex solutions however there are simpler management tasks to perform which will help make this seemingly daunting process easier.
It is important to know the various formats that exist in your collection because they will deteriorate differently and require different treatment. The various types and sizes will also have very specific software and hardware requirements for access.
Differences in disks can be seen both physically and logically:
The disks we are most likely to find in our collections of publications can be divided into two main categories of physical construction: the magnetic and the optical disks.
Magnetic disks, commonly called Floppy disks - The common features of these disks are a magnetic coated flexible disk encased in a more or less square stiffer plastic casing. These are now most commonly found in two different sizes, 3.5, which is the still commonly used form, and also 5.25 which is nearing extinction. Different sized disk drives are obviously needed to use these disks.
Optical Discs - This refers to a range of discs read by laser light technology, eg. CD-ROM, CD-R, DVD-ROM.
CD-ROM - (Compact Disc - Read Only Memory) These discs are made from a stamped or moulded plastic disc coated with a reflective metallic layer. The disks being made now are quite stable and apart from physical damage such as warping or scratching, these disks are expected to last longer than the technology used to access them.
CD-R - (Compact Disc - Recordable) These discs are different in construction. They contain a dye layer that can be altered by light (ie. a laser). The dyes used vary considerably between disc brands and qualities. The use of dye which is designed to be altered means that these discs are less stable than CD-ROMs, however it is still most likely that the information on them is most endangered through hardware changes.
DVD-ROM (Digital Versatile Disc Read Only Memory) These new discs are also optical discs which contain layers of higher density information. Different laser arrangements are required to read these discs than CD-ROMs.
Disks can also be further classified by the operating system they are designed to operate in conjunction with, eg. IBM compatible PC versus Apple Macintosh.
Therefore to use and preserve access to the data on disks you must be able to provide each specific combination of software and hardware that is required.
Here are a few practical steps that you can take to manage the publications on disk you hold for the long term. They are based on the main threats to our disks, which are dealing with the deterioration of the media and inevitable changes in technology.
It is essential to be able to identify electronic materials in your collection that require preservation by name/number and by type. Therefore it is essential to record a minimum amount of information about each publication in a database which should provide searching capabilities for material by format and system requirements, eg search for all 3.5 floppy disks for a Macintosh that require Homestack software.
This important information, or metadata, should include the details necessary to use the data:
Usually this information can be found in a list on the packaging or in accompanying text. The amount of memory required might also be an issue for your library for the use of recent publications, but it is unlikely that future computers will have less memory available than is currently required for these publications, therefore it would not be considered essential metadata for preservation.
These details need to be recorded in an appropriate system for searching. It may be possible to record these details in the catalogue record for example, but they need to be recorded in a standard way, such as using a standard vocabulary in a common field. This system must also enable detailed searching of both publications entirely in computer format as well as publications where the disk is only a part of the publication, for example a disk in the back of a text.
We recommend following guidelines such as AACR2 and ISBD (ER): International Standard Bibliographic Description for Electronic Resources. However, the level of detail recorded for material for preservation is generally greater than their specifications, eg AACR2 use the term computer optical laser disk instead of CD-ROM or DVD-ROM, which is vital information for the use and preservation of these disks.
At the National Library over two years ago we started with a search of the catalogue but the information recorded in catalogue records did not provide enough details for preservation purposes, especially for disks accompanying texts. The General Notes Area (MARC 538) of a catalogue record has been suggested as a possibly appropriate place for recording these kinds of details in the future. Meanwhile we are storing the metadata on a Microsoft Excel spreadsheet using a standard notation to ensure accurate search results. This has involved a detailed survey to record the h/w and s/w requirements of our existing collection, and introducing new procedures to work with new items as they are brought into the collection.
In an ideal world deposit libraries might preserve everything, but operating in an environment of technical and resource constraints means that priorities must be set. Already there are enough indications of difficulty to recognise that it may not be possible to preserve access to everything, and that some options will be costly and/or ineffective for some materials. [2]
Preservation of digital materials will be a time consuming and resource intensive operation for the foreseeable future, because it will include activities to be repeated every time technology changes. This is important to consider when deciding which of the materials you possess warrant such treatment.
Assess the long-term value of the content of your materials to set priorities for preservation, and other issues might also influence the quantity of material you give high priority to. Issues you might consider include:
Record the decision to preserve or not preserve an item when it is decided so the work of making them wont have to be repeated, and so that your librarys intentions are known by relevant parties.
At the National Library we examined all these issues and policies have been developed on selection of physical format digital publications for acquisition and also on their selection for preservation [3]. We have also proposed that information on physical format digital publications that are currently being collected be shared between the National and State Libraries. Working with the State Libraries in the longer term we need to have a collaborative selective approach at a national level rather than at the institution level.
When you know what disks you have in your collection and which ones you want to preserve, you can take steps to avoid losing access to materials due to media deterioration and technological obsolescence.
We considered these threats to the disks in our collection and decided that physical deterioration of floppies was the most urgent challenge to address. The solution we chose also covered another pressing issue, the emerging threat of 5 ¼ disks becoming obsolete.
Floppy disks, with proper storage and handling, can last up to 10 years [4] but they may deteriorate much earlier and do not give you any indications when they are wearing out. It is necessary to act and make a backup before deterioration begins to occur or it may create an error in a critical sector of a disk and render it unusable. A small but growing number of publications on floppy disk have been found to have unrecoverable disk errors. It is basically too late to save these publications.
Due to the copyright restrictions on publications we cannot copy these materials to a network drive, a logical option possibly from a technological point of view. Agreements we have for the use of these materials specifically state the use of a stand-alone computer. We are effectively creating a copy for use and backup of these publications only and not creating new versions.
It would be possible to create back-ups on new floppy disks, ie refreshing, but it would be preferable to transfer the data to a media that will have a higher storage capacity and a greater life expectancy to reduce the frequency of back-up. CD-Rs are reported to have life expectancies over 70 years [5][6]&[7]. Making an allowance for inaccuracies in the manufacturers estimates, this is still an acceptable figure considering the expected length of use of the CD reading technology. They can also store roughly 400 times more data than a floppy disk.
After back up, access to the Librarys publications will be redirected to the CD-R copy and the original magnetic disks sealed to prevent further access.
As technology progresses and new media are commonly used further back-ups are expected to be made on the newest stable media available.
The next most pressing issue is dealing with changes in technology, technological obsolescence, the evolving software and hardware environment.
To access digital publications there are specific system requirements but we must be aware how they will be affected as technology changes. A program written for X may have difficulty running under Y for example. Software developers are not compelled to include backward compatibility.
Use the knowledge gathered in recording the preservation metadata (above) to determine what hardware and software are required by the whole collection. It is beneficial to establish whether you have the hardware and software required to provide access to your publications because this will affect your ability to preserve them.
Regularly examine these requirements and be aware of impending changes in hardware and software and incorporate these needs when planning IT changes. This will help identify precise problems that can be addressed, eg this process should be able to pick up when a piece of hardware or software is about to go out of use and allow action to be taken before its disappeared completely.
At the Library we have implemented a list of hardware and software maintained and it is now regularly examined and used to flag potential changes in technology and to signal that various types of hardware or software should not be disposed of while it is still required by the collections. Collection areas and Preservation Services maintain the list, and it is regularly presented to the Information Technology section for awareness of collection needs when planning changes of hardware and software in the Library.
When a change in technology has been identified, various methods may need to be examined in addition to those already mentioned. Changes in operating systems and software for example are far more complex.
Methods such as migration and emulation currently appear to be the practical options that may be employed. However there has been little practical experience to report of these methods being applied to publications.
The migration method generally relies on conversion software to change data from one format to another. This also depends on having the source code if you want to migrate a type of software and also having the appropriate conversion software. This is a very complex solution if there is no such software available.
Emulation involves software that emulates or pretends to be another type of software or operating system, eg a Commodore 64 emulator to be used on a PC. This may only be a short-term solution until the current operating system becomes out of date and we need an emulator for that as well. A series of emulators for emulators is likely to be unreliable.
Our options to find migration, emulation or other solutions will rely on cooperation and research. Communication with other libraries with similar situations and advice from the publishers could be a starting point for help. Computer societies and enthusiasts might be able to help transfer near obsolete materials. Commercial help and advice is available for text and images, but may be harder to find for more complex documents such as spreadsheets or software.
Currently in our experience at the Library we have employed migration to provide access to digital text documents in the Manuscript collection. A number of documents obtained by the Library were originally in proprietary formats we did not have the software to access. Through migration to a common format we can now view the documents using Microsoft Word, the standard word processing software in the Library. This is an issue we hope to explore further through the proposed research agenda [8] including the desirability of preserving these documents in a printed text format as opposed to digital storage formats.
We are committed to communication and cooperative work with other institutions internationally to find more of these solutions, as stated in the paper Developing national collections of electronic publications: issues to be considered and recommendations for future collaborative actions [9]. We have been and will continue to be active in international forums such as the 6th DELOS workshop on "Preservation of Digital Information", and communicate with relevant projects such as NEDLIB and CEDARS, and other organisations such as the national institutions active in digital preservation in Europe and North America.
When you have material in a viable condition and a state of current system requirements there will be a period where the most basic preservation measures should be in place to assist the plans for long-term use.
A basic need in any preservation plan is to provide appropriate storage and environmental conditions. A stable environment, such as air-conditioning or an insulated storage area, is ideal (ie storage free of large fluctuations in temperature and relative humidity) A cool dry environment slows down the processes of physical deterioration, which will extend the useful life of an item.
Reduce exposure of magnetic media to magnetic fields, for example do not store on motorised compactus, or expose to electronic equipment in close proximity.
Limit light exposure by turning lights off or boxing material.
Security should also be considered. Procedures for access should limit chances material could be stolen or corrupted. Security boxing could be used to protect material in storage. Sticky labels should not be placed directly on CDs due to increasing the chances of delamination and corrosion.
Do not bend any types of disks and avoid the use hard tipped writing implements on them.
In our experience at the Library air-conditioning at levels of human comfort appears to be suitable. Cool storage is used for some special, high priority back-ups. It should be recognised that technological change is most likely to cause problems with disks before their own deterioration becomes a serious problem, especially with media such as CD-ROM, which has been estimated to last longer than 100 years [10].
Part of the National Library's strategy for security and to reduce exposure to magnetic fields is the use a separate storage location for most material. Another strategy used was placing disks in clear, lockable cases for security but this proprietary solution has proven too expensive and will only be continued for material in publicly accessible areas. The separate storage run is soon to be moved to a locked room as an alternative security arrangement.
The intention of providing these suggestions is not to give a definitive answer, but to supply a basic guide for beginning the process of managing and preserving physical format digital publications.
It should also be understood that this is just the beginning of the process. Digital preservation will require an ongoing commitment to repeat many of these steps if we are to keep up with changes in technology.
[1] National Library of Australia, A Draft Research Agenda for the Preservation of Physical Format Digital Publications National Library of Australia, (1998-08-31) http://www.nla.gov.au/policy/rsagenda.html
[2] As above
[3] Policies available from "Collections and Collection Development Policy " http://www.nla.gov.au/policy/colldevt.html
[4] National Media Lab/National Technology Alliance, Questions and Answers, http://www.nml.org/MediaStability/QuestionsAndAnswers/index.html (accessed 1998-09-29)
[5] Longevity (accessed 1998-10-06) http://www.maxell.com/cdr/osta/long.htm
[6] Lifetime of Kodak CD-R media (accessed 1998-10-06) http://www.cd-info.com/CDIC/Technology/CD-R/Media/Kodak.html
[7] TDK CD-R Technology (accessed 1998-10-06) http://ww.cd-info.com/CDIC/Technology/CD-R/Media/TDK.html
[8] see [1] above
[9] National Library of Australia, Developing national collections of electronic publications: issues to be considered and recommendations for future collaborative actions (1998-08-20) http://www.nla.gov.au/nla/staffpaper/int_issu.html
[10] Dr John W.C. Van Bogart, National Media Lab/National Technology Alliance, Archival Stability of Digital Storage Media, http://www.nml.org/Publications/Technical Finalreport1994/11_arch_stability.html July 1994 (accessed 1998-09-29)
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |