Staff Papers

Farewell my Floppy: a strategy for migration of digital information

Paper by Deborah Woodyard, Electronic Preservation, National Library of Australia

Abstract:

Practical strategies for long term access to publications on computer disk are being developed by the National Library of Australia. A survey of general collection material on disk was conducted and practical aspects of migration of floppy disks to CD-R have been tested and are reported in detail. Integral to the benefit of this work is cooperation with State Libraries to develop a national strategy for the preservation of electronic publications in physical formats. This topic was the subject of a special meeting with publishers and State Libraries on October 27 and 28, 1997.

Introduction

The National Library of Australia has been collecting Australian electronic publications in physical formats for some 15 years - the first floppy disks entered the collections in about 1983. There has been some active management of their acquisition, cataloguing, storage, and access over the years. A decision was made in the early 1990s to rationalise their storage by creating special runs for them, deferring further decisions on their long term preservation until later. Early in 1996 the Library decided to review the arrangements under which floppy disks and CD-ROMs are managed, especially with a view to long term accessibility.

Managing for long term accessibility is made up of many steps. In a complex library environment it is not simply a matter of focusing on an arbitrarily defined act called "preservation". Virtually everything that happens to this kind of material from the time it is created has a profound impact on its long term accessibility. Many of these influences affect the medium or the information directly - for example, the conditions in which it is stored, the way it is handled, the effectiveness of security arrangements. However, we also recognised that many factors having a trivial impact on carrier longevity may significantly influence our ability to manage the survival of the information. If we cannot find, operate and transfer this kind of material easily, it may in effect be useless even while remaining in a good physical condition.

The discussion paper entitled National Strategy for Provision of Access to Australian Electronic Publications (1) has confirmed the commitment of the National Library to providing long-term access to publications in electronic formats, both networked and in physical formats. It indicates that the Library will initiate and participate actively in national discussions and research in the management and preservation of these materials.

This paper will refer to two particular activities which illustrate the Library's commitment in the area of publications entering the collections as floppy disks and CD-ROMs. The first is a case study of the research undertaken at the NLA on the migration of material from floppy disks to CD-R. Secondly, the results of this research and other NLA activities formed the basis for a national meeting held on October 27 and 28, 1997 and I will be highlighting in my talk the steps taken at this meeting towards the development of a national management and preservation model for physical format electronic publications.

The Issues

Issues such as media instability and technological obsolescence are threatening to greatly reduce the longevity of these publications unless they are carefully managed.

Preservation includes all the actions taken to extend the useful life of these materials and particularly the information contained on them. Therefore control, storage, handling and security are involved, as well as copying and reformatting or migration. Overall this is a very complicated process and many issues must be addressed before libraries can effectively preserve this material, including:

Management Issues

  • preservation of electronic library materials requires knowledge across a range of disciplines and a great deal of cooperative work within and between institutions;
  • these new formats require a new approach and attitude towards their preservation - in particular their treatment must be proactive because the useable life cycle is much shorter than for most traditional library materials;
  • current copyright law is unclear regarding preservation roles and methods for electronic materials;
  • we need more detail to be recorded for effective preservation management - for example, we need to be able to identify publications in the collection based on their particular hardware, software or operating systems;
  • priorities for preservation must be determined on the assumption that we will be unable to preserve everything;
  • resource implications have not been clarified but could be significant.

Generic Preservation Issues

  • appropriate environmental conditions for storage need to be defined and provided;
  • special handling procedures and housing arrangements may be needed;
  • security issues arise in storage and when providing access.

Media Deterioration Issues

  • the dynamic nature of electronic media means the rate of deterioration is largely unpredictable;
  • the diversity of electronic media means managing the preservation response to the rate of deterioration is difficult.

Technological Obsolescence Issues

  • hardware or software required to operate particular publications may be unobtainable or unworkable;
  • some material may be effectively unusable because the necessary hardware and/or software is not available on site.

New Preservation Pathways

  • there are no standard procedures for preservation of these publications;
  • if migration of the information content to a new format is achievable, it will be an ongoing process that must be periodically repeated as the technology platform changes;
  • the integrity of information may be endangered in the migration process - it may not work the same way, and it may not look and feel the same, even if all of the data is successfully migrated;
  • emulation - the use of special software programs to recreate a required operating environment - is a largely unexplored avenue.

The Research

CD-ROMs are still a current and popular technology with relatively stable media, so the National Library of Australia has first focussed attention on the floppy disks in the collection. It has been agreed within the Library that copying and migration of information on computer disks is likely to be the most useful preservation method currently available to us. In order to examine how we might implement these strategies a series of trials will be conducted. The results of the first trial are described below.

The overall aim of this project was to migrate Australian publications that exist on floppy disks to CD-R (recordable compact disc).

It became clear that this was more an exercise in copying data from one medium to another, which is only a part of the process of migration, rather than completely addressing the problem of changing technology, such as the operating system or associated software dependencies. This has a valid purpose, considering the average life of a floppy disk is estimated at less than ten years.

Identifying and Locating Disks

The ILMS (Integrated Library Management System) was recognised as the most useful source of information for identifying and locating disks in the collections. Searches were made for material with:

  • a GMD (General Material Designation) of "computer file" (1023 records found)
  • "disk" in the collation field of items without the GMD "computer file" (363 records found)
  • "disc" in the collation field of items without the GMD "computer file" (146 records found)
  • a GMD of "interactive multimedia" (7 records found)

This produced a total of 1 439 records and appears to have captured a majority but not all items. The only serials located in this search were those where the primary material was a computer file. Print serials with accompanying disks were not located. Many of these records however were overseas publications which are generally outside our selection guidelines for preservation.

It has become apparent that more information than is currently available in catalogue records must be recorded for the successful preservation management of this material.

Planning has commenced on including the extra metadata and information required for these physical format electronic publications in the PANDORA database which is being created primarily for online publications. To help simplify the management of databases and metadata in the NLA it makes sense to utilise the same database for all electronic collection materials.

Selecting a Representative Sample

In order to determine the various complications that may be encountered in transferring publications to CD-R a representative sample of material had to be chosen for testing. A survey (2) was carried out of all Australian material located in the general collection to determine the range of material to be considered.

The survey covered 469 publications, 405 of which contained computer disks, and 64 of these were chosen as a representative sample of material containing floppy disks. The sample of publications were chosen to represent a number of variables:

  • operating system (eg DOS vs Mac OS),
  • hardware requirements (eg processor speed etc),
  • software requirements (eg Word 1.1 vs Word 6 etc),
  • disk size (5 1/4" vs 3 1/2"),
  • number of disks per title (eg 1 vs 6),
  • age (up to 12 yrs old),
  • proportion of publication on floppy disk (whole publication vs supplementary material vs set-up software for CD-ROM)

Copying

Basically the copying process devised was to transfer the data from the original floppy disks to a network drive, download the data to the hard drive of the PC containing disk writing software and then write to CD-R. The entire process (with the exception of file comparison) was carried out with hardware and software currently available in the Library.

A detailed description of the copying process is given below.

a. Select appropriate computer hardware and operating system, ie PC vs Mac, with correct size of floppy drive.

23 items could not be used at all due to a lack of the appropriate hardware or software. Another 1 item could not be used because a security tag had been stuck over the metal sliding cover on a 3 1/2" disk. This reduced the number of materials to continue through the rest of the trial to 42.

b.Virus check of original floppy disk

All 42 items were checked for viruses using the latest version of virus scanning software in the Library. No viruses were found.

c.Physical disk scan of original floppy or similar process to check for media deterioration

Scandisk was used on PC material, and Nortons Utilities Disk Doctor was used on Mac material to examine the disks for errors and deterioration. Only 2 PC disks had problems. One disk had errors which resulted in it no longer being operational so it was taken off the list for preservation, but the other had a second copy and could still be used. This reduced the number of titles for the remainder of the trial to 41.

d.Install and run from floppy

At this stage of testing one publication was discovered to have a blank disk although it was labelled as containing the appropriate files. Therefore it was also eliminated from the rest of the trial, leaving the final 40 publications.

When trying to run the files further difficulties experienced include:

  • 12 publications were found to not have the appropriate software or hardware available, and
  • inability to figure out how one other item was supposed to work, although the disk appears to contain information of considerable intellectual content which is worth preserving.

Twenty-seven items worked as they were supposed to when used from the original floppy disks.

e.Copy contents of floppy disk to network drive

This was successful for all 40 items.

f.Run comparison software to check for differences between original and copy

This was successful for all PC items. Mac items were not tested because appropriate software for the task could not be located.

g.Set as Read Only

This was tested for PCs only. It was not until after this part of the trials was completed that it was confirmed that "locking" a file on a Mac has the same effect as "read only" on a PC. This was mainly to test for the ability of the files to continue to function as required in a read only state as they would exist on the CD-R.

h.Install and run from network drive

Running files that had been set as read only did not cause any problems for these items.

All items continued to function after having been put on the network drive, although 4 items had to be copied back from the network to the root directory of a floppy in A: drive, and one of these required the renaming of the floppy disk, before these copies would work. This is because the expected location for the files, A: drive, has been hard-coded into the item.

The amount of time required to process material to this point in the copying procedure (ie steps a-h) was measured for 18 items, and took an average of 40 minutes per item.

i.Unset Read Only

Files that had been changed to read only status were returned to their original state so that they would still represent the original as faithfully as possible.

j.Organise titles in appropriate file structure using appropriate naming convention.

In an effort to apply widely accepted standards to promote longevity the data was arranged into a file structure that applied the ISO 9660 standard (3). This entailed using only A-Z (all capitals), 0-9 and the underscore in the 8 character plus 3 character extension naming convention for creating the directory structure for storage. The depth of the directories can reach a maximum of 8 levels.

The original file names for many publications however introduced illegal characters and extended names which could not be changed because their own programs would not recognise the new names and would cease to operate. Mac files in particular did not follow the basic ISO 9660 naming convention.

k.Create and include documents with appropriate details such as installation instructions for each item.

A general file was created in ASCII text to describe the contents of the CD-R, how files are arranged on it and refers the user to individual ASCII text files for instructions on how to use the individual items.

For this trial the individual ASCII text documents were created for only 4 items to test whether this is a successful method for carrying important information about each publication. In practice there would be one file each created for every item on the CD-R.

Apart from obvious details such as title, the ASCII files for the individual items include:

  • hardware required
  • software required
  • notes to assist installation
  • number of files
  • storage size (Mb)
  • original format and number of disks
  • existence of accompanying materials
  • content description (eg text, database, software)
  • any copyright statement from the publisher

These in future should also include date the disk(s) was copied to a new format and what that format is. This may seem obvious now, but if this data is migrated several times it may be desirable to have the history recorded.

l. Write to CD-R

This involved copying the files from the network drive to a computer set up with the software for creating CDs and a CD writer. During the process this software checked the file structure against the ISO 9660 file naming standard with Joliet extensions to allow for larger file names, eg for Windows 95. Four file names were rejected for containing illegal characters and had to be changed before the writing could continue. The rejected file names were all from Mac files which contained illegal characters according to the standard and extensions. This would most likely render the publications unusable.

Writing the 40 items to one CD-R (approximately 1,900 files, a total of around 164Mb) took 45 minutes.

m.Run comparison software to check for differences between original and CD-R copy.

Material for PCs compared favourably between the original and CD-R versions. Mac material could not be tested due to lack of comparison software.

n.Install and run from CD-R

When the CD-R was used to install and run the material the PC files did not appear to have any further problems, but it became apparent that the Mac files did not work.

The final result out of 40 items:

  • 22 items (all PC) were functional before and after copying to CD-R.
  • 5 items (all Mac) were functional before but not after copying to CD-R.
  • 12 items did not have the required software to function appropriately but were written to the CD-R, and some these items for PCs tested with other similar software to that required and tested by file comparison, still proved to have copied faithfully.
  • And one item that could not be tested by installation and running showed favourable file comparison results.

Further investigation has shown that Mac files need to be written to a CD-R using the HFS file system, which will require different CD writing software.

This also suggests that copying Mac and PC files separately onto separate CD-Rs would be simplest.

Research Conclusion

The testing has produced useful results. Not all the material in the sample was successfully migrated, however this was expected. We now have more informed views of the issues involved, the hardware and software required by the collection for use and whether it is available in the Library, and also the copying processes.

Further Action for Internal Consideration Flowing From this Trial

This trial highlighted the need for the Library to:

  • continue to investigate the best means of recording and managing the relevant metadata, including the development of the metadata archive associated with the PANDORA project;
  • develop Guidelines for selecting material to be preserved and for subsequent procedures.
  • determine what to do about currently held material which we are unable to operate or copy due to hardware and software deficiencies;
  • continue tests for Macintosh materials;
  • obtain the necessary support software for the transfer such as file comparison software for both PC and Mac;
  • copy existing floppy disk publications selected for preservation to CD-R as soon as possible;
  • establish ongoing copying procedures for floppy disks, probably on an annual batching basis (expected to be a short term commitment as the number of floppy disks entering the collection declines);
  • continue trials on other forms of migration. The feasibility and value of migration across operating systems and other major technology changes will remain unclear without further testing;
  • examine the possibility of emulation as an alternative to migration;
  • recognise that these methods are not a guarantee of preservation but appear to be necessary steps to increasing the period information remains accessible;
  • establish and maintain contact with others involved in similar research elsewhere in Australia and overseas.

Aims of the National Consultative Meeting in October 1997

The results of the trials just described were presented to the participants of the national meeting in October 1997. This meeting was coordinated by NIAC (the National Initiatives And Coordination Branch at the NLA) and involved the State Libraries, National Library and invited publishers in discussion of issues relating specifically to floppy disks and CD-ROMs in deposit libraries.

The aims of the meeting were to:

  • explore the methods used by publishers in producing, promoting and distributing their products;
  • reduce any potential or perceived threat to the publishing industry by libraries;
  • discuss the efficiency and effectiveness of current procedures for moving this category of material from the producers/publishers to National and State Libraries;
  • discuss legal issues;
  • improve cooperation and communication networks between all parties;
  • discuss methods of cataloguing, preservation and access;
  • explore methods of sharing the preservation responsibility nationally;
  • develop an action plan setting out a timetable, resource implications, and responsible parties.

More recent details can be obtained by viewing this paper at the VALA ‘98 website.

Addendum

Outcomes of the National Consultative Meeting 27-28 October 1997

Below is a summary of the main observations from the meeting:

  • There appears to be a trend for many publishers of physical format publications to move towards using the internet as the new medium for their publications.
  • There may be scope for the provision of customised copies of publications for libraries, which will be suitable for preservation purposes.
  • It is clear that preservation and access are inextricably linked. Long term access cannot be provided unless preservation is achieved.
  • Boundaries defining the processes involved in the management of the access and preservation of these materials are blurred. Practical experience gained through working with these publications at all levels will provide vital information for the preservation activities required. Essentially, providing continued access to these materials is conducting the preservation of them.
  • There are overlapping issues between the access and preservation of physical format and online materials, but the meeting required a focus on the physical formats.

The National Library will produce a more detailed position paper stating clearly defined aims and activities which will lead to the production of a draft national policy for electronic materials.

Report on the First National Consultative Meeting on the Management of Australian Physical format Digital Publications 27-28 October 1997

Overview of each library’s current strategic directions and digital initiatives

Summary

The answer has not been provided yet but these activities, the meeting and the migration trials, are all steps towards defining our roles and making progress in the preservation of electronic materials.

Notes

1National Strategy for Provision of Access to Australian Electronic Publications: A National Library of Australia Position Paper [online] (Canberra, Australia): National Library of Australia, 24 December 1996 [cited 10 October 1997].

2 WOODYARD Deborah, WEBB Colin. Physical format electronic publications in the National Library of Australia: report on a preservation survey [online] (Canberra, Australia): National Library of Australia, 9 September 1997 [cited 10 October 1997].

3 HARRIS James R. iso-9660 Simply Cds [online] [cited 10 October 1997]. Formerly available at: http://www.success.edu/~jharris/html/iso.html

Back to the top