National Library of Australia Staff Papers, 2001

Font Size:  Small  Medium  Large

Data Recovery and Providing Access to Digital Manuscripts

A paper presented by Deborah Woodyard, Digital Preservation, at the Information Online 2001 Conference held in Sydney, Australia, 16-18 January 2001

Contents

 

Abstract

Recently the National Library of Australia has embarked on a project to find electronic materials in its vast manuscript collections and apply new preservation methods to provide long term access to these materials. Aging media, as well as recent acquisitions, frequently appear in unreadable formats. Thanks to translation software these disks can now be read and transferred to new formats. However digital manuscripts raise many new issues, and methods of managing and providing access to this new format have had to be developed.


Introduction

Although the literal meaning of manuscript is 'handwritten', the National Library of Australia (NLA) manuscript collections cover all kinds of unpublished written records and many contain published and pictorial records as well. The kinds of records are extremely diverse: letters, diaries, notebooks, speeches, lectures, drafts of books and articles, research or reference files, cutting books, photographs, drawings, minute books, agenda papers, logbooks, financial records, maps and plans. (1)

The Manuscript Collection has vast collections which cover a broad range subjects including Federal politics and government, journalism, historical research, literature, music and the performing arts, and much more.

Most of the records are paper based, but computer disks and digital files are becoming more common as record creators move towards corresponding and creating documents in electronic formats. In response to this trend, concern for digital preservation has grown considerably in recent years.

We recognise that digital materials require more immediate attention than their paper equivalents if they are to be accessible in the long term, so new guidelines and procedures for these materials are being developed and applied much closer to the acquisition stage. However, many are already quite old when they reach us and a number of disks are likely to have found their way into the Manuscript Collection without specific attention to their needs. When these materials are finally encountered they often require data recovery methods for the provision of access.

At the National Library of Australia we have the ability to decipher many obsolete floppy disks and word processing formats with specialised software. If the disks have deteriorated however, we need to send the disks to a company dealing with the forensic recovery of data, similar to the approach described in the comprehensive British document "Digital Archaeology: Rescuing Neglected and Damaged Data Resources"(2).

Issues

There are two categories of significant issues which need to be considered when dealing with digital manuscript materials. The first category deals with the nature of digital manuscripts and the second refers to the procedures for management of digital manuscripts. Components of these categories include:

Digital Manuscripts:

  • Life expectancy
  • Accessing the content of digital media
  • Selecting storage media
  • Arrangement of files
  • Using the files
  • Selecting file format
  • Maintaining layout
  • Authenticity
  • Persistent storage location

Procedures:

  • Locating disks and files in a collection
  • At acquisition
  • Selection
  • Providing immediate access
  • Security and restricted access
  • Recording preservation metadata

These issues are explained below and include a description of how the NLA has responded to the challenges they present.

Life expectancy

How long a digital manuscript will survive depends on storage media, system requirements and the methods used to deal with many of the other issues listed above. The longevity of the storage media is important, but the ability to access the file types with the necessary hardware and software is also a determining factor. The predicted failure of either of these factors can strongly influence the priority of action taken to manage these materials. For example, media deterioration must be anticipated and addressed within an appropriate period or the first evidence of a problem may render the material unusable.

So far, digital manuscripts at the NLA have been received on floppy disks within a collection of papers. Unfortunately floppy disks have a limited life span (3), although it is difficult to compile exact figures of life expectancy due to commercial confidentiality and wide variety in brands and quality available. The inherent instability in this media has prompted the development of methods to transfer floppy disk material to other storage media as the highest priority task to perform.

Accessing the contents of digital media

A major difficulty is that manuscript material is often years old by the time it reaches a collection. This increases the risk that digital components may have already suffered media deterioration and may be in formats which have become less common or obsolete. For example, 8" disk drives are obsolete and access to 5.25" floppy disks is becoming increasingly difficult, and disks may be recorded in different formats, single density, double density, and a variety of arrangements, such as segmentation of data. Fortunately the floppy disks in the NLA collections so far are a standard size, either 3.5" or 5.25", and we still have a functioning 5.25" disk drive. However, the disks have been used to record data in a variety of common and proprietary arrangements. To assist us with reading these disks we have purchased specialised Intermedia software that can automatically analyse, recognise and read a very large variety of disks. This allows us to access the stored data and we can copy it to a more robust storage media.

Selecting storage media

Obviously, data should be stored in such a way that it will be accessible for as long as possible. Removable storage media such as CD-R (recordable compact disc) may be used but need to be monitored to ensure transfer of the data to a newer media before it is affected by deterioration or obsolescence. Networked or online storage is another option provided that regular back ups are made and storage space is available. There also needs to be an agreement with network administrators not to delete material on the basis that it has not been accessed, which is a common criteria for administration of some IT systems. Networked material also needs to be migrated when systems are replaced.

NLA digital manuscripts are currently transferred to general networked storage, however it will become part of a specialised Library system currently being developed called the Digital Services Project (DSP) (4). The DSP includes a Digital Object Storage System component to store all the digital content in the Library's collections.

Arrangement of files

The arrangement or order of paper manuscripts created by the author is often considered to be significant and may be preserved or recorded. This may also apply to digital manuscripts, including directory structure and naming conventions that have been used for the files.

We recognise that arrangement and naming of files may be difficult to maintain when migrating or translating files. Part of the NLA preservation strategy for digital materials is to keep an exact copy of the original data for future reference which will maintain the original structure and naming conventions.

Using the files

An inhibiting factor for use of the files is the frequent lack of information about the file formats used and their software requirements. Digital manuscripts are frequently text documents or may be database files and they may have been created using any number of software packages. It can be difficult to access the files again without their original software.

The Intermedia software used by the NLA for reading a variety of disks also includes a facility to read a wide variety of text based files created in many old word processing and database systems. This software can also interpret what the data means and migrate it to a different format. It contains more than 180 translation tables for translating different types of word processing files, but also will allow you to create your own translation of any obscure format that it doesn't have a ready made translation table for.

Selecting file format

In the process of migrating text files to another format we need to carefully choose the target format. To make this decision several factors are considered:

  • The format should not be proprietary because when change occurs or the format is no longer supported you may have no ability to convert to another format or could be locked into a limited selection of expensive options.
  • It is best to choose a popular and widely adopted format. The greater the market share, the greater the market forces and likelihood for support of this format for as long as possible.
  • A format with an open source is ideal for future access. This means that you can get the specification for how the file is encoded. The benefit is that in the future, if you keep a copy of the specification (that you can read) you will be able to decode the contents of your digital files even without the original rendering software.

In the light of these guidelines, at the NLA we are currently transferring manuscript text files to Rich Text Format (RTF). This is a widely supported, open source format. We will also store a copy of the specification.

There have been many suggestions that text files should be stored in XML (extensible markup language). Using XML appears to be a suitable option for conversion of many similar documents or records in a database. However, the papers and documents in the Manuscript Collection are generally unique and creation of XML versions of these documents is not trivial due to the variation in structure and content.

Maintaining layout

It is important to be sympathetic to the structure and layout of many documents. It is reasonably simple to recover the text of many old files but for some types of documents there is meaning in more than the words. The layout of words on a page can be as significant as the text itself. Poetry is an excellent example of this. Another, is the layout of text in a script which is especially important for readability and comprehension, e.g. character names generally appear on the left, an indent for speech and another type of emphasis such as italics for stage directions.

For example: Below is an excerpt from a poem, "The Black Abacus" by Robyn Ravlich (5), which illustrates how important layout can be in literature. The piece has a completely different "feel" when display with correct layout.

Without layout:

All the burnt places hang out through the town & tell the poets meet in them Everyone is discovering art is made from abandoned things

With layout:

All the burnt places
hang out
through the town

& tell the poets
meet in them
Everyone

is discovering
art is made
from abandoned things

When translating a word processed document with Intermedia software we can see the file as a long string and layout markers are noticeable amongst the text, much like HTML tags. For example, if you want to make text bold you would have a "start bold" tag at the beginning and "end bold" tag at the end.
i.e. <b>This is bold</b> = = This is bold

Elements of a text document can be translated by a method of selecting a series of characters and choosing a replacement for them. For example, reading the data it may be possible to see that an unusual combination of characters regularly occurs where you would normally expect an end of line character, so you can tell the software to always replace this occurrence with a current end of line marker.

The example below is based on files from a significant Australian playwright which were recently translated at the NLA. We did not have the files in print and were unable to determine the exact file type. Therefore, we could not apply exact translation tables, and so we worked on the text as it appeared after using our specialised software to extract text from the obscure disk format.

Here is a sample textµµµµµµµµµµµµµwhich you might find onµµµµµµµµµµµµµa page with some ºbºoºlºdµµµµµµµµµµµµµand _u_n_d_e_r_l_i_n_e_d features.µµµµµµµµµµµµµ

The series of characters "µµµµµµµµµµµµµ" appears regularly and suggests it might be an end of line marker. When we replace the string of characters with an actual end of line marker the result is:

Here is a sample text
which you might find on
a page with some ºbºoºlºd
and _u_n_d_e_r_l_i_n_e_d features.

Now we can see the last two lines contain other markers which we might surmise are suggesting emphasis should be placed on these characters. While we cannot be absolutely certain that this interpretation is correct, the result appears logical.

Here is a sample text
which you might find on
a page with some bold
and underlined features.

Authenticity

Digital files can be easily changed without any indication that the file you can see has been edited. This raises the issue of proving the records we are preserving are originals and exist as the creator intended them. Proof of authenticity in digital records can be assisted by the use of special tools such as check sums, but this is a complicated topic which requires further investigation of the complex issues involved (6)(7).

Our aim is to keep an original version of all files, preserving their original structure, organisation and file names which may be significant. It would also be sensible to apply some kind of authenticity tool such as a check sum to these files in case there is doubt in the future over possible changes. A difficulty is that many files must be migrated to a different format for access purposes, which essentially changes the file and the check sum is no longer applicable. However, for preservation purposes we want to keep original versions of files in case alternative preservation paths are chosen in the future.

Original files will be copied and maybe even migrated to another format, and then they may be restructured and even renamed to make them easier to use. All of these changes need to be documented and recorded in the equivalent of a conservation file for the item to retrace it's history and which parts of the item belong to the original document or may have been altered post creation by the original authors.

Persistent storage location

The Library stores digital manuscripts in directories named by collection title and number corresponding to the paper component. The existence of the digital files is noted in the Finding aid created for that collection and the location is noted on the collection sheet. An issue to consider is how to record file locations. They should not be based on format as these will change over time. e.g. files copied to CD-R now may be transferred to DVD in the future, or files stored on the network drive G may be moved to another drive or a whole other system in the future. This will avoid the need to constantly revisit and update records.

Locating disks and files in a collection

The Library has been collecting manuscripts since about 1910 and it has one of the two largest collections in Australia. It currently occupies about 10 kilometres of shelving. There are about 1700 major collections and the total number of individual items probably exceeds 20 million. And lurking among those items are an unknown number of computer disks, of various ages, formats and content.

The first step is to identify and locate digital materials in the collection. When the collection is large it is impossible to look through every folder. At the NLA we are currently using existing knowledge and documentation to narrow down this task.

Manuscript collections unlikely to contain digital material can be eliminated by date. The early 1970s saw the introduction of 8" floppy disk drives. 5.25" drives appeared in 1976 for the first time and were commonly used until 1987, and 1980 saw the 3.5" disk introduced (8). We have not found manuscript or any other collection material on 8" floppies, so we may concentrate our search on collection material from the late 1970s onwards .

Knowledge of the collections possessed by the staff is also a useful guide. e.g. It has been observed that earlier collections containing disks are from organisations rather than individuals which is probably because they had greater resources and need for use of electronic media.

This still leaves us with a significant portion of the collection to examine.

Collection material is given a detailed record or finding aid when the collection consists of more than 3 boxes of material. Staff also have noticed that collections of less than 3 boxes (roughly 90% of the collection) are very unlikely to contain disks of significant content, therefore the finding aids are a good starting point for searching. Unfortunately description of electronic materials has not been consistent in the finding aids, therefore disks could easily be missed at this point. Using this logic, we have discovered 22 collections so far that are reported to contain computer disks.

New material that comes in is now systematically documented in a separate register.

At acquisition

It is extremely important to at least register digital materials when they arrive in a collection. The donors of material may also provide a valuable source of information about the disks at this point while they can still be contacted. Being able to easily relocate and access digital items is essential. It is also preferable to perform copying, transfer and back up tasks as soon as possible.

Manuscript staff add details of new digital manuscripts to a specific register to record the known file details and system requirements. The digital material is also assessed at this stage to establish whether these digital files are a high priority for preservation and then arrangements are made for copying them to the Manuscript area of the network drive.

Selection

Not all digital materials may warrant time consuming management for preservation. A series of collecting principles based on the overall manuscript collecting policy may be applied to determine what materials the NLA wishes to preserve. For example, at the NLA a printed version accompanying the disks may be sufficient and the digital version not preserved. However, consideration should be given to the value of maintaining documents in a digital format when they are likely to incur heavy use as electronic access will be convenient.

Providing immediate access

When providing access to digital materials there are a number of points to keep in mind. The most important is to avoid direct access to the original files to prevent loss, changes or corruption. The machine provided for to view the files will need the appropriate software and hardware for access. Care should be taken with the copying facilities provided, allowing only appropriate amounts (possibly none) of printing, copying to disk or sending online.

We provide access to the files currently on a stand alone PC in the Manuscript Reading Room. When a reader requests the files, a copy is made from the internal network drive to a floppy disk and loaded on the reading room machine. There are currently no printing facilities provided and the copy is deleted after use.

Security and restricted access

Storage should aim to be completely secure and tamper proof. As a security precaution files should also be set to read only to avoid accidental changes. At the NLA we are currently storing material on a network drive only accessible by Manuscript staff and also set file properties to read only.

Some collections have particularly sensitive materials and may be restricted or closed for a set period, e.g. 20 years, or even culturally restricted by gender. These restrictions must be carefully managed to ensure that they are guaranteed. And it is important assure IT staff that this material is required indefinitely although usage is low, otherwise it is possible an over-keen administrator will apply standard IT logic and delete files not recently used.

The DSP will handle access issues including rights management and access restrictions applicable to all digital materials stored and selected for preservation at the NLA.

Recording preservation metadata

Any good quality preservation strategy will insist on the necessity of recording item and treatment details. In managing digital collections and items the need for information or metadata that will support effective and efficient decision-making is even greater than with traditional collections - there is less opportunity to recognise and understand problems just by looking at the object. Often one can only recognise and manage the material itself through its associated information. There is also likely to be a much larger amount of material to be managed, with much shorter preservation cycles, so management processes need to be automated as much as possible, based on easily interpreted metadata.(9)

The NLA has invested in drafting its own model of preservation metadata (10), which is a statement of the information it believes will be needed to manage the preservation of its digital collections. This information includes details on technical description and system requirements of the material, actions taken on the material, and statements of preservation responsibility.


Future challenges

At the NLA we are also aware that file formats not discussed above will have an impact on our Manuscript Collection in the future. Digital materials such as email and copies of Internet home pages for example might appear in the near future. Email in particular are predicted to have especially important issues relating to selection, authenticity and restricted access.

As previously mentioned, the Library's DSP will store and manage these materials. If Online materials are selected they are likely to be handled in the same manner and system as our online publications in PANDORA. (11)


Conclusion

The Library has preferred to collect printed versions of digital manuscripts for a many years, but is now accepting the importance of materials which are more suitably retained in electronic formats. This requires close examination of the issues which arise out of this new format, and development of new procedures for the management, access and preservation of these materials. Data recovery of old formats is likely to continue to be an important step in providing access to digital manuscripts.

Introduction of the Digital Services Project, which the Library aims to complete in 2001, will also provide many benefits to the storage and management of digital collections at the NLA.


References

(1) National Library of Australia: Our Manuscript Collection
http://www.nla.gov.au/ms/mscoll.html

(2) Digital Archaeology: Rescuing Neglected and Damaged Data Resources
Seamus Ross; Ann Gow In: JISC/Publishers Association Working Party Papers and Reports
(Date Created: Feb 1999) (UK)
ISBN: 1 900508 51 6
http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf

(3) Media Use and Storage Life
Kozierok, Charles M.
August 26, 2000
http://www.pcguide.com/care/care/media.htm

(4) National Library of Australia: Digital Services Project
http://www.nla.gov.au/dsp/

(5) Modern Australian Poetry: An exhibition of material from the Rare Book Collection
Monash University Library
18 March to 10 June 1999
http://www.lib.monash.edu.au/hss/rare/xpoetcat.htm

(6) PADI: Authenticity
http://www.nla.gov.au/padi/topics/4.html

(7) Authenticity in a Digital Environment
Cullen, Charles; Hirtle, Peter; Levy, David; Lynch, Clifford; Rothenberg, Jeff
(Date Created: May 2000)
ISBN: 1-887334-77-7
http://www.clir.org/pubs/reports/pub92/pub92.pdf

(8) Floppy disk drive primer: A Brief History
Accurite Technologies Inc.
http://www.accurite.com/FloppyPrimer.html

(9) PADI: Preservation metadata
http://www.nla.gov.au/padi/topics/32.html

(10) Preservation Metadata for Digital Collections
Phillips, Margaret; Woodyard, Deborah; Bradley, Kevin; Webb, Colin
(Date Created: 15 Oct 1999) (Australia)
http://www.nla.gov.au/preserve/pmeta.html

(11) PANDORA, National Library of Australia
http://www.nla.gov.au/pandora/