Staff Papers
Stairways to Digital Heaven? Preserving Oral History Recordings at the National Library of Australia
Paper written by Colin Webb, Manager, Information Preservation.
The National Library of Australia has embarked on a program of digital conversion of its large and important Oral History collections. The Library has been using and investigating digital recording and editing facilities since 1991. Responding to the benefits offered by digital technology, and recognising the need to transfer from analogue to digital to maintain access, the Library initiated the purchase of a digital system in early 1996. The new system addresses the medium term requirements of digital audio archiving; its design also includes the possibility of automated migration to later digital archiving systems, including possible mass storage.
The system, managed by the Sound Preservation and Technical Services (SPATS) unit - part of the Library's Preservation Services function - uses a network of computer hard disk-based Digital Audio Workstations writing to recordable CD (CD-R), analogue reel-to-reel tape and analogue access cassettes. As procedures develop, it is likely that the analogue components of the current system will be dropped in favour of full digitisation. The size of the collection and the conversion task mean that even with the full digitisation of current procedures the collection is likely to contain analogue tapes for many years to come.
This paper provides some of the context for this project and the planning processes involved in the decision to digitise. The process is then described, with some discussion of advantages and disadvantages, and some issues for the future.
Background
Sound preservation activities in the Library are primarily undertaken against corporate objectives concerning the collections and services to the community:
- To collect, organise, preserve, promote and make universally accessible documentary materials ... of national significance in providing an understanding of Australia and Australians
- To provide efficient, timely information services from the Library's collections...
In this context, SPATS provides a comprehensive internal audio preservation service dealing mainly with the technologies and formats found in the Library's Oral History collections. The collections, currently containing about 30,000 hours of original recordings, cover a very wide range of notable Australians, social history and folklore projects from the late 1940s onwards. It is arguably the most important oral history collection in the country.
The great majority of items in the collection are unique, ie the Library holds the only copy. The collections were previously almost wholly carried on reel-to-reel and cassette tapes with cellulose acetate, PVC and polyester bases, with a variety of binders. There are examples of materials right across the 50 year history of magnetic tape recording. Many of the tapes are old, unstable and deteriorating; all require access and safety copies. To deal with this SPATS preservation-copies around 1,500 hours of recorded material per year. Until recently this was onto a stable analogue reel-to-reel format.
Facilities and skills have developed over a number of years to address the Library's objectives, by servicing these collections and serving its clients. Major capabilities in the preservation of audio recordings include:
- recording music and speech to archival standards;
- duplication of sound recordings to archival standards;
- physical and sonic restoration of deteriorated magnetic, tape-based sound carriers;
- storage and handling of sound recordings for long term preservation;
- training capabilities in the use of portable professional equipment;
- duplication of older recordings and the production of contemporary recordings for publication and presentation;
- systems design and maintenance.
From the mid-1980s the Library developed and adopted an internal set of technical standards for production of analogue originals to be added to the collection, and for preservation-, working- and access- duplicates. These form the basis of the Library's preservation policy for audio materials. They have also enabled effective quality control over outsourced preservation work.
From around 1992 most acquisitions were recorded directly to Digital Audio Tape (DAT) at 48 kHz, 16 bit linear. Though excellent for acquisition, and possessing very high reproduction quality at the time of recording, DAT is suspect as anything more than a short-term storage medium, and we have continued to rely on analogue tape for preservation and medium-term storage. However, the prospect of waning commercial support for professional analogue tape and related equipment forced us to start looking seriously at options for digital archiving in 1994.
The limitations of the standard analogue method for preserving audio material have always been apparent. It can only be carried out in real time; it introduces noise and distortion - some of which is irreversible; and the whole process will have to be repeated within 50 years or so, when the archival carrier itself will be in danger of degradation (as much of the earlier parts of the collection are today). To preserve the Library's digital (DAT) original by analogue copying means converting from digital to analogue, a process that introduces distortion, and recording onto a magnetic tape that introduces noise. This negates the advantage of digital audio recording. It is also extremely unlikely that an analogue tape player will be commercially available in 50 years.
These factors have caused a shift in electronic archives thinking. The sound archivist no longer focuses on extending the life of the carrier as far as possible. Instead, the archivist aims to select a digital standard that preserves the integrity of the recorded material, and a carrier whose archival life will exceed its projected obsolescence. As the recopying period is much shorter, the rate of the migration of the digital audio and the costs to do it are major issues.
When we started our investigations we expected to end up with a digital mass storage system - something that would hold all the data in a black box and do everything we could ever want. (We thought we were looking for that kind of solution - I believe it's called Digital Heaven.) The more we looked into it the less feasible the black box idea became.
It also became apparent that at this stage there is no single apparent successor to the analogue reel tape, and that for some time to come changing formats and shifting standards are likely to be the only predictable factor.
Technical Description of the Collection
When we considered converting the collection to digital format we had to face the fact that no-one knew very much about the size of the task. A detailed collection information system simply did not exist. This made it difficult to make good decisions about the scope of the digital conversion, and impossible to allocate reliable priorities. To provide effective management of the collection, and to guide the conversion process, a comprehensive, searchable information system was needed. The Library made a significant investment in designing such a database and collecting data, taking every tape off the shelves and recording basic information about its make up, age, intellectual and physical control, transcription status, access restrictions, and preservation status and needs. The database provides information to support detailed priority setting and planning of preservation, cataloguing, transcription and collection management work.
At a collection-wide level, for example, the database shows that we have about 29,000 original recordings, covering about 30,000 hours of original recorded sound. We now know how much, and what, material has been copied to preservation standards (28%), how much has been fully catalogued (32%), how much transcribed (25%). We also know which items are on unstable types of tapes and warrant a high priority for action.
Digital Audio Archiving
Investigating options
Library staff spent roughly two years in detailed investigation of the technical options for digitising the Oral History collections. This required consultation with local and overseas institutions, discussion with equipment suppliers and users, and in-house evaluation of options, advantages and disadvantages.
We came up with a list of at least 40 factors we wanted to take into account in making a decision. Critical criteria included things like the expected life of the technology and the support it was likely to get from suppliers; the amount of change that would be needed in our facilities, expertise and procedures; the down-time involved in getting a new system working efficiently and effectively; maintenance requirements for the new media; the availability of standards. And, of course, budget realities.
We were particularly interested in building in flexibility that would allow us to adapt to the inevitable development of new technologies and obsolescence of whatever system we would choose. The number of factors to consider, the different weighting of criteria, and the uncertain answers to many of our questions, all made decision-making complex. We found we could be more rigorous with some of the criteria and more intuitive with others. We recognised that it would be pointless to wait for the perfect, proven and stable technology to arrive, but that we would also be foolish to pioneer an attractive but untried and unsupported technology.
Bridging strategy
These investigations led to a bridging strategy that can be looked at from two perspectives - that of the broad technology chosen, and that of the specific program we set up to introduce the technology into the Library.
Broad strategy
When we looked at all the shifting technological options we recognised we would not be able to move directly to a mass storage system - digital heaven had not arrived, at least within budget realities. We decided we would be best to look for a widely used, flexible technology that would allow us to record digitally and give us better archiving potential than DAT, while allowing us to migrate to some future digital technology in a fairly automatic manner. That way we would at least be introducing a bridge to a future mass storage system.
We chose Recordable CD (CD-R) - as the most available technology to achieve this. CD-R is not the ultimate solution to digital archiving, and many other technologies offer some specific advantages over it. Overall, however, it appeared to be the broadly acceptable solution to the Library's requirements for the foreseeable future.
CD technology has become so widespread, in so many different forms, that it will take a very powerful technology to replace it. That will surely happen, but we believe we have maybe another 10-15 years before CD technology is superseded. That should be less than the life of the media itself, so we should not have the costs of continual re-copying that would apply to, say, DAT, during the life of the technology.
There have been many questions about the long term permanence of CDs, largely stemming from experiences with cheap, poor quality pressed CDs, or from early forms of CD-R that used dyes that were vulnerable to UV. CD-R manufacturers now offer much more stable dyes that should do the job for at least as long as we will need them.
For the Library's purposes, CD-R offers a number of technical advantages, including the following:
- CD stores digital audio in an easily retrieval and readily useable form. Access times per disk are quite fast. PQ codes (indexing points inserted into the recording) allow for fast access to specific information.
- In audio format (red book) there has been a massive market penetration, millions of CD players have been purchased. Consequently it is likely that the effects of technical change will be (relatively) gradual, and format support will be available for a time after its eventual demise.
- Audio is stored on CD-R in a 16 bit, 44.1 kHz linear form which gives a high enough sampling rate for high quality sound capture.
- CDs can be loaded into a jukebox and automated for unattended back up of data to other and future digital technologies. Jukeboxes would also provide for local online access.
- CD-R is a write-once technology and cannot be erased or overwritten once the TOC (table of contents) has been written.
There are some technical disadvantages:
- The write-once technology, unlike magnetic technology which can be overwritten, means that an easily made error in the audio/PQ processing produces an unusable CD which must be discarded.
- The organic dyes used as the recording surface are subject to degradation (as are other components of the CD at varying rates).
- It is heavily dependent on its inbuilt error correction to make it playable.
- Data transfer rates from CD-R to other digital systems are constrained by the capacity of CD players to play at a maximum of around 4x real time.
Specific strategy
The system we have chosen is basically a network of digital audio workstations (DAWs) with CD-R writing capabilities. The Sonic Solutions system we have installed consists of 3 networked ports in a hub of 8 possible ports, with direct recording or uploading to computer hard disk, management and editing of the recording, indexing, incorporating metadata, and so on, on-screen and then writing to CD, while simultaneously producing analogue reel to reel and access cassette copies.
Installation commenced in March this year and the system was fully productive by July. The Library's main contractor has also installed a similar system so that work sent to contract can be handled in the same way.
To minimise the failures associated with writing CD-Rs, to verify recording levels, and to ensure that the indexing points are placed correctly, the audio material is first assembled on a hard disk-based DAW. The CD is then written at real or double time on a SCSI writer in the background while the operator starts recording and assembling material for the next CD-R.
CD-R quality will be monitored with a CD tester and the relevant data relating to each disk will be stored. Longer term testing of the collection will allow degradation of the carriers to be tracked.
Archiving Procedures
The change to digital technology has resulted in the need to re-evaluate previous archiving standards, procedures and terminology. In the analogue domain the production of material for the collection followed procedures accepted generally in Australia and overseas. An original recording made for the collection was recorded on a stable analogue tape. If an old and potentially unstable recording was acquired, then a stable preservation duplicate was made of it. In either case, a working reel on a different variety of tape and an access cassette were made from the original. The process was predicated on extending the life of the carrier and minimising the generations of copies needed (in order to save money and to minimise the impact of signal degradation associated with analogue copying).
With digital technology we have changed the procedure slightly but significantly - and more significant changes are likely to follow. We still make an analogue preservation copy on a stable tape. This copy may be better thought of as a backup or safety copy against the possibility of something happening with the CD, or against CD technology turning out to be a failure. We will almost certainly stop using analogue tape within a few years and make only digital copies, but for now we are still using an analogue preservation master.
Recordings - either from the studio, from field recordings, or from existing analogue masters, are also written to CD-R. This CD-R copy does a number of things: it takes the place of the second analogue copy we previously made as a working copy; when it comes time to migrate the data to another digital format, we will use the CD-R as the data source, for higher speed migration, so it is in that way also a preservation copy. It may also be used as our access copy eventually, within a secure networked environment, although for now we are still making access cassettes.
Longer Term Perspectives
There are still uncertainties to be dealt with. For example:
- So far we have really only tried to address the getting, organising
and keeping aspects of the sound archiving process. And yet many of the most
profound changes are likely to be driven by evolution in the technology of
access, and changing expectations likely to flow from that. So far we have been
protected from having to cope with open network access by three things: the
small amount of audio material that we have had available in digital form, the
impact of the bandwidth required to transmit significant audio data, and access
restrictions. Removing some of these constraints over the next few years will
open up opportunities and challenges. It will also probably drive storage
technology in the direction of more automated mass storage systems to
facilitate rapid and non-linear retrieval.
- Mass storage systems that meet our requirements (including budget
requirements) will almost certainly develop - the trends in storage densities
and storage costs seem unmistakable. This will require management decisions as
great as those involved in this first digitisation stage - although hopefully
the conversion processes themselves will be easier. An effective archiving
strategy using audio CD formats should readily allow the automated transfer of
all digital material to the next storage medium.
- Finally, we must not forget that preservation of this material isn't a once and for all exercise. The volatility of the market will undoubtedly result in other digital technologies becoming available, with the eventual demise of the CD and its associated recording formats. We are assuming that there is no archival medium: there are only archiving procedures. The question is not how long a digital medium will last but how easily it will transfer to the next digital technology. It is likely that the Library will still be in the process of converting its analogue recordings when opportunities offered by new digital technology become available and the CD era is threatened with obsolescence.
These challenges in sound archiving are not happening in isolation from the rest of the Library. Developing long term strategy is becoming increasingly difficult, and we are all finding it necessary to build today on the best foundations we can establish, while keeping options open to help us adapt to what will be needed tomorrow.
