Preserving the very recent

Preserving the very recent
New media does not mean longer lasting
8 July 2015

I was introduced to the world of digital preservation when I was a youngster. Sitting in my high school IT class, there was a video we were asked to watch which told the history of computing. It had all sorts of wonderful information about the early days—the 1940s—when small furry creatures got into the computer hard drives and were coined computer bugs. 

The moth discovered by Grace Hopper in the Harvard Mark II Relay #70—often cited as the first ever computer bug, 9 September 1945, image courtesy of Wikimedia 

The video also taught us of the early days of NASA and the Apollo program. This stroked my interest as I was (and still am) a pretty big space nerd. The nice man in the video showed us a big old bit of tape the size of the average worker’s desk here at the NLA.

"This is some of the data from the NASA Apollo program, when it was shut down in the early 70s they had no time to preserve or transfer any of this data as the department’s budget was cut so quick, it was all just put in a series of warehouses for later archival storage which never properly happened." [They were AMPEX FR900 tapes, each drive was about 6 feet (1.8 m) tall, 3 feet (0.9 m) wide, as deep as a refrigerator, weighed 600 pounds (270 kg)]

This was when I realised that digital preservation was a really big issue for humanity. Fortunately, thanks to recent efforts, we have been able to recover this NASA data, but for many years it was thought to be lost completely. Read more 

Those who grew up in the 1980s or early 1990s will also remember floppy discs. I had the privilege of owning an early computer which used such a floppy disc system. It was a Mac II. To me it seemed pretty cool. Much like being able to watch people play Angry Birds on YouTube seems pretty cool to my five year old son.

Those “floppy” discs that came in eight inch, five and a quarter inch and three and a half inch were the most widely commercially available, image courtesy of Wikimedia

However, within a few years the floppy had become obsolete and we were all doing things on CDs. Those floppies were still lying around, full of files that people had placed on them, to share, to give. The issue was now almost no one had the capacity to read them. How long before the same happens to CDs?

There are many other examples. Commercial forces have seen the death of HD-DVD, Jazz and Zip drives, Beta Videocassettes and quite a number of CD-like imitations. Everyone who has used floppies or even modern USBs know all media carriers have a shelf life. How long do storage media keep their data? Ten years? Twenty years? What happens then?

In the National Library’s collection there are literally thousands of CDs and DVDs, as well as floppy discs of various sizes, audio-cassettes, and even stranger media. If the CD's shelf life is about ten years in even the best conditions of low humidity and exposure, audio-cassette or floppy maybe a few years more, what do we lose as a country if this data goes? How do we fix this problem?

Enter the National Library’s Digital Preservation Program

This program has the ambitious aim of preserving every last born digital item in our collection by taking the data off the soon to be obsolete media and placing it in a digital preservation system for safe keeping, and access in the future.

As a part of this overall aim, I was given the task of preserving a large number of legacy materials that had been selected for priority digital preservation according to our collection development policy. I used a system developed in-house at the National Library called Prometheus, which transfers the data from physical carriers via a customised mini-jukebox attached to my workstation, to the Library's digital mass storage system. To begin each job, I entered information about the item being preserved (called metadata) such as the catalogue record number, copyright status, any notes about the item, parts or issues (for serials), carrier identification information and a job name. Then pressed the magic button and Prometheus swung into action and began extracting the digital files and creating a directory structure for them.

I used an excel file to manage the progress of the project, including keeping statistics on the format of the items preserved. As no project exists in isolation, I used a very simple language for the excel data that kept a track of the media as I moved them through this process. If the project had to be paused for any reason, another person could understand exactly what was going on and where I was up to. Communication and leaving a detailed record of what has happened is really quite critical for future tracking of such projects.

Of course, there were some issues. Sometimes an item had become totally unrecoverable. The data was gone or completely corrupted. This was very disappointing as it meant that someone’s intellectual efforts had been lost from the national collection. It served to highlight the importance of undertaking this preservation task now. How many more would have been lost in the coming years? Thankfully we won’t have to know.

Currently, for a number of reasons, we can’t deliver the preserved digital content. However you can see which items have been preserved in the Library's catalogue. If a record has a holding of “Digital preservation master” this means that the data from the original item has been extracted and safely preserved. For example, you can see one here on the record for my favourite thing I have ever catalogued Guitar: the Australian journey.

At some point in the future, musicologist, researcher, guitar enthusiast or just a curious member of the public will be able to listen to the Audio-CD that is preserved here

Of course my project isn’t the end of the tale for these files. Now that we have the data stored on our servers, the files have to be maintained and perhaps migrated to new formats so they can be accessed via the digital devices of the future. That is the challenging task for our digital preservation staff and their colleagues throughout the world.

I am really pleased to being a part of this project and to be able to know what I have done to help keep our nation’s digital heritage for the future.

Comment: 
Great article Anne, fascinating. I have a personal interest in data preservation. I've done work in Local History at the library digitising newspapers, photos off microfilm, broadleaf and some people in the workroom did a few maps too. I cringe when I go into the State Library of WA and see all the new data being stored on microfilm and the enormous readers. I'm interested in metadata creation and can't wait to do some training in the area. Another labourious undertaking has been digitising audio via Dictaphone>8mm aux jack>PC from music workshops. An invaluable optical media ripper I found is: http://www.roadkil.net/program.php?ProgramID=29 ; "Recovers files from disks with physical damage. Allows you to copy files from disks with problems such as bad sectors, scratches or that just give errors when reading data. The program will attempt to recover every readable piece of a file and put the pieces together. Using this method most types of files can be made useable even if some parts of the file were not recoverable in the end." What a massive subject to get involved in. I commend you and your colleagues!