Staff Papers

Transfer from Analog to Digital: Selecting a System

This paper was presented to the 2nd National Preservation Office Conference: Multimedia Preservation - Capturing the Rainbow, in Brisbane, Queensland, 28 - 30 November 1995 by Colin Webb, Manager, Information Preservation


CONTENTS


Introduction

Let me start by saying that for a place like the National Library of Australia the major shift from analogue to digital is probably in managing digital information, rather than reformatting our existing collections into digital form. Unless we manage the digital information that is being created we will lose a lot of it. So when I am talking about digitising collections, I recognise that it isn't the Library's only focus.

I approach this subject from a preservation background, but one that has been looking for some time at preserving information from a range of collection formats, using a range of reformatting and preservation management techniques. For the moment I want to bypass the question of whether digitisation is a useful preservation process - although ultimately I won't be able to avoid it.

What my experience leads me to are some principles that I believe should guide us. I then want to talk about a few specific projects and how we have tried to apply principles like these.

I won't be presenting any technical information or comparing actual systems - it seemed too ambitious to take on what I want to talk about as well as the mass of available technologies in a relatively short paper.

What I have to say seems pretty obvious, but I hope it is still worth saying. It is very easy to believe there is one process called digitisation; very easy to assume that convergence means everything can be treated in the same way, as if there were a truck called "digitisation" on which we can throw all of our dead and dying technologies to be carted away and turned into something new and useful. It is easy to believe we can choose one of many desirable outcomes and ignore the processes we have to go through to reach it. One very desirable outcome is the ability to make information from all of our collections available to anyone anywhere, with rapid and thorough retrievability, and with seamless access so that data from all over the place can be put together in ways that are simply not possible with different analogue formats.

Although parts of that vision are achievable right now, as a whole it is a long way off, and there are many issues to resolve, many dilemmas along the way. (I keep thinking of a drawing by Lord Baden-Powell, of a boy scout paddling a small canoe between rocks towards a beautiful sunrise emblazoned with the word "happiness".)

This paper talks about some of the issues and dilemmas, and some of the approaches we are taking at the NLA.

Some Guiding Principles

So, let's start with some principles. Most of them should be familiar to anyone involved in preservation management, or library management, or public administration, just about anywhere.

  • we must be accountable - we are going to be spending lots of someone else's money, so we had better be sure that we can justify it, both in terms of what we digitise and how we do it (we also remain accountable for our existing collections and for what is going into them, so we have to do a good job of managing competing demands for our attention and resources).
  • there are a lot of things we don't know, so we had better remain flexible. It is hard to predict costs, technologies, storage capacities, bandwidths, what people will want to look at in a digital form, how they will want to use it, what the community will be willing to pay, what users will be willing to pay. (Already some of these look complicated. I hope I can convince you, if you need convincing, that we need to be mindful of all of these, but not debilitated by them.)
  • (taking up that thread) we need to be both cautious and decisive. We need to step forward, and there is generally no point waiting until everything is perfect. While it is tempting - and foolish - to rush in, it is also tempting to wait, discuss, agonise, until everything is stable and understood.
  • there are different ways of doing things, so we have to choose between alternatives. How do we choose? If I put accountability and flexibility together, I come up with pragmatism - by which I mean a commitment to solutions that work, and realism about what is likely to be possible, and about what may get in the way. Like other speakers, I am struck by the similarities and the differences between the old and the new. There is / was? an idea that everything would be different - everything had to be done differently because everything has changed. A truly pragmatic approach says: "we will apply what still seems applicable from the past - we will not assume that it must be wrong because we have done it that way before. If a model is useful, we will use it." It also says, "if it isn't useful, we won't use it, we will do it differently."
  • reflecting that pragmatism, I would say that we need to be experimental. We need to give ourselves room to undertake some cautious projects that bring expertise together, and address some clearly defined objectives. We should seek to discover information that we need on specific issues.

These are general principles that apply to digitisation projects as well as elsewhere. There are a couple of other principles of particular relevance to digitising collections of the National Library:

  • there is no one "system" applicable to all of our collections - at the most we may find a system that works well for a particular part of the collection. Even when different kinds of collection material have been digitised, there are likely to be differences in the way groups of primary users want to use them, and certainly differences in the storage capacity and network bandwidth needed. These are likely to be significant enough to keep them apart in the near future, even though highly selected bits will be brought together when someone wants to produce a multi-media package.
  • we also believe we need to be systematic, rather than digitising bits and pieces, and to be comprehensive rather than selective. (By "systematic" I don't mean we start at item one and work our way through to item 23 million, and by "comprehensive" I don't mean that we will digitise everything.) The Library serves a number of roles. It is an educator, and a creator of what I might call recreational or entertainment products. In those roles it has an interest in packaging information. But it also has fundamental roles as a very significant research resource - we would say a pre-eminent resource- and as a major archive of documentary heritage. Digitisation must support our business, and we think it can do that best if we digitise systematically rather than ad hoc.
  • and as a final starting principle, it's worth saying that digitising always costs more than you think, and raises more issues that you thought of (and in this it is very much like microfilming only more so!)

Applying the Principles - Some NLA Projects

How have we have tried to apply these principles? I will talk about a few projects selected to suit my purpose of trying to draw out some of the learning that has gone into and come out of those projects.

The ones I want to talk about are:

  • a project to digitise the Oral History collections;
  • a project to make digital copies of some of the original portraits in the Library's Pictorial collections;
  • the Australian Cooperative Digitisation Project that Ross Coleman [Collection Management Librarian, University of Sydney Library] has spoken about [in his paper The Ferguson, 1840-45, collection proposal: adopting the hybrid approach at the same conference]; and
  • the digitisation of current Australian serials.

This does not account for a number of important proposals regarding maps, manuscripts and microfilm. It doesn't even mention our publications, education and exhibitions programs. And it doesn't pretend to articulate the Library's main digitisation strategy, which is currently being finalised. There are significant larger issues associated with that digitisation strategy that are beyond the scope of this paper. Here I mainly want to use projects for the way they illuminate certain principles, not as case studies, and certainly not as a definitive statement of what the Library will seek to achieve in digitisation.

What do these projects illustrate? The first principle I mentioned was accountability. Trying to be accountable we recognised a kind of chain of understandings:

  • i. we can't digitise everything, at least not all at once, and maybe never. This is partly a matter of resources, but also a matter of priorities.
    ii. we have to start somewhere.
    iii. we need to base our priorities - our starting points - on two related things:
    • - areas of the collection where there is a demonstrated imperative to digitise or we will lose access ; and
      - things that clearly address the Library's strategic directions.

The first of these - demonstrated need - led us to the Oral History collections. Being a tape-based collection, we are dependent on supplies of professional quality analogue tape. When we became aware that suppliers were planning to stop making it in the next few years, we realised we had to look at a digital system or else lose our ability to preserve and give access to our collection. Having made a decision about priorities, we have looked at what benefits we could get from the project, what we can integrate into it. I will discuss the project in more detail a little later. For now I would ask you to note this theme in our priority setting - working out what needs to be done for a primary purpose, then looking for opportunities to get the maximum benefits out of it.

The other projects I mentioned are not being driven by the same operational necessity, although they are based on some strategic necessities.

The portraits exercise came out of a concern that we had a major documentary resource that is not as widely known as it should be, partly because it is difficult to provide safe and easy access to a collection of oil paintings, watercolours, prints, photographs and 3-D objects. The Library's support for and involvement in the National Portrait Gallery has led to increased collecting and increased demand for access to our portraits.

There were other advantages to this project. It addressed an identifiable component of the collections that are stored in one place, are already under consistent and good intellectual control, and where much of it is not encumbered with copyright restrictions. It was also a small enough project to be achievable in a short time frame, large enough to be significant, and it matched the available funds quite well.

Of course, there were also problems - quite a lot of the material is still covered by copyright; we had concerns about the technical and preservation difficulties in scanning original works of art; and we had been told that scanning technology for handling large colour images was still at a fairly immature stage of development.

We have sought to take advantage of these problems. We want to know about the copyright challenges - how artists will respond to requests to copy and give networked access to their works; what measures we can put in place to protect their intellectual property rights while providing the level of access we want; even just the administrative logistics. We intend to learn by dealing with these problems.

Likewise, we decided to take advantage of the fact that about half of the items were already photographed onto professional quality colour transparencies - so we decided to photograph the rest. This allows us to scan the trannies rather than the originals directly, adding a preservation-enhancing step to the process. We are interested in exploring the preservation implications of digitisation, but at the same time we are cautious about relying on it.

We are aware that responding to preservation concerns in this way adds considerably to the short term financial cost. This project will give us very useful benchmarking information on the image quality we can expect from this kind of pictorial material; it will also give us experience with one approach that uses an analogue technology as a preservation backup to a digital database.

Ross Coleman has already spoken at length about the Australian Cooperative Digitisation Project and the thought processes behind it. From the National Library's point of view, it is a project worth being part of for a number of reasons. It responds to a formally articulated demand from a primary user group; it lets us build on the experience we have in cooperative microfilming, testing and developing our collaborative procedures to make them applicable to digital reformatting. We will learn about the similarities and the differences, the costs and the surprises. We will also learn a lot about options for archiving large amounts of data, for providing networked access, and about some legal issues associated with ongoing ownership and responsibilities.

The project is also attractive because it involves historical print materials - a good contrast with our portraits project.

The project to scan articles from current Australian serials indexed under APAIS (the Australian Public Affairs Information Service) takes a very different approach. Whereas our retrospective digitisation efforts involve struggles with priorities, finding best copies, preparing material that is sometimes in poor condition, and so on, the APAIS scanning project is based on a much simpler concept of building a digital collection as we go. If we can digitise before the material is put on a shelf we can avoid many of the costs associated with retrospective projects.

From a preservation point of view this is going to be fascinating: unlike our other projects there will be no preservation intermediate copy - we will be relying on our ability and will to migrate the data we create, without recourse to microfilm or transparencies should our migration paths fail. This experience will help educate us as we address one of the major issues - can we rely on digital imaging for preservation? Of course, the project has not been set up to answer that question - in fact, its focus is access, access, and more access, but having created this growing database, we will be faced with preservation questions to which we have to find answers.

In all of these projects, and at each stage of them, we have tried to get the right balance between making a responsible and accountable decision based on thorough consideration of all the factors, on the one hand, and experiment, on the other. Learning is not an adequate justification for any of these projects, but we expect it to be a major outcome of all of them.

What is important to understand from this is that these projects were not chosen at random: they came from ordinary priority-setting processes -operating in slightly extraordinary circumstances - to pursue outcomes in line with the Library's strategic objectives.

It is important to put the main point of this paper in this context. When we talk about choosing a system we must go back to our priorities, and ask ourselves why we are doing this - what do we hope to get out of it? It is the answer to this question that we need to carry with us as we look at options, talk to systems designers, and put ourselves at the mercy of sales reps.

Some Issues to Consider

What are some of the issues to consider when we look at systems for digitising collections that we have identified as worth doing? how are we going to choose the best solution? I will try to mention a few issues that I think are worth pursuing.

I could sum it all up by talking about a cost/benefit analysis. This doesn't seem like a paradigm shift - we basically want to know whether the benefits are worth the costs, and to be able to compare the benefits and the costs we will wear with different systems. I think that process can be as formal and rigorous, or as informal and intuitive, as we need to be, but it should go beyond the obvious.

There are some obvious costs - what we pay to a scanning service provider, what we pay to buy or hire equipment and staff to do it ourselves, the cost of staff identifying and preparing the material. Then there are costs that don't get factored in if we are just thinking about the cost of scanning - things like building the access interface, storage and maintenance of the digital data we will create, the infra-structure that may be needed to make the data available. And there are obscure costs in managing the project, developing specifications, tendering, monitoring performance. It is easy for a few cents per page for scanning to turn into many dollars. And there is the cost of other activities foregone - what we could have done with the money, time, energy, and creativity that will go into the project. (As I have already said, this doesn't necessarily have to be done with rigour, but at least we need to get some gut-feeling for it.)

All of these costs are sounding like they add up to too much, so it is worth looking at the benefits. Hopefully they will be high. The benefits are usually not quantifiable in the way costs are, but we should be able to compare the benefits between systems. Because, like the costs, they will vary between systems. Again, we need to look at them seriously.

As well as the obvious benefits, such as improved access, we should try to make a realistic estimate of the preservation potential, the learning potential, what I might call the synergy potential. Just as some systems have hidden costs, there can also be hidden benefits that maybe should influence our choice of one system over another.

But benefits also have a way of looking more promising than they are. It's worth asking whether we are really going to get them, and whether they are really what we want?

Often we are making a huge investment when we digitise, not just in making the information available for remote and multiple users, but also by improving access within an item or collection. It represents a major investment in improved accessibility, in many ways doing the work the reader or a librarian or an archivist was once expected to do. This sort of investment should be made consciously, based on how the information will be used.

In choosing a system, there is a decision to be made about convergence. I have already said that I don't expect all digital data to be handled in the same way. At the same time, my digital Baden-Powell in his little canoe was paddling towards a glorious sunrise of happiness and integrated information access. Choosing a system involves making decisions about what can be converged, how easily, and when. Perhaps I have already answered my own question by talking about integrated information access. We probably aren't going to use the same system for capturing and digitising information from all of our collection areas, but we almost certainly will want to be able to make all of it available to a user, using the one access tool, without having to move from place to place. So we have to make some sophisticated judgments about exactly where we seek convergence, choosing the point that gives us the maximum benefits and the minimum pain.

In many cases deciding on a system also means deciding whether it will be a system or a collection of components. It is easy enough to walk through the door marked "do we have a system for you!" - easy to walk in and say "yes, I'll have one of those please". I find it encouraging that fewer people are doing that. When we open the door marked "components we can put together to meet your needs", we find we are blessed with the range of possibilities before us, though cursed with the responsibility of wider choice.

In most cases, what we need are systems - even if we are building from components we still need to be integrating them into a system that works as a whole. Systems like this need design, they need a unifying understanding of what has to be done. Probably like most institutions, we have to decide at how many levels we can make this system design work. We know we want a digital set up in our sound preservation area that works beautifully as a system; we have to decide on how far we need to design that into a grand concept of a Library-wide system. I am fairly sure these are questions of "when" rather than "if".

One thing we are blessed with is the growing ability and readiness of the market place to build things that talk to each other, that can be put together to make what we are calling a system. This is an encouraging sign!

I should like to think that this came about through preservation awareness, but it really has to do with winners and losers. Because we have a small number of very big winners the market has tended to be pulled into line with them. There is a big lesson in this for us when we are choosing digitising systems. In a very fluid market place we are trying to pick winners: the people whose technology will be so widespread that it will be superseded in ten years rather than two; the people who will stay in business and be able to offer us support for our system. There are many good reasons for choosing something that isn't likely to become an unserviceable white elephant, that was once a great innovative idea.

A question that worries me sometimes: should we digitise once for all? We have tended to assume that we should only digitise once; that we want a system that gives us the resolution - if I can use that term very broadly - to meet all of our anticipated needs. That approach surely makes sense, as it does with microfilming. But of course it comes with a cost - the more the resolution, the bigger the digital file, the more it costs to store, the longer it takes to copy, and all the rest. There is understandable pressure to set some sensible limits on resolution - let's limit it to the resolution of viewing equipment, or what people can see and hear, or what publishers need to make art print quality reproductions? "No, no," we say, "you don't understand: the time will come when we can view at much higher resolutions, when we can send massive uncompressed pictorial and sound files through networks with no delays; where will we be then, unless we have very high resolution images? And what about preservation - surely we must be capturing enough data to reproduce all of the original?" " Ah," the answer comes back, "I thought you weren't relying on this for preservation; I thought this was why we had to have microfilm as well. And, when the day comes when we can store and transmit very large files at such a low cost, will we not also be able to digitise at a much lower cost? Why not do it now for the purposes we have now, and accept that we will be able to do it again later for tomorrow's purposes?"

This is almost an open and shut case, but perhaps it is not completely shut. I know what my current approach would be, but maybe there is an interesting debate in some cases.

Now this debate hardly applies at all to the Oral History digitisation project. We don't think we will have the option of re-digitising from "the originals" in 50 years' time. We think this really is a "once and for all" job. (It also means that we haven't had to pay attention to the otherwise critical question of whether we can digitise without causing damage to the originals.)

Criteria for a Digital Sound Preservation System

I want to finish by showing you the detailed criteria we have been using to look at options for this project. As I do so, it may well look like an impossibly complex task. There are so many factors, some of which need more weight than others, so many spaces where it is hard - even impossible to fill in the gaps. It is complex! I can only say that we have found room to be more rigorous with some of these and more intuitive with others, but that we are trying to take all of them into account.

For every factor we ask ourselves these questions:

  • is it part of the system?
  • how is it going to be done?
  • what are the costs?
  • how easy will it be?
  • what flexibility does it give us?

And these are the factors we have been looking at:

  • equipment, software, accessories
  • installation, maintenance
  • expertise required, training provided
  • infra-structure needed
  • opportunities to make use of existing infra-structure, facilities, expertise (Any system is going to have much more chance of success if we can operate it without having to take on a team of technological whizkids to nurse it through)
  • transfer rates (how quickly and easily) for analogue to digital, and for digital to digital (we recognise that the conversion to digital is the start of a cycle of copying)
  • how much digital storage capacity the data takes up
  • what kind of storage media are required (or possible)
  • any special storage requirements for media
  • life of the media
  • reliability of the media - how much trouble it will cause us?
  • error detection and correction - one of the great advantages of digital systems is the ability to build in programs to deal with data degradation. We want to know how effective it is, how much it costs.
  • other maintenance procedures needed for the media. (Exercising tapes every couple of years might or might not be more than we could manage)
  • the expected life of the format and the technology. (We are assuming that there is no archival medium, only archival procedures. We want to work out where we are in the life cycle of the technologies we are looking at, and of course, when the technology does change, we want to be confident that we can migrate the data forwards. Choosing a ubiquitous technology probably increases our chances)
  • reliability, consistency, standards - are we entering an operating environment with established standards, or one where we take our chances and keep our fingers crossed?
  • security of data, copyright, access restrictions - how well will the system protect the data we are storing from unauthorised access, unapproved copying, and corruption?
  • ability to operate as a network - because we are a reasonably large operation with a number of studios and control rooms, we want to know about internal networking
  • ability to index data and ability to link information- we want to be able to do our indexing and intellectual control processes as efficiently as possible
  • support prognosis - we don't want to be left stranded on a system that does a good job most of the time, but that leaves us waiting for weeks before someone can fix up any problems. And we expect to need that support for quite a few years, not just for the first 12 months
  • ability to deliver to external networks - although we are being driven by a preservation necessity, this is where we start looking at how we go beyond our internal system. We are interested in the capability and costs of connecting to and delivering data through networks, how we will provide access, how easily we can give copies to users, how well our system can be part of a larger system within the Library and beyond.
  • convergence potential - how compatible we will be with other parts of the Library
  • compressibility? - the question of data compression comes up quite often. This affects storage capacities and how well we can give networked access - and the likely impact of our data on the other systems around the place. So far we don't believe we can use any compression algorithms for sound recordings that we are preserving.

On the other hand, we may be able to use them for access - it's one of those areas where we need to work out how users want to use our data, and where best to apply the technology we have.

Finally some factors that are easy to forget, factors that take the decision beyond technological questions:

  • the management implications of any proposed system - Is it going to give us users with different expectations? Is it going to cause or maybe overcome OH&S problems? Will it require changes in structures (I guess I am thinking about organisational structures, but it surely applies to building structures too!)? Is it going to require additional staffing, or even just different kinds of staff?

In the Commonwealth we need to take account of government policies on procurement, including the development of local industry, and guidelines on the use of outsourcing and in-house services.

We need to be very conscious of budget cycles, and available funding, asking ourselves: "is this realistic? It may be the best thing since sliced bread, but if we can't afford it when we need it, it's probably no use choosing it"

And last of all, we need to keep one eye on the possibilities for positioning and revenue generation. That's another whole story, though one that links very closely with this one.

Kevin Bradley, head of our Sound Preservation unit, also has another story - one that I hope has even closer links with this one. It is the more detailed technical story of some of what I have been talking about. He has presented it as a poster session entitled Choice, change and continuity: digital audio archiving and transfer.

This paper was presented to the 2nd National Preservation Office Conference: Multimedia Preservation - capturing the rainbow, in Brisbane, Queensland, 28 - 30 November 1995.


Author Autobiography

Following training in bookbinding and in paper conservation, Colin Webb has worked professionally in the preservation of archive and library collections since 1983, first with Australian Archives and since 1993 with the National Library of Australia, where he is the Manager of Information Preservation. His current areas of interest include the preservation of digital information and the preservation use of digital technologies, microfilming, sound preservation, and preservation management.

He can be contacted at:

Preservation Services, National Library of Australia
Canberra ACT 2600
Telephone: (06) 2621381
Facsimile: (06) 2571703
E-mail: cwebb@nla.gov.au

Last updated 10 June 1998

Back to the top