Staff Papers
Limitations and pitfalls of digitisation
A paper by Colin Webb, Director of Preservation Services, National Library of Australia, for the Pacific History Association Conference, Canberra, June 2000
I'm feeling a little nervous about my role here this afternoon. I'm not an historian, but what makes me more nervous is the strong probability that what I am going to talk about - the limitations and pitfalls of digitisation - runs counter to a mood of excitement about digitisation.
Please be assured that I while I will sound negative, this is not the whole story. In fact, I'm often an enthusiastic advocate of digitisation, and I'm as pleased as anyone about the wonderful things that can be achieved by the combination of scanning technology, computing power, and modern telecommunications.
But in all the justified enthusiasm for digital access, I believe it is worth educating ourselves about some of the things that can go wrong, some of the disappointments that can occur, so we can use this great technology to its best advantage.
I'd like to talk about some of the potential problems with digitisation in three areas:
- from an access perspective
- from a preservation perspective
- from the perspective of managing the digitisation processes.
In most cases I will be talking about things we need to bear in mind to avoid pitfalls, rather than a series of horror stories. One of my basic messages is that you can get almost anything you want, if you're willing to pay for it and if you do what's necessary to get it.
1. Access perspective
As I've already indicated, I agree with most of the claims people make about digital access, although my first caution is to remember that I'm looking at copies of source material, not the material itself, and that often the copying process imposes changes in the way I perceive the material. Despite any appearances to the contrary, these are copies, reflecting the time and processes by which they were made.
The first big issue is legibility. While you can use digital techniques to reveal hidden text, look into the spines of bound volumes, and do all sorts of fancy things to improve legibility, these are specialised applications that are costly and require one-off handling.
Deliverability of images is one of the main issues that leads to compromise, especially with large format material with fine detail. It's hard to capture fine detail without building large digital files, and these take a long time to send down the wire, and a long time to print. Thankfully, our approaches to compressing and delivering such material are getting better all the time, but it's still common for users to be frustrated by delays in access, and by the necessity of moving around the page to read the text.
Digital images are often just that - images. To make the information they contain more accessible we often want to also create searchable text. And yet many source materials are problematic for OCR (optical character recognition) software. Again, it is improving all the time, but we shouldn't assume we can automatically convert handwritten, or even printed text with a variety of type faces in poor condition, into highly accurate searchable text.
One pitfall is to ignore the very large investment sometimes required in organising and describing material, and in designing user interfaces that work. It doesn't happen so much nowadays, but I often used to hear people say they had a bunch of unnamed photographs that they were "just going to put on the web" to make them accessible.
Finally, there are a host of issues to do with ownership, rights of access, and cultural sensitivities that really have to be addressed.
So, there are some issues that can interfere with access, even though access is what digitisation does most excellently.
2. Preservation perspective
As well as access, I often hear about the wonderful things digital imaging will achieve from a preservation point of view. This is my area of main expertise, so I take a close interest in such claims.
I've heard three broad claims to which I give some credence:
- the most modest is a claim that digital copies will allow us to put our precious originals away and divert most use to the copies, allowing us to protect our collections from damage.
- another expectation is that we can use digital copies as replacements for unstable originals, in the way we've been using microfilm for brittle books and newspapers.
- a third claim is that we can use digital copies as back-ups in case of loss to our originals, much as we use photographic copies of artworks now. We hope it won't happen, but if it did, at least we would have an accurate copy.
I've also heard some quite extravagant claims that digitisation technology could replace all other preservation approaches, offering tremendous savings to institutions. For the reasons that we need to be careful about the credible claims, I don't believe this last claim is at all credible.
While I think digitisation can help achieve some preservation ends, the question we have to ask ourselves in any project is not whether it can, but whether it will. This seems to depend on three things:
- getting suitable image quality for users to want to use the copy rather than the original,
- controlling the level of risk to the originals in the imaging process, and
- being able to provide and maintain access to the digital copy.
These are all problematical.
The issue about image quality hardly needs discussion, and yet it's one of those things easily lost when complications and unexpected costs start to appear. We need to keep our objectives clearly in mind. If we're happy to encourage people to come and look at our originals we may not need great image quality, but if we want to divert use away from the originals we need to provide images that meet most of their needs. (Interestingly, many institutions report a greatly increased use of their originals as a result of digitisation, rather than decreased use.)
And yet it is sometimes hard to define how much image quality is enough - how much resolution, how much information each pixel should carry, how far images should be manipulated. I've been involved in two projects that illustrated this. The Australian Cooperative Digitisation Project, 1840-1845, undertaken over the past 5 years by three major Australian libraries, made something like 70,000 images of mid-19th century Australian serials, all in bitonal black and white. The images are mostly okay - many people tell us how happy they are to have access to them - but we know that we could have achieved much better image quality using greyscale or even colour. Recently at NLA we copied an explorer's diary written in manuscript in very faint pencil. The very high resolution full colour images are superb, although still barely legible, because of the problems with the original. From this I conclude that suitable image quality is a relative thing, and needs to be decided for every project and possibly for every item where there is a lot of variation.
The second preservation issue concerns the affects of digitisation on the originals. I remember the early days of infatuation with microfilm as a preservation medium, when some projects basically destroyed rare original material in order to get good quality copies. I think we have a more sophisticated understanding today, and recognise that generally we want to get the best image we can without unduly harming our source documents. However, it may not turn out that way without thought, planning and vigilance. Understandably, many vendors believe they're being asked to maximise image quality at minimum cost, so they may guillotine the spines off books, or place single sheet material through sheet feeders, or expose sensitive material to unacceptable levels of light and heat. None of these needs to happen, but they require decisions on our part to ensure our originals get the treatment we want.
The third preservation issue - and the one that is probably the most difficult to see in the excitement of thinking about what digitisation can achieve - is the long term maintenance of the digital files.
We face a relentless threat of lost access. This is because of the instability of the carriers we use to store digital files, and the changing standards in the market place for all the things needed to provide access - all the hardware and layers of software needed to make this or that format run in the way it was meant to run. We have a much better idea of how to maintain analogue originals and analogue copies over long periods of time, than we have for digital copies.
Now, I'm not one of the doomsayers who believe we won't manage it. Because of the huge investments already made in digitisation I believe we will eventually work out how to keep our growing collections of digital files across long periods of rapid technological change. However, I also believe it may cost more - a lot more- than we want it to, and we don't yet know how to do it. I guess some people are happy enough to cross their fingers and hope a feasible solution will appear when they need it, but as a preservation manager I'm a little uncomfortable with that approach.
Much research and discussion tell us we have to build quite complex infrastructure to manage digital materials for long term access. Many of the costs are current costs, not ones that can be put off until access has been lost maybe 10 or 20 years down the track. Im often horrified to hear that people who can't afford other ways of maintaining their collections are being convinced that digitisation is the answer, when the net effect will almost certainly be to increase the overall preservation burden and preservation cost when the digital files are included in the equation.
So, from a preservation perspective, digitisation does offer some real pluses - but we have to do what's necessary to achieve them: they don't come automatically with the digitisation territory. And it offers some real preservation headaches as well.
3. Management perspective
If management is about achieving the most you can of what you want to achieve, from the resources you are able and willing to bring to it, then digitisation presents many classic management issues.
I want to propose three rules or principles that seem to apply:
- Digitisation projects are complex, and the bigger they are, the more collaborative, the more ambitious, the more experimental they are, the greater the complexity.
- Almost anything is achievable, at a cost - sometimes the cost is financial, sometimes it is in tying up expertise that might be needed elsewhere, sometimes it is in accepting compromises on other objectives.
- The cost is always more than you expect.
There probably are some projects where serendipity is okay, where it's worth taking what you get, but I assume that for most projects we want to set some objectives and try to achieve them.
This means that the complexity of the project demands appropriate management if things are not going to fall in the many holes that lie before us. My experience tells me:
- that I need to understand and communicate my objectives as well as I can
- that I need to put some dedicated staff resources into the project
- that I need to establish some good steering structures like a committee or a good person to sponsor and guide it
- that I need to develop excellent specifications based on the best advice I can get, and fully tested with the people who will do the work
- that I will still have to do a lot of quality control checking
- that I will need to use good project management tools
- that it may require complicated procurement processes to get the best service providers on the job, and
- that I may have to break my work into small batches so I can test my specifications and adjust them without jeopardising the whole project. We have learned the hard way that even minor changes in equipment, software, personnel and procedures between pilot projects and production projects can cause unexpected disasters.
Although they are often hidden, project management inputs are often very large - much larger than the (still considerable) costs of information capture.
Within the project, there are other issues. I've already mentioned the concept of production projects. For small projects that can be hand-crafted, such a concept may be irrelevant, but once you start looking at larger projects you probably want to achieve some kind of production model that minimises costs and gets the work done in a reasonable timeframe. In digitisation even production models tend to be slow, as things have to be adjusted to accommodate the needs of the materials. Unless you and your vendor have taken this into account, it creates a tension that shows up elsewhere in the project, especially if you are working to tight deadlines and a tight budget (and who isn't?).
There is also a big issue with expertise. Embarking on imaging projects usually involves a very steep learning curve. It's sometimes hard even to evaluate the "expert advice" we receive from vendors and other people. It would be ridiculous to expect everyone associated with digitisation projects to become experts on all aspects, but we do need to develop ways of assessing whether we're getting what we want. I think the answer lies partly in using cross-discipline teams to manage projects, so we can at least draw on a range of views and experience.
The final management pitfall I want to mention happens outside the digitisation process itself. Because these projects can be so complex and so costly, and take up so much management time, they can easily skew your other priorities. It's fairly common to find that you have put more energy into your digitisation program than you meant to, and that other programs have suffered.
So, this looks like a gloomy picture. I hope it doesn't put anyone off, because there are such good things to be achieved. We just need to do the things that are required to achieve them.
