Long-Term Management Issues in the Preservation of Electronic Information
Paper presented by Maggie Exon, School of Information and Library Studies, Curtin University of Technology at the 2nd National Preservation Office Conference: Multimedia Preservation - Capturing the Rainbow, in Brisbane, 28-30 November 1995.
- Introduction
- Digital Materials and Preservation
- Definitions of Permanence and Durability
- Changing Concerns about Permanence and Durability
- Benign Neglect
- The Need for 'Live' Information
- Selection
- Strategies for Long-Term Preservation
- Access and Preservation
- References
Introduction
This paper is deliberately discursive, historically-based and apparently, at least at first, little to do with the technical or managerial problems of dealing with electronic information. However, I hope you will bear with me. There are many people who argue that a world in which information is stored and disseminated electronically is so different from a paper-based world that comparisons between the two are necessarily invalid. Any attempts to draw conclusions about the way we should approach such information from a historical view of what has happened to information in the past makes them uncomfortable. I admit to feeling no such qualms. I find analogies between the two worlds are useful because they illuminate what we are finding difficult to come to terms with by dealing in the familiar. I hope you will see by the end of this paper that there is some merit in taking this approach.
It is also necessary to state that much of what I will say is not consistent with generally held beliefs in this area. It may be that I am overstating my case. Perhaps I do not believe it all myself. However, I am very strongly of the opinion that counter views are worth expressing, if only for the value they have in stimulating questioning of apparently unquestioned beliefs. I am also aware that I have not had the space to deal with all problems particularly those of the ownership of information and copyright. These may not, in any case, be relevant in the long-term if short-term use can be controlled in some way.
It really is necessary before going any further for me to define exactly what I mean by "long-term". A clue to my perspective on this might be given by the fact that I am in origin and perhaps still am a medieval historian. I have recently finished a piece of research into an aspect of the changeover from manuscript to print culture in the fifteenth to sixteenth centuries. My definition of long-term must therefore be at least several centuries. This contrasts with most people I know who are hooked on electronic information. To often, for them long-term seems to mean around ten years. Ten years coincides with the length of time many librarians and users see as the maximum useable life of much of the information we keep. For example, scientific serials are often only kept in active usage for this length of time. Only a few days before this conference I read a letter in a university magazine (Smith's weekly: 1995) in which an academic bewailed the fact the university library had rejected a run of scientific journals which he had offered them, even though the serial was not in the collection, because the run finished in 1980. Most of the commercial material produced in the formats covered by this conference may have even shorter lives as "current" information". All those who have collected non-textual material know that visual images may go out of date long before the information conveyed. Seventies flares and hairdos create hilarity and scepticism in users even when the information itself is still useful.
Once you are past this date you are really preserving to satisfy our instinct to know about the past. We wish to know how people lived, what they thought and the influences on them. Age here makes little difference. There is comfort in talking about information as if it gradually moves from being current to being of historical interest to being no longer needed so that you can speak of preserving material "until it is no longer required" but this is not realistic. As time moves on at one end, the past does not fall off at the other. Once you are past the most recent history of individual memories, the interest shown in an age does not depend on how long ago it was. In fact, it is sometimes difficult to know why certain civilisations catch the popular imagination and others do not. The problem of satisfying this need grows with it.
We cannot know the future, even in the short term. Much of the work reported in a conference like this derives from government and other initiatives to do with preserving Australia's cultural heritage. Many of the discussion papers which have been produced are explicitly and unashamedly nationalistic. But my idea of long-term preservation takes in the possibility of a time when there is no country called Australia. The history of this part of the world in the next few hundred years may be like the history of the United Kingdom in the last seven hundred years, where many national institutions, instruments of government and many parts of the built environment have had a unbroken and evolving history. But it could just as easily be like the history of the British Isles in the seven hundred years before that when disruption and discontinuity were apparent and a whole civilisation, that of the Roman Empire, disappeared or was changed out of all recognition. At the time of its fall, Roman rule in the British Isles had lasted twice as long as European settlement in Australia. The people of the time must have thought Roman rule was eternal.
Can anything survive such change? Of course it can. We know something of all previous civilisations on this planet. But our richness of knowing comes from a variety of evidence of which documentary evidence is one of the most important. It may be unfashionable to say this, but it is language in its richness and capability of abstraction which is the vital ingredient in our understanding of the past. Images are unfathomable without it. We need to ensure that a rich variety of evidence remains about the way we lived materially, thought intellectually and experienced artistically and spiritually long after we have gone.
Nevertheless our knowledge of past civilisation is incomplete, patchy and difficult to interpret. Much of that which has survived has survived through chance rather than deliberate selection. We therefore have to ask ourselves, does this matter? Could we cope if we had everything? Does our view of the world of the past suffer greatly from the fact that it is created from imperfect record the contents of which are almost entirely controlled by chance? The answer is probably not. We can have a view of the past and draw support for our place in the present from quite imperfect evidence. A mere sample of what might once have been is enough. I have estimated that only about one in a hundred manuscript books created in England in the medieval period survives to the present day. If they all survived would we know a hundred times as much about medieval book production?
One of the first vows we should perhaps make in dealing with the problems of the preservation of digital information is that we should lighten up and stop worrying so much. There may be relatively short-term concerns about archival evidence and legal responsibilities but long term it probably doesn't matter much as long as something survives.
Digital Materials and Preservation
Unfortunately, as we all know, digital materials do provide particular problems for preservation. There is a very real possibility that nothing created, stored and disseminated electronically will survive in the long term. The problem does need to be stated this dramatically. I have an unfailing sinking feeling whenever anybody links the concepts of digitisation and preservation. I have a profound and unchanging disbelief that these two concepts belong in any sense in the same world.
The crux of the problem is the management of a processes called variously migration, transfer and refreshment. Since these are processes they need to be managed and planned for. Can we rely on the continuation of any process which is necessary to ensure that information continues to be migrated in the long-term as defined here? The answer has to be an unequivocal no. We cannot guarantee that migration will continue to happen. All that we can guarantee is that at some unknowable time in the future it will cease to happen. In order to understand this process, it is necessary to look in detail at the well established preservation concepts of permanence and durability.
Definitions of Permanence and Durability
The concepts permanence and durability are well established in the literature of preservation. Permanence refers to those qualities and characteristics of an information carrier which enable it to resist deterioration through aging, assuming that it is not subject to severe environmental conditions over a period of time or a cataclysmic disaster. The definition of permanence implies that deterioration will take place whether or not the information carrier is handled and its information accessed. Durability refers to those qualities and characteristics which enable an information carrier to resist damage through the process of use.
Both of these concepts need some clarification if they are to be useful in the era of electronic information. Firstly, the definition of durability has to exclude intentional damage or destruction. No information carrier is totally invulnerable to war and vandalism and this is just as true of electronically-stored information as paper-based materials. All the carriers of electronic information can be destroyed and we have come to accept that no computer system is entirely invulnerable to hackers and virus makers. Durability is only a useful concept if it is applied to normal use patterns. It should be possible to ensure that all systems are structured in such a way that no user can accidentally corrupt the information stored. The problem with durability seems to be that we find difficulty reconciling it with a type of information which is specifically designed to be manipulated and changed by the end user. We continually emphasise the mutability of information in this new environment and then worry about capturing the changes. However, if we do this we may give in to feelings of despair. There is some merit in seeing our information stores as being in two parts or levels. One is a resource level in which change is difficult, the other a manipulation level where it is not. It should be easy to move information from the first to the second, but difficult to move new formulations of that information back to the resource level. In a sense the moving back would be the digital equivalent of publication with all the filters and quality control that this implies. What is not yet clear is quite who would be responsible for this filtering. Durability as a concept would only really be applicable to this resource level.
The digital age has led to a widening of the concept of permanence. There was a stage in which we worried mostly about the physical longevity of the information carrier itself because we are used to the idea that if the carrier of the information survives, then the information it carries will survive also. In fact, this has never been true. Information may be in an unknown and undecipherable language or script or, as is the case with some early account books, structured in such a way that it cannot be understood. We are now sensible enough to realise that we need to be concerned about the permanence of the information itself, rather than its carrier. It is now commonplace to assert that obsolescence of carriers will set in long before physical decay. Later I shall give a new formulation of the definition of permanence which I find particularly useful when dealing with the myriad new forms of information available.
Changing Concerns about Permanence and Durability
If a long view is taken of the history of information storage and survival, it is possible to discern a definite shift of concern from durability to permanence. Librarians and archivists through most of human history have worried most about the terrible effects users might have on their collections. Some collections in medieval libraries were chained to the shelves, which not only prevented them from being taken away, but also meant that they could only be read on the sloping desks provided. New readers at the Bodleian Library in Oxford still swear an oath, laid down by Sir Thomas Bodley himself in the early seventeenth century, to regulate their behaviour. It includes the injunction that they should "not bring into the Library nor kindle therein any fire or flame". Access restrictions were often applied. However, there was little concern about the permanence of the collections. Custodians assumed that the materials in their care, if left undisturbed on the shelves, would survive indefinitely. They have been shown to have been perfectly justified in this confidence. For example, the library of Salisbury Cathedral was catalogued by Patrick Young in the early seventeenth century. About 150 of the medieval manuscripts he described still exist in the same location, in much the same condition and probably largely untouched in all those centuries (Ker, 1964, p.171-176). Their content (mostly medieval theology in Latin) and their format (simple, unilluminated manuscripts) protected them because nobody wanted to consult them. They also survived because Salisbury, as a secular cathedral governed by a Dean and Chapter rather than one administered by a house of a monastic order, was largely unaffected by the English Reformation and the dissolution of the monasteries.
There was a change in the conditions of survival in the mid nineteenth-century but this change was not appreciated until well into the post-second world war period, when it was realised that most paper from the late nineteenth century onwards was not permanent. Information professionals now had to worry about both damage caused by use and deterioration. As research into the complexities of why modern paper deteriorates continued, the need for careful planning of storage environments intensified. It was no longer possible to believe that material left on shelves was safe from harm. However, the problems of damage through use could not be forgotten because materials which are inherently weaker need even more careful handling. Nevertheless, a clear shift in concerns towards the regulation of macro and micro environment in which materials are stored is apparent in the literature. The latest edition of Keeping archives (1993, p.90-91) devotes less than two pages to the minimisation of damage through use in a whole chapter on the regulation of the environment to extend the life of inherently non-permanent materials.
With electronically-stored information, the paradigm shift from concern about durability to concern about permanence has been completed. We may worry about hackers but we do not worry about genuine use. In fact, we revel, we positively boast when we can show an exponential growth in the use of the information services we provide. It requires a large shift in perception to realise that the best chance electronic information has of being preserved is that it should go on being used, regularly and continually. As soon as it is not used, it is in trouble. Although electronic information is not invulnerable to deliberate attack, as has been emphasised above, the ease with which electronically stored information can be copied, backed up and security controlled, means that it should be possible to recover from accidental damage completely with no loss of information quality. Problems in accessing such information tend to be dramatically obvious, leading to complaints from users and immediate maintenance to fix the problem. It is the users who will ensure that these problems are put right. This is very different from the situation with paper materials, where the deterioration is slow and gradual and users may not notice the problem until it is too late to deal with it quickly and effectively.
Benign Neglect
It was stated earlier that the vast majority of the medieval manuscripts which once existed have not survived. The reasons are various and include deliberate destruction and simple wearing out through use. However, many must have perished through simple neglect, such as being kept in damp basements. The irony is that the survival of many manuscripts can also be attributed to neglect; the kind of benign neglect which has kept them untouched and undamaged through being unused but has provided for them a reasonable environment.
I am increasingly finding a concept of benign neglect a useful one, because it neatly excludes deliberate damage and storage in obviously poor conditions. It also leads to a simple definition of permanence as being a measure of the ability of information and its carriers to survive benign neglect. Although it may seem harsh to accuse the world's major libraries and archives of dealing in benign neglect, that is in fact always been their main management tool in the treatment of their collections.
The emphasis on migration and transfer for electronic collections has developed precisely because we perceive that information stored digitally cannot survive benign neglect. In fact, this concept, so vital to the survival of parts of Salisbury Cathedral's library, no longer has any meaning. There is no possible benign neglect in the world of digital information. All neglect will be non-benign. This is the worry that has led to this conference and so many other meetings and projects around the world. With electronic information the scales are too heavily weighted in the direction of ultimate destruction.
Can we legislate against neglect or plan and manage to avoid neglect? Can we ensure against human frailty and weakness and organisational change and decay? The promise of the proponents of digitisation is that the ease of copying of electronically-stored data means that we need not worry about an exponential growth in the rate of technological change. Migration is easy. The trouble is that I cannot at the moment believe in information which manages its own migration without human intervention and human beings are fallible.
The Need for 'Live' Information
I have already said that information will survive best in the digital age if it continues to be used. The very worst thing we can do is to rely long-term on formats which can be placed on shelves in nice little boxes and then try to put in place procedures in which the collection is worked through systematically and continually like painting a bridge, where as soon as you get to the end, you start again at the beginning.
If we cannot merely leave materials on the shelf, what is the best way of storing electronic information? It seems clear that information which is kept "live" by being continually online has the best chance. Information which is not heavily used or has periods when it appears to be of little interest has its best chance of survival in "keeping company" with information that is being used. As the carrying capacity of servers increases, it will become less and less necessary to store information on off-line carriers, whatever format they may take in the future.
There is, of course, a danger in the increasing capacity of servers. We may see this as an opportunity to be unselective and keep a vast amount of ill-organised material. It is necessary to discuss the selection of materials for preservation before looking at a model for the long-term survival of at least part of the culture of our period.
Selection
What are we preserving information for?
This may seem a rather stupid question and one which is easily answered, but it is actually rather complicated. Long-term storage means that we are keeping information for historical purposes, because we believe that people in the future will want to access the information for the light it may throw on the past. I am deliberately emphasising this historical use because of more recent emphasis on short-term preservation particularly in the fields of archives and records management. There has been a concentration on legal, accountability and administrative reasons for keeping materials. Long-term preservation is different. There are basically two different questions which are asked of historical materials. There are very specific questions, where very specific records or publications are wanted. Then there are general questions in which we have to say that some general view of past societies can be gained using only a sample of the evidence that might once have existed. Historians are used to this idea. They know that evidence survives of only an infinitesimal proportion of happenings of the past. Most have disappeared into oblivion. Even of those events which left a trace in some kind of record, only those where the record has survived can be reconstructed, and this survival has been capricious, ruled by chance.
As I have already said, we are faced with a grave problem of perception caused by the promises of the new technology; that somehow, as long as we can solve the technical and managerial problems, we can easily save more of our documentary heritage than in the past. It is this that feeds our angst every time we realise that, as the new technology delivers more storage and dissemination capacity, the production of information overtakes it. In addition it has offered us a promise of dealing with all the decaying materials of the era of brittle paper. We expect to do so much. In truth, we might be wrong to try. Maybe we should just let chance have its way or set realistic goals for the planned survival of small amounts of materials.
We may need to make a distinction between materials which provide primary evidence for particular events and people and those which are merely indicative of the nature of times, fashions, beliefs, attitudes and ways of living. These do nor have to be comprehensively saved, in fact they cannot be comprehensively saved. A sample should be and preferably a well-chosen sample, but chance might do an equally good job. I would suspect the highly personalised home pages of some people on the Internet may not long be a fashionable phenomenon. Goodness knows we do not need to preserve all of these. But it would be a shame if at least a few did not remain to show people the somewhat strange juxtapositions of interests and beliefs that are so freely made available at the moment to anyone who will listen. But they are not like the diaries and commonplace books of he past. Benign neglect will not save them.
Nationalism and localism
It my be the case that we should reverse the trend towards nationalism in the responsibilities we feel towards information. The ational Preservation Office, as part of its early strategies, gave much thought to defining "national significance". This suits the tenor of out times and the growing interest within countries like Australia in our national heritage. However, it is also fits well with the notion of a world pooling of information through networking which allows each country and area top focus on its own production and rely on others for theirs. A long-term view would be that this is dangerous. We cannot predict what parts of the world will be subject to disruption and the destruction of information. We may deplore the imperial instincts and the cultural plunder which led to the transport to institutions like the British Museum of so much of the surviving artefacts of the ancient world. But the uncomfortable fact remains that if the sculptures of the Parthenon had remained in situ, they would not have survived. With electronic information, we do not need to worry about priceless originals. We do however have to make sure that we take responsibility for providing mirror sites, not merely to help access over world networks, but also to act as international backups.
Selection criteria
There is always much talk about the convergence of different kinds of information caused by the use of digital information. The disappearance of the concept of the published work and a lack of distinction between text, non-verbal communication and realia are two examples. However, it is possible to take this belief in convergence too far. It leads to confusion, a sense of helplessness and a lack of clarity in selection for preservation. Whatever forms of storage and data structures we use, there will remain a difference between the following kinds of information:
- primary records of events and transactions, research data and base level descriptive data;
- secondary interpretations of these at the scholarly level;
- creative literary and artistic work; and,
- any of the previous repacked for a transitory popular or educational market.
Years ago I asserted at a conference, in a somewhat peevish tirade against the overuse of the word information, that I did not see the day coming when a serious enquiry about the history, design and building of Westminster Bridge could be adequately answered by a librarian striking a pose and beginning to recite "Earth hath not anything to show more fair...." especially as Wordsworth's poem, despite the evidence of his title, is not even about Westminster Bridge. A deep-seated part of my consciousness still finds it difficult to conceive of a poem as being in any sense "information".
If we look at material which is created, stored, disseminated and used as digital information, we can still see the same distinctions applying. A very great deal of the commercial multimedia produced now comes firmly into my last category, that is repackaged information for a particular audience. It will be interesting for future generations for its expressed attitudes indicative of its time, for its design characteristics, in fact for all the same reasons that we find children's books, popular histories and so on of past ages fascinating. But a sample will do. In scholarly publishing, there will always be the need the recognise the importance of quality in scholarly discourse and put in place a system of quality control in this area. We know that we need to ensure the continuation of Newton's writings while vast quantities of poor and derivative scientific writing of the late seventeenth century is forgotten. The same applies, even more strongly, to the products of this 'publish or perish' era.
Record material has always been sampled both for current use and for archiving. We need to rigorously enforce the same retention and disposal schedules we have always used and still keep in mind such old saws as "fewer and better records".
We seem to have become embroiled in worries about all the material discussed above to the exclusion of what I believe is our biggest problem, that is the survival of creative endeavour. As artists and writers increasingly pursue the possibilities offered by the new technology, we may not be able to keep up. They will push the boundaries of the standardised formats we develop too far, and the problem of retaining evidence of the creative process will be the most difficult to achieve.
The implication of all this is that we need to retain relatively small, well-chosen collections of information, which represent just an indicative sample of an era, rather than a comprehensive capture of information in a particular category. We will need to extend to the whole of knowledge ideas of retention and disposal which are already used to reduce vast quantities of records into useable archives. With the dilution of the concept of published work, we need to get rid of the notion that we can save a copy of every work published. That will no longer possible. There is just too much.
Strategies for Long-Term Preservation
What are we to make of all these conflicting problems? How are we to cope with the fact that we have lost the rock on which our professions have relied, the saving grace of benign neglect?
The first and most important thing is to convince all those who make decisions which affect preservation that, just because something is technically possible, it will not necessarily happen. A task like information transfer or refreshment which needs to be repeated over and over again, will at some point fail to happen. This failure may take place a few years from now or a few hundred years but it will surely take place.
The second and more comforting is that, in order to keep as much faith with the future as the past did with us, we need only save some things from the wreckage and we may even hope to be a little more deliberate and organised in what we hope to save than in the past.
How can we do this? What information from the period 1995 to 2000 will still be around in 3000? Undoubtedly much of it will consist of a chance collection of materials in those formats which consist of accessible, real-world images on permanent carriers, such as archival-quality paper and microform; those carriers which can survive benign neglect and still be read without equipment or with the simplest lens/light source technology. There will also be a random collection of bits of digitised information which will be stored using the current technology, whatever that is, and which has survived by being continually incorporated into new work, in the same way that lost works of the ancient world survived by being quoted in newer works which have survived. However, if we plan well, there is some hope that more deliberate collections of material could survive. My ideas about the characteristics that such collections should have are in some ways directly contradictory to current accepted thinking.
Responsibility
In order to survive collections will be managed by large national institutions which will take total responsibility for their future. These institutions will not merely be controlling bodies providing centralised recording of decentralised collections. The information itself will reside on their own local systems. In this respect they will be the direct descendants of our current national libraries and archives in their responsibility for centralised collections. Such institutions have the best chance of remaining in existence over long periods of time or of being replaced with others with similar responsibilities. The importance of the information they hold will be assumed because of their status.
All the accepted strategies for collaboration in collection building and preservation in the short term are fine, and many institutions could contribute to these centralised collections. But the ultimate responsibility has to be localised, even though this may have dangers of reduced democratic control and a single viewpoint.
Content
The information will be highly selective and controlled in its content. Its importance will be so obvious that there will always be people who are prepared not to see it be lost. They will react to catastrophic events by trying to save the information, rather as a portion of classical scholarship was kept alive by Arab scholars and dark age monks or as early book collectors in England saved material from the wreck of the monasteries in the sixteenth century. The significance of the collection will be kept obvious by its clear emphasis on materials of significance undiluted by trivial items.
There will be clear and precise boundaries around the body of information with no "live" links with material stored elsewhere. There will be no reliance on networking to bring the information together as a virtual library. The problem of maintaining the links and ensuring they still work is too great. This does not mean that users will not be given world-wide access to the collection, but the links, indexes and databases they use to access the material will explicitly not be part of the information collection itself and the information will retain its useability without these extras.
Format
The information will be highly controlled in its format. The information will be software independent and standardised formats will be used which are as simple as is possible consistent with the information still being useable. This, of course, poses a problem with much commercial and non-commercial multimedia, in which complex relationships are set up between image, sound and text. However, the move towards standardised software and hardware formats has been very strong in recent times and I can see this continuing, leading to the possibilities of finding ways of translating these formats into relatively simple migratable formats.
Much of the "value-added" enhancement of information which is possible now will not be saved. The concentration will be on base-level material. Sophisticated navigation tools, indexing systems, complex linkages and display formats may reside in current software tools, with capabilities we cannot dream of. The constant reconstruction of methods of access may be the price we play for ensuring that information is still useable.
The information will be kept live and on-line at all times, thus restricting its bulk to the amount that can conveniently be dealt with in this way according to the technology of the time. It will not be placed on some off-line information carrier, except for backups to deal with disasters or breaches of security. The fact that some of the information is always currently being accessed will protect the material which is not currently being accessed. The process of transfer and refreshment should involve one operation in which all the material can be dealt with as one body. When new systems are introduced, some data is always transferred from the old system to the new. By making it as simple to transfer all the material as a subset of it, you restrict the temptation for the system change to be seen as a golden opportunity to dispose of material.
The information will be backed up by being held at mirror sites elsewhere in the world through international agreement, as well as being backed up on offline media. Thus one carrier of the material will be the organisation which has primary responsibility for it. The mirror sites will be designated backups and will also have clear responsibilities, of a distinctly more serious level that that of sites which mirror data mainly to improve access to it. In records management terms, the holder of the original collection holds the "record copy". The backup sites contain copies which also have record status. Other sites have mere "copies" which they can dispose of at will. Countries should not through misplaced nationalism try to keep information to themselves. When the tides of history turn against previously stable and peaceful areas of the world, their history can be saved elsewhere, just like the Elgin marbles.
Access and Preservation
In conclusion, I should like to reiterate my belief that there need be no tension between access and preservation as long as we can put in place suitable security systems to ensure the integrity of the information. It is through providing access world-wide that we keep our information live. Placing digital information carriers on shelves is the sure recipe for their destruction.
My greatest concern is that we may be trying to be too elaborate; the plethora of projects, organisations and competing systems is a worry. In particular, everything we are building now appears dependent on elaborate webs (world-wide or otherwise) of cooperation. Maybe the idea of localised, bounded collections has not quite yet had its day and the world's large collecting institutions should not so readily divest themselves of their selection and collection responsibilities in favour of vaguer notions of the coordination of devolved responsibilities.
References
Keeping archives (1993), editor, Judith Ellis. Port Melbourne, DW Thorpe.
Ker, NR (1964). Medieval libraries of Great Britain: a list of surviving books. 2nd ed. London, Royal Historical Society. (Royal Historical Society guides and handbooks; no.3).
Smith's weekly (1995): weekly newsletter of the University of New England, vol.36, no.44, 17 Nov., p4.