Staff Papers

Future Directions in Access and Preservation Technologies and New Electronic Formats

Paper by Colin Webb, Manager, Information Preservation, for the National Conservations and Preservation Strategy Public Forum, October 1996.


CONTENTS


Purpose

The purpose of this paper is to address Policy Statement 6 of the National Conservation and Preservation Policy for Movable Cultural Heritage:

The Commonwealth, State, Territory and Local Governments recognise that conservation and preservation are essential to provide ongoing access to Australia's movable cultural heritage for current and future generations.

The paper seeks to provide an overview of recent technological developments which assist in both providing access and protecting original materials, and to address the challenges posed by the need to preserve information found in new electronic formats.

Introduction

This paper is built on three fundamental assumptions.

The first is that a civilised country will support the preservation of its cultural heritage. The Commonwealth's commitment to establishing a National Cultural Network may be seen as an expression of a community desire to explore the role of digital reformatting in doing this. This paper looks at the contribution modern digital reformatting methods might make, and at what is needed to preserve cultural information that exists in digital form.

To explore this, I have chosen to look for significant issues and principles rather than technical detail. My second underlying assumption is that technology will develop to deal with most of the technological barriers. This will still leave us with critical decisions needed to build and guide any strategy.

Many of these decisions are not simple either/or ones - there are complexities that must be understood - but this in itself need not be a barrier. Although we may not be able to see everything that will be needed to ensure preservation for a thousand years, we can at least make a start on what we know needs to be done for the next five, ten or 20 years.

The paper also does not aim to be comprehensive or definitive, but to suggest some of the territory needing further work and debate. In this, my third assumption - that existing collecting institutions will be major players over the short-to-medium term - should be obvious. Undoubtedly there are other players who must be considered, but it seems to make sense to build on existing infrastructure already invested by the community with resources and responsibilities to achieve basically these goals of access and preservation.

There is the prospect of tremendous benefits to be gained, but only if we are willing to understand the complexities, to provide and manage resources, and to commit ourselves to ensuring the survival of important digital information.

Background

Although the paperless office may not have materialised quite as quickly as expected, over the past decade there have been great advances in a number of cross-fertilising digital technologies that have worked in two major directions relevant to access and preservation of cultural heritage materials:

1. Reformatting

Copying original items to produce a version that can be used instead of the originals has been employed for some time as a way of addressing the conflict between preservation and access. The ability to scan analogue images, text and sound recordings, producing digital copies that can be easily manipulated and distributed for use, has been used increasingly by cultural institutions in recent years, although it has a longer history in the records management field.

Its usefulness for improving accessibility is obvious. In theory it makes it possible for Australia's collecting institutions to provide access to their collections to anyone, anywhere, with the necessary technology. It is possible to detect at least four increasingly demanding visions of how it might be used to support preservation:

(The validity of these visions is explored further in the next section.)

Once created, digital images become a subset of the broader field of digital information.

2. Digital information

Computers have been used to create and store digital data in Australia since the mid 1950s, with a huge increase in the use of computers in this country over the past few years. Recent innovations in networking and in authoring software (for example), have made it possible for information to be created and distributed in a plethora of digital formats at relatively low cost and without the need for complex programming skills that were once needed. As a result, more and more information is becoming available from what might have otherwise been inaccessible sources. At the same time, digital sound recording and digital photography are also being widely used. Much of what collecting institutions seek to collect is now available in digital form.

As a way of producing, distributing and storing information, this offers many apparent access and preservation benefits including remote and multiple access, speed of retrieval, searchability, a range of different access and storage paths (including server/client networks, optical compact disk, magnetic disk and tape, and so on), seamless access to different kinds of material from different sources, perfect and undetectable manipulation, perfect clones at every copying, and high storage densities compared with paper and film. There are effectively no "originals" that need to be protected, and the more widely the material is used, the greater the chance of survival.

Looking beyond the apparent benefits an ironic conundrum appears - "digital technology" (with all that it means) is touted as the ultimate answer to preservation problems - information that will last forever, that will never wear out, at very little cost; the ability to clone objects so they are available to everyone, everywhere, to use as they wish, and to copy endlessly without loss. At the same time, it may be the most vulnerable of all information systems, the ultimate in ephemera, changing before our eyes, disappearing without trace, present but irretrievable.

In fact, both extremes have some truth, but reality is probably more complex because of the issues, challenges and barriers to maximising the preservation potential.

What Issues and Barriers need to be Dealt with in Relation to Digitisation as a Preservation Strategy?

The following issues are involved in using digital reformatting as part of a national preservation strategy. In many cases these issues apply to national approaches, and also to individual institutions and projects.

Issue 1: clarifying the preservation role of digitisation

Focus question: Is digitisation a preservation strategy? In what way and to what extent?

The previous section presented four hypothetical visions of increasing reliance on digitisation as a component of a preservation strategy. What would be needed to achieve those visions?

Are these achievable? Four basic problems currently affect digital technology:

These "facts" add up to a powerful challenge to any vision of digitisation as a long term preservation panacea, because they question the ability of digital technology to deliver adequate image quality in a cost effective way, while also calling into question any guarantee of being able to maintain long term accessibility.

However, digitisation is probably best seen as a tool for preservation, rather than a complete answer or no answer at all. In some fields such as audio preservation it will probably be the main tool because analogue technology will desert the stage; elsewhere it will be best seen as one tool in the preservation toolbox. Its net effect is unquantified as it may actually encourage demand for access and display of some originals, and deepen, rather than alleviate, preservation problems.

A major challenge will be to maximise its usefulness without letting it outgrow its useful role. For digitisation is not preservation - to digitise is not to preserve; whatever it may contribute to preserving, it depends on getting many other issues right, and on long term management of the data created. In fact, in creating digital copies we are arguably creating a new preservation problem rather than solving an existing one.

The preservation role of digitisation is not an open and shut case. It requires discussion and debate about what we should aim to achieve, and what is acceptable. The preservation role should not be taken for granted, but needs to be clarified, and some basic indicators agreed on that would help to indicate when the technology, or expectations, had shifted sufficiently to warrant review of the debate.

Such a national debate needs to be informed by similar debate at the level of each sector, each institution and each project. This would help institutions and managers to make responsible and accountable decisions about objectives that will guide projects in the long sequence of management choices and compromises that most projects seem to entail.

(A significant concern clouding this issue is the uncertainty of making such decisions stick. Because digitisation is expensive there may be a great temptation to assume later that preservation goals have been achieved. This kind of wishful thinking is already common with microfilming, where it is easy to assume that materials have been "preserved" by microfilming and can be discarded, although the quality of the filming may never have been adequately checked and any masters may have been poorly stored and are not retrievable. Similar unjustified assumptions could easily arise with digitisation. This may demand responsible and agreed approaches to identifying and managing risks.)

Possible solutions: Some of the preservation limitations of digital reformatting may be overcome by technology development to improve image quality, and by experience in migrating data. In the meantime, the proper role of digitisation could be clarified by shared information - many institutions now have solid, practical experience to sharpen their assumptions - and debate about achievable objectives, informed by access to a wide range of experimental preservation-oriented projects, and a forum for consistent critical evaluation of projects.

Issue 2: Resources and conflicts

Focus question: What resources are required; are they justified, and where will they come from?

While the number of digitisation projects in Australia and elsewhere grows almost daily, one consistent message is that it is expensive, both because of the quantity and kind of materials involved, and because of the need to deal with the kind of issues being discussed in this paper.

Just how expensive is difficult to quantify, because there are so many cost variables. While it is easy enough to compare the unit price of scanning with that of photography or microfilming, these are relatively minor components of a raft of costs that change with each project and program. At the same time, quantifying the real and potential benefits of digitisation, and comparing them with other preservation methods is almost impossible - in some cases there simply are no equivalent benefits, while in others there are no equivalent risks. Ultimately, however, such comparisons must be made.

For institutions the most obvious question is funding. Should digitisation be seen as an additional activity, or as a replacement for some existing activity? If the latter, what priority should digitisation have, compared with existing activities and functions? Will, for example, the care of existing collections have to suffer to pay for digitisation?

On the other hand, if it is an additional activity, can enough efficiencies be found to fund it from existing budgets, or will outside resources be needed? And where will they be found?

While resolving the underlying resource issue will not be a sufficient solution - there are many other issues that need to be resolved as well as (in some cases before) allocation of resources - it is a necessary one. Without adequate resources digitisation simply will not happen, and evidence so far suggests that most institutions in Australia will be not be able to fund more than very small digitisation programs out of their existing funds, while income generated by marketed products such as CD-ROMs has generally been poor.

Just as critical as the need for funding per se is the need for some certainty in longer range funding to facilitate forward planning.

Possible solutions: Some of these issues may be dealt with by access to more information about the resources required, based on a wider range of cost models. Resource difficulties may be influenced by a commitment by governments to fund a program of experimental digitisation to establish effective models, and by approaches to government and other sources of funds to support or sponsor ongoing digitisation.

Issue 3: The status of copies

Focus question: How do we protect the rights of copyright holders, and maintain the authenticity of digital copies?

Reformatting of any kind presents problems concerning the right to copy and how the copies are used. Copyright and authenticity are two problems that are exacerbated by the easily distributed, reproduced and manipulated data that results from digital reformatting. Digitisation programs that do not explicitly address the interests of copyright holders, and the interests of users in knowing the authenticity of copies, while providing a level of access that justifies the investment involved, are likely to compromise any preservation strategy.

Possible solutions: This issue may be best dealt with through a combination of improved technology and information. Technology such as encryption that restricts unauthorised access and manipulation while facilitating authorised access, will continue to develop to serve commercial and data security interests, but managers will need to decide what is appropriate. This would be helped by information and critical appraisal of what technology is available, but also by a clear uderstanding of copyright law as it applies to digital reformatting, and by best practice models for negotiation with copyright holders and for strategies to protect copyright and maintain authenticity. Although much harder to achieve, some consensus on what is the right balance between access and restriction is needed.

Issue 4: Decisions about the process of digitisation

Focus question: how do we improve the quality of all decisions that are involved in each digitisation project?

A comprehensive description of the possibilities and their implications is beyond the scope of this paper; some of the key process decisions that must be made have been included in Appendix 1. However, it is important to recognise that the ability of managers to make good decisions is critical to developing an effective national preservation-related digitisation strategy.

Possible solutions: Managers need information to guide decisions that may have an important impact on the direction and effectiveness of any digitisation program. Judicious experimenting in pilot projects should provide more information about what works and what doesn't. The critical review of models from existing and future programs, and possibly the development of guidelines, would support good decision making.

Issue 5: Coordination of a national digitisation effort

Focus question: How much coordination, and of what kind, is needed for a national strategy?

Establishing priorities for a national digitisation program, applying consistent procedures and standards across all projects, setting up a national registration system to describe and control digital data, and providing "seamless" access for users whatever the source of the information, are all attractive as evidence of a truly national approach. As well as the question of how such coordination might be achieved, however, there is a more fundamental question of how much integration would be beneficial and how much diversity should be valued and encouraged.

Are there middle ways between complete integration and complete ad hocery that might be better than either extreme, retaining the special features of each institution and each sector's approach but producing a synergy that serves national goals? For example:

Possible solutions: investigation associated with the current Department of Communication and the Arts Digitisation of Collections consultancy on policy issues, and the establishment of the National Cultural Network, should provide information and guidance on this issue. However, there may need to be further discussion of the benefits to be sought and the losses to be avoided in coordinating a national approach, and of how it might be achieved especially in the key areas of priorities, metadata and access.

Issue 6: Entry into an effective archiving system

Focus question: are there adequate archiving systems available that will provide long term protection to the data?

Digital reformatting produces data that must be stored, backed up, managed, archived and preserved if it is to offer any kind of ongoing access and preservation benefits. These issues are explored in the next section.

What Issues and Brriers Need to be Dealt with in Order to Have an Effective National Strategy for the Preservation of Digital Information?

Introduction

Many of the characteristics, both positive and negative, of digital data have already been mentioned or implied, particularly the instability of the media and technology platforms that must be used for access. Added to these difficulties, there are many other aspects of the way electronic publishing and record keeping are developing that present preservation challenges, including the explosion in creating and distributing data; the variety of sources, formats, and quality; and the dynamic nature of the information. These issues are explored in more detail in Appendix 2.


Issue 7: Conceptualising the process

Focus question: how can we conceptualise the preservation process for digital information?

There is not total unanimity about how to describe conservation and preservation processes for traditional materials, but the terminology of digital preservation is much less precise, reflecting the diversity of largely untested understandings of what might be involved. Confusion over terms currently hampers discussion about objectives, outcomes, and options.

For this paper, I am using the term "archiving" to refer to a "mark and park" process that basically puts digital information in a safe place where it will be protected from sudden loss for some time and where decisions can be made about longer term protection. As such it is a second stage of a larger process that I am calling "preservation", that ensures long term maintenance and accessibility. The first stage of preservation is effective creation.

Possible solutions: Sorting out a common language in this field requires more agreement about the processes that lead to long term accessibility. While some progress can be made by discussing current concepts and experience, there may need to be more shared experience before there is agreement about what is possible.

Issue 8: Appraisal

Focus question: what should be kept, and for how long?

"Appraisal" is a term denoting quite specific procedures in archival practice: I am using the word in a much more general way here to refer to all the processes involved in making judgments about what should be kept.

It seems unlikely that all digital data can be kept forever, even if it should be, (which is a problematic issue in itself). This concept is not foreign to traditional preservation practice in institutions, although it may have been masked by relatively slow rates of deterioration and it may have been inconvenient to draw attention to it. For digital information the issue is sharpened because the window of opportunity for effective intervention is so much narrower.

Determining priorities for preservation - and even more importantly, determining the basis for priorities - will be critical, as a springboard for action.

However, working out which items warrant preservation action will not be enough. Judgements are also needed about about the significant elements or aspects of digital information that should be maintained. For some items aspects like interactivity, links to other documents, and the look and feel of the information, may be sacrificed for the sake of preserving some kind of information content. Decisions about whether to take snapshots of information, which versions to archive, whether to accumulate or overwrite versions, how far to pursue links, are all practical questions that ultimately come down to the issue of what is important enough to justify the effort.

Possible solutions: Establishing the basis for archiving priorities nationally, or even within sectors, will be difficult. A reasonable start would be for national institutions to make available for critical discussion any collection development policies they may have that cover digital materials, or selection guidelines they may have developed specifically for digital materials.

Issue 9: Roles and responsibilities

Focus question: who should keep digital information?

Sorting out roles and responsibilities for preserving digital data will be critical. Given that "benign neglect" offers nothing and active intervention is required, there is great potential for loss in an environment where no one has clearly defined their preservation role.

A blurring of formats and roles, and the fear of duplicating effort, along with an equal fear of being swamped, may well leave important information uncovered by any effective preservation arrangements.

While there are strong arguments for a network of repositories committed to specific and complementary preservation roles, it is less clear who should be involved in such a network, and how it might be made a reality. What should be covered, and whether some performance monitoring mechanism is needed to ensure that preservation commitments are being met, will also have to be sorted out.

Possible solutions: Developing and negotiating an appropriate model for preservation responsibilities, at least within sectors, should be a very high priority . This should be based on a recognition of common interest. One path in this direction would be the development, sharing and discussion of preservation policies and/or position statements by major institutions. Such an approach is likely to delay action, however, and a less formal process of trial archiving projects within and between institutions with a history of cooperation, might lay a more effective foundation for negotiating national preservation networks.

Issue 10: Facilitating preservation

Focus question: what is needed to make it easier to maintain long term access?

There are a number of barriers that need attention in order to facilitate preservation:


Issue 11: Keeping

Focus question: how should digital data be kept?

Although it may not be a useful distinction to make in all cases, it may be helpful to look at short term and long term approaches separately.

Short term, the main archiving issues may well be ones of finding a site willing and able to take responsibility for storing and making the data accessible in as close to its original context as possible. The best ways of doing this will probably vary, but the use of large World Wide Web servers appears to offer promise in providing storage capacity, easy access, and similar retrieval paths to those used for live sites (even though some live features such as extensive hyperlinks may be lost). Even in the short term, however, the need to go off-line as server capacity is exceeded will introduce retrieval and file management challenges in dealing with data stored on tape in linear sequence.

Preservation beyond the short term means copying because of the media instability, machine readability and technological obsolescence problems already discussed. Short term refreshing - copying to new carriers within the same technology - will eventually encounter the problem of technology change. The three main proposed options for dealing with this - migration, emulation, and maintaining obsolete technologies - are all problematical. All require ongoing commitment, but the maintenance of a museum of dead technologies seems to demand an unachievable commitment to maintaining skills as well as equipment and software. This offers some short term potential while custodians are unable to do anything else, but it must soon become unsustainable except as a curiosity that may be useful only in the most exceptional circumstances.

Migration - copying from one technology platform to another - seems the most feasible option, and it has been long practised in the computing industry, although it will need to be managed on a scale and with a complexity not experienced before. Just how it can be managed, at what costs and with what losses to the information, are still not clear: although in theory digital data can be copied endlessly without loss, much of the context and structure of complex modern digital objects may demand more elaborate reconstruction than they are worth.

Similar objections can be raised to the idea of emulation. The concept of building software that makes hardware and other software act as if they were artefacts of obsolete or different technology is used widely in computing to provide cross platform access to competitors' software, for example. However, there is virtually no chance that the rate of technological change and innovation will slow down, so the chances of being able to rely on emulation to solve all access problems long term is very slim. It may be possible to build emulators for any data, but it almost certainly won't be cost effective in many cases, and information will be lost because it wasn't worth the effort of finding or building the required emulator.

All of these options would be aided by steps to minimise the number of formats and by the use of open standardised systems that reduce the number of variations needing to be accommodated. The development of data standards in the digital spatial data field, and standard mark up languages that can be applied to a great range of documents, are two examples of "technology proofing" that offer great promise (although their proofing is only relative, and they will almost certainly need to be migrated on at some stage to maintain effective accessibility).

Possible solutions: There is probably not going to be a single grand solution - all of these approaches, and others as well, are likely to play some part. It is also possible that sustainable solutions will not be found in time to save large quantities of digital information. Ultimately, the greatest need will be the commitment to find manageable solutions; however, this commitment will need information that leads to decisions and action. At this stage, the highest priorities seem to be trials of migration paths, the development of cost models, and support for the development and use of "technology proof" formats and standards that will at least extend the technology cycle and reduce the frequency of migrations.

Issue 12: Resources

Focus questions: where will the resources be found to do this?

Technology for short term, small scale archiving is reasonably inexpensive and should only be a barrier for the most poorly resourced institutions. The main short term resource barriers are likely to be the expertise and labour required to set up and manage archiving sites. It is still not clear what the longer term costs of preservation are likely to be. On one side the cost of storage capacity and information management software is going down; on the other, the information superhighway is a true freeway in that the amount of traffic seems to increase to take up any increased capacity. The one clear prospect, however, is that resources will be a major issue affecting the willingness and ability of institutions to make the necessary long term commitment to action.

As with digitisation, for many institutions the preservation of digital data is an additional activity in that digital information has not replaced older formats so much as added to the volume and diversity of information that must be managed and preserved.

One particularly important resource is expertise. Managing and preserving digital data demands a variety of expertise in the areas of technology and information management, librarian/archivist/curatorial skills, preservation perspectives, and management skills. Developing and harnessing this expertise is a major challenge, made more difficult where preservation and other professionals do not see the roles they each have to play.

Possible solutions: Resource requirements for digital archiving and preservation need to be estimated on the basis of the best available information. A comparison of costs from different trial archiving projects would help. Cost models for longer term commitment will necessarily be tentative, but they may form a useful basis for discussion about the level of resources and the levels of activity that can be sustained.

Effective leadership that manages existing resources and brings together existing expertise to the best effect will also be highly desirable.

General Comments

From this discussion of barriers and issues relating to digital reformatting and to the preservation of digital information, some general comments are worth emphasising.

It is reasonable to expect that there will always be innovation and change, and people wanting to take advantage of them. Standards will help, but they are unlikely to ever remove the need for technical expertise and for difficult judgments based on understanding of needs and objectives. This might mean that establishing standards for interoperability, with codes of preferred practice, might be more useful than a large scale focus on developing formal standards for every part of every process.

Priorities and Recommendations

A list of possible recommendations associated with the issues in this paper is given in Appendix 3. The following are more general areas of action which may warrant the highest priority:

Dealing with these issues with an eye to their complexities, but also a willingness to find solutions, should do much to ensure that digital technology plays an appropriate part in a national conservation and preservation policy for Australia's movable cultural heritage.


Appendix 1

Decisions about the process of digitisation (Issue 4):

Focus question: how do we decide the best way to digitise for preservation among all the available options?

These are some of the process issues on which it will be critical for managers to make good decisions if a national digitisation strategy is going to address preservation goals:


Appendix 2

Some characteristics of the way electronic publishing and record keeping are developing that present challenges in archiving and preserving data:


Appendix 3

Some possible recommendations attached to the issues

Issue 1: clarifying the preservation role of digitisation

It is recommended that institutions should undertake experimental digitisation projects and report on all aspects of them.

It is recommended that a forum such as an email discussion list or World Wide Web site be established for reporting and critical evaluation of projects.

Issue 2: Resources and conflicts in digitisation

It is recommended that there be a forum or clearing house set up for reporting and critical analysis of cost models for digitisation projects.

It is recommended that Australian governments commit funds to support a program of experimental digitisation in institutions over the next two years, to be followed by a review of resources required to support an ongoing digitisation program.

It is recommended that institutions identify potential partners (including government) who might be willing and able to support or sponsor ongoing digitisation.

Issue 3: The status of copies

It is recommended that a guidance paper for institutions on copyright law as it applies to digital reformatting be prepared and made available.

It is recommended that institutions and others set up a forum for presenting and discussing models for handling copyright and for protecting the authenticity of copies.

Issue 4: Decisions about the process of digitisation

It is recommended that institutions report on the detail of projects.

It is recommended that guidelines be developed for best practice in digitisation.

Issue 5: Coordination of a national digitisation effort

It is recommended that institutions note and discuss the findings of the DoCA consultancy.

It is recommended that a set of core metadata elements for all digitisation projects be investigated.

Issue 7: Conceptualising the digital preservation process

It is recommended that discussions of digital archiving and preservation include a clear explanation of terms until a widely agreed terminology emerges.

Issue 8: Appraisal for digital preservation

It is recommended that a clearing house for collection development policies and digital archiving selection guidelines be set up, or that existing clearing house structures such as the National Library's Distributed National Collection (DNC) Office be used, and that leading institutions in all sectors involved in archiving digital data compare selection policies.

Issue 9: Roles and responsibilities in digital preservation

It is recommended that institutions with an interest in archiving and preservation of digital data undertake and report on individual and joint trials, and that institutions act to identify likely partners and initiate negotiation of sector-based national preservation networks for digital data.

Issue 10: Facilitating preservation

It is recommended that institutions establish means of providing advice to creators about what will facilitate archiving digital data.

It is recommended that collecting arrangements and technologies that minimise the resource cost while delivering the required digital information to appropriate archiving sites be identified and developed.

It is recommended that institutions work together to develop, test and promulgate appropriate metadata standards, including do-it-yourself elements that can be used by creators.

It is recommended that current review of copyright legislation be finalised to address the legitimate interests of all stakeholders.

Issue 11: Keeping digital data

It is recommended that institutions investigate the costs and challenges in establishing feasible migration paths for digital data, recognising that this can only be tentative until major migration is really needed. They should also further investigate the potential for "technology proofing" standards and formats, and emulation, while keeping a watching brief on other possibilities.

Issue 12: Resources for digital preservation

It is recommended that institutions develop and discuss cost models for digital archiving and preservation action, as a basis for decisions about activities and resource bids to cover them.

Top

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us