Paper by Colin Webb, Manager, Information Preservation, for the National Conservations and Preservation Strategy Public Forum, October 1996.
The purpose of this paper is to address Policy Statement 6 of the National Conservation and Preservation Policy for Movable Cultural Heritage:
The Commonwealth, State, Territory and Local Governments recognise that conservation and preservation are essential to provide ongoing access to Australia's movable cultural heritage for current and future generations.
The paper seeks to provide an overview of recent technological developments which assist in both providing access and protecting original materials, and to address the challenges posed by the need to preserve information found in new electronic formats.
This paper is built on three fundamental assumptions.
The first is that a civilised country will support the preservation of its cultural heritage. The Commonwealth's commitment to establishing a National Cultural Network may be seen as an expression of a community desire to explore the role of digital reformatting in doing this. This paper looks at the contribution modern digital reformatting methods might make, and at what is needed to preserve cultural information that exists in digital form.
To explore this, I have chosen to look for significant issues and principles rather than technical detail. My second underlying assumption is that technology will develop to deal with most of the technological barriers. This will still leave us with critical decisions needed to build and guide any strategy.
Many of these decisions are not simple either/or ones - there are complexities that must be understood - but this in itself need not be a barrier. Although we may not be able to see everything that will be needed to ensure preservation for a thousand years, we can at least make a start on what we know needs to be done for the next five, ten or 20 years.
The paper also does not aim to be comprehensive or definitive, but to suggest some of the territory needing further work and debate. In this, my third assumption - that existing collecting institutions will be major players over the short-to-medium term - should be obvious. Undoubtedly there are other players who must be considered, but it seems to make sense to build on existing infrastructure already invested by the community with resources and responsibilities to achieve basically these goals of access and preservation.
There is the prospect of tremendous benefits to be gained, but only if we are willing to understand the complexities, to provide and manage resources, and to commit ourselves to ensuring the survival of important digital information.
Although the paperless office may not have materialised quite as quickly as expected, over the past decade there have been great advances in a number of cross-fertilising digital technologies that have worked in two major directions relevant to access and preservation of cultural heritage materials:
Copying original items to produce a version that can be used instead of the originals has been employed for some time as a way of addressing the conflict between preservation and access. The ability to scan analogue images, text and sound recordings, producing digital copies that can be easily manipulated and distributed for use, has been used increasingly by cultural institutions in recent years, although it has a longer history in the records management field.
Its usefulness for improving accessibility is obvious. In theory it makes it possible for Australia's collecting institutions to provide access to their collections to anyone, anywhere, with the necessary technology. It is possible to detect at least four increasingly demanding visions of how it might be used to support preservation:
(The validity of these visions is explored further in the next section.)
Once created, digital images become a subset of the broader field of digital information.
Computers have been used to create and store digital data in Australia since the mid 1950s, with a huge increase in the use of computers in this country over the past few years. Recent innovations in networking and in authoring software (for example), have made it possible for information to be created and distributed in a plethora of digital formats at relatively low cost and without the need for complex programming skills that were once needed. As a result, more and more information is becoming available from what might have otherwise been inaccessible sources. At the same time, digital sound recording and digital photography are also being widely used. Much of what collecting institutions seek to collect is now available in digital form.
As a way of producing, distributing and storing information, this offers many apparent access and preservation benefits including remote and multiple access, speed of retrieval, searchability, a range of different access and storage paths (including server/client networks, optical compact disk, magnetic disk and tape, and so on), seamless access to different kinds of material from different sources, perfect and undetectable manipulation, perfect clones at every copying, and high storage densities compared with paper and film. There are effectively no "originals" that need to be protected, and the more widely the material is used, the greater the chance of survival.
Looking beyond the apparent benefits an ironic conundrum appears - "digital technology" (with all that it means) is touted as the ultimate answer to preservation problems - information that will last forever, that will never wear out, at very little cost; the ability to clone objects so they are available to everyone, everywhere, to use as they wish, and to copy endlessly without loss. At the same time, it may be the most vulnerable of all information systems, the ultimate in ephemera, changing before our eyes, disappearing without trace, present but irretrievable.
In fact, both extremes have some truth, but reality is probably more complex because of the issues, challenges and barriers to maximising the preservation potential.
The following issues are involved in using digital reformatting as part of a national preservation strategy. In many cases these issues apply to national approaches, and also to individual institutions and projects.
Focus question: Is digitisation a preservation strategy? In what way and to what extent?
The previous section presented four hypothetical visions of increasing reliance on digitisation as a component of a preservation strategy. What would be needed to achieve those visions?
Are these achievable? Four basic problems currently affect digital technology:
These "facts" add up to a powerful challenge to any vision of digitisation as a long term preservation panacea, because they question the ability of digital technology to deliver adequate image quality in a cost effective way, while also calling into question any guarantee of being able to maintain long term accessibility.
However, digitisation is probably best seen as a tool for preservation, rather than a complete answer or no answer at all. In some fields such as audio preservation it will probably be the main tool because analogue technology will desert the stage; elsewhere it will be best seen as one tool in the preservation toolbox. Its net effect is unquantified as it may actually encourage demand for access and display of some originals, and deepen, rather than alleviate, preservation problems.
A major challenge will be to maximise its usefulness without letting it outgrow its useful role. For digitisation is not preservation - to digitise is not to preserve; whatever it may contribute to preserving, it depends on getting many other issues right, and on long term management of the data created. In fact, in creating digital copies we are arguably creating a new preservation problem rather than solving an existing one.
The preservation role of digitisation is not an open and shut case. It requires discussion and debate about what we should aim to achieve, and what is acceptable. The preservation role should not be taken for granted, but needs to be clarified, and some basic indicators agreed on that would help to indicate when the technology, or expectations, had shifted sufficiently to warrant review of the debate.
Such a national debate needs to be informed by similar debate at the level of each sector, each institution and each project. This would help institutions and managers to make responsible and accountable decisions about objectives that will guide projects in the long sequence of management choices and compromises that most projects seem to entail.
(A significant concern clouding this issue is the uncertainty of making such decisions stick. Because digitisation is expensive there may be a great temptation to assume later that preservation goals have been achieved. This kind of wishful thinking is already common with microfilming, where it is easy to assume that materials have been "preserved" by microfilming and can be discarded, although the quality of the filming may never have been adequately checked and any masters may have been poorly stored and are not retrievable. Similar unjustified assumptions could easily arise with digitisation. This may demand responsible and agreed approaches to identifying and managing risks.)
Possible solutions: Some of the preservation limitations of digital reformatting may be overcome by technology development to improve image quality, and by experience in migrating data. In the meantime, the proper role of digitisation could be clarified by shared information - many institutions now have solid, practical experience to sharpen their assumptions - and debate about achievable objectives, informed by access to a wide range of experimental preservation-oriented projects, and a forum for consistent critical evaluation of projects.
Focus question: What resources are required; are they
justified, and where will they come from?
While the number of digitisation projects in Australia and elsewhere
grows almost daily, one consistent message is that it is expensive, both
because of the quantity and kind of materials involved, and because of the need
to deal with the kind of issues being discussed in this paper.
Just how expensive is difficult to quantify, because there are so
many cost variables. While it is easy enough to compare the unit price of
scanning with that of photography or microfilming, these are relatively minor
components of a raft of costs that change with each project and program. At the
same time, quantifying the real and potential benefits of digitisation, and
comparing them with other preservation methods is almost impossible - in some
cases there simply are no equivalent benefits, while in others there are no
equivalent risks. Ultimately, however, such comparisons must be made.
For institutions the most obvious question is funding. Should
digitisation be seen as an additional activity, or as a replacement for some
existing activity? If the latter, what priority should digitisation have,
compared with existing activities and functions? Will, for example, the care of
existing collections have to suffer to pay for digitisation?
On the other hand, if it is an additional activity, can enough
efficiencies be found to fund it from existing budgets, or will outside
resources be needed? And where will they be found?
While resolving the underlying resource issue will not be a
sufficient solution - there are many other issues that need to be resolved as
well as (in some cases before) allocation of resources - it is a necessary one.
Without adequate resources digitisation simply will not happen, and evidence so
far suggests that most institutions in Australia will be not be able to fund
more than very small digitisation programs out of their existing funds, while
income generated by marketed products such as CD-ROMs has generally been poor.
Just as critical as the need for funding per se is the need for some
certainty in longer range funding to facilitate forward planning.
Possible solutions: Some of these issues may be dealt with by access
to more information about the resources required, based on a wider range of
cost models. Resource difficulties may be influenced by a commitment by
governments to fund a program of experimental digitisation to establish
effective models, and by approaches to government and other sources of funds to
support or sponsor ongoing digitisation.
Focus question: How do we protect the rights of copyright
holders, and maintain the authenticity of digital copies?
Reformatting of any kind presents problems concerning the right to
copy and how the copies are used. Copyright and authenticity are two problems
that are exacerbated by the easily distributed, reproduced and manipulated data
that results from digital reformatting. Digitisation programs that do not
explicitly address the interests of copyright holders, and the interests of
users in knowing the authenticity of copies, while providing a level of access
that justifies the investment involved, are likely to compromise any
preservation strategy.
Possible solutions: This issue may be best dealt with through a
combination of improved technology and information. Technology such as
encryption that restricts unauthorised access and manipulation while
facilitating authorised access, will continue to develop to serve commercial
and data security interests, but managers will need to decide what is
appropriate. This would be helped by information and critical appraisal of what
technology is available, but also by a clear uderstanding of copyright law as
it applies to digital reformatting, and by best practice models for negotiation
with copyright holders and for strategies to protect copyright and maintain
authenticity. Although much harder to achieve, some consensus on what is the
right balance between access and restriction is needed.
Focus question: how do we improve the quality of all decisions
that are involved in each digitisation project?
A comprehensive description of the possibilities and their
implications is beyond the scope of this paper; some of the key process
decisions that must be made have been included in Appendix 1.
However, it is important to recognise that the ability of managers to make good
decisions is critical to developing an effective national preservation-related
digitisation strategy.
Possible solutions: Managers need information to guide decisions that
may have an important impact on the direction and effectiveness of any
digitisation program. Judicious experimenting in pilot projects should provide
more information about what works and what doesn't. The critical review of
models from existing and future programs, and possibly the development of
guidelines, would support good decision making.
Focus question: How much coordination, and of what kind, is
needed for a national strategy?
Establishing priorities for a national digitisation program, applying
consistent procedures and standards across all projects, setting up a national
registration system to describe and control digital data, and providing
"seamless" access for users whatever the source of the information, are all
attractive as evidence of a truly national approach. As well as the question of
how such coordination might be achieved, however, there is a more fundamental
question of how much integration would be beneficial and how much diversity
should be valued and encouraged.
Are there middle ways between complete integration and complete ad hocery that might be better than either extreme, retaining the special features of each institution and each sector's approach but producing a synergy that serves national goals? For example:
Possible solutions: investigation associated with the current
Department of Communication
and the Arts Digitisation of Collections consultancy on policy issues, and
the establishment of the National Cultural Network, should provide information
and guidance on this issue. However, there may need to be further discussion of
the benefits to be sought and the losses to be avoided in coordinating a
national approach, and of how it might be achieved especially in the key areas
of priorities, metadata and access.
Focus question: are there adequate archiving systems available
that will provide long term protection to the data?
Digital reformatting produces data that must be stored, backed up,
managed, archived and preserved if it is to offer any kind of ongoing access
and preservation benefits. These issues are explored in the next section.
Many of the characteristics, both positive and negative, of digital
data have already been mentioned or implied, particularly the instability of
the media and technology platforms that must be used for access. Added to these
difficulties, there are many other aspects of the way electronic publishing and
record keeping are developing that present preservation challenges, including
the explosion in creating and distributing data; the variety of sources,
formats, and quality; and the dynamic nature of the information. These issues
are explored in more detail in Appendix 2.
Focus question: how can we conceptualise the preservation
process for digital information?
There is not total unanimity about how to describe conservation and
preservation processes for traditional materials, but the terminology of
digital preservation is much less precise, reflecting the diversity of largely
untested understandings of what might be involved. Confusion over terms
currently hampers discussion about objectives, outcomes, and options.
For this paper, I am using the term "archiving" to refer to a "mark
and park" process that basically puts digital information in a safe place where
it will be protected from sudden loss for some time and where decisions can be
made about longer term protection. As such it is a second stage of a larger
process that I am calling "preservation", that ensures long term maintenance
and accessibility. The first stage of preservation is effective creation.
Possible solutions: Sorting out a common language in this field
requires more agreement about the processes that lead to long term
accessibility. While some progress can be made by discussing current concepts
and experience, there may need to be more shared experience before there is
agreement about what is possible.
Focus question: what should be kept, and for how long?
"Appraisal" is a term denoting quite specific procedures in archival
practice: I am using the word in a much more general way here to refer to all
the processes involved in making judgments about what should be kept.
It seems unlikely that all digital data can be kept forever, even if
it should be, (which is a problematic issue in itself). This concept is not
foreign to traditional preservation practice in institutions, although it may
have been masked by relatively slow rates of deterioration and it may have been
inconvenient to draw attention to it. For digital information the issue is
sharpened because the window of opportunity for effective intervention is so
much narrower.
Determining priorities for preservation - and even more importantly,
determining the basis for priorities - will be critical, as a springboard for
action.
However, working out which items warrant preservation action will not
be enough. Judgements are also needed about about the significant elements or
aspects of digital information that should be maintained. For some items
aspects like interactivity, links to other documents, and the look and feel of
the information, may be sacrificed for the sake of preserving some kind of
information content. Decisions about whether to take snapshots of information,
which versions to archive, whether to accumulate or overwrite versions, how far
to pursue links, are all practical questions that ultimately come down to the
issue of what is important enough to justify the effort.
Possible solutions: Establishing the basis for archiving priorities
nationally, or even within sectors, will be difficult. A reasonable start would
be for national institutions to make available for critical discussion any
collection development policies they may have that cover digital materials, or
selection guidelines they may have developed specifically for digital
materials.
Focus question: who should keep digital information?
Sorting out roles and responsibilities for preserving digital data
will be critical. Given that "benign neglect" offers nothing and active
intervention is required, there is great potential for loss in an environment
where no one has clearly defined their preservation role.
A blurring of formats and roles, and the fear of duplicating effort,
along with an equal fear of being swamped, may well leave important information
uncovered by any effective preservation arrangements.
While there are strong arguments for a network of repositories
committed to specific and complementary preservation roles, it is less clear
who should be involved in such a network, and how it might be made a reality.
What should be covered, and whether some performance monitoring mechanism is
needed to ensure that preservation commitments are being met, will also have to
be sorted out.
Possible solutions: Developing and negotiating an appropriate model
for preservation responsibilities, at least within sectors, should be a very
high priority . This should be based on a recognition of common interest. One
path in this direction would be the development, sharing and discussion of
preservation policies and/or position statements by major institutions. Such an
approach is likely to delay action, however, and a less formal process of trial
archiving projects within and between institutions with a history of
cooperation, might lay a more effective foundation for negotiating national
preservation networks.
Focus question: what is needed to make it easier to maintain
long term access?
There are a number of barriers that need attention in order to facilitate preservation:
Issue 11: Keeping
Focus question: how should digital data be kept?
Although it may not be a useful distinction to make in all cases, it
may be helpful to look at short term and long term approaches separately.
Short term, the main archiving issues may well be ones of finding a
site willing and able to take responsibility for storing and making the data
accessible in as close to its original context as possible. The best ways of
doing this will probably vary, but the use of large World Wide Web servers
appears to offer promise in providing storage capacity, easy access, and
similar retrieval paths to those used for live sites (even though some live
features such as extensive hyperlinks may be lost). Even in the short term,
however, the need to go off-line as server capacity is exceeded will introduce
retrieval and file management challenges in dealing with data stored on tape in
linear sequence.
Preservation beyond the short term means copying because of the media
instability, machine readability and technological obsolescence problems
already discussed. Short term refreshing - copying to new carriers within the
same technology - will eventually encounter the problem of technology change.
The three main proposed options for dealing with this - migration, emulation,
and maintaining obsolete technologies - are all problematical. All require
ongoing commitment, but the maintenance of a museum of dead technologies seems
to demand an unachievable commitment to maintaining skills as well as equipment
and software. This offers some short term potential while custodians are unable
to do anything else, but it must soon become unsustainable except as a
curiosity that may be useful only in the most exceptional circumstances.
Migration - copying from one technology platform to another - seems
the most feasible option, and it has been long practised in the computing
industry, although it will need to be managed on a scale and with a complexity
not experienced before. Just how it can be managed, at what costs and with what
losses to the information, are still not clear: although in theory digital data
can be copied endlessly without loss, much of the context and structure of
complex modern digital objects may demand more elaborate reconstruction than
they are worth.
Similar objections can be raised to the idea of emulation. The
concept of building software that makes hardware and other software act as if
they were artefacts of obsolete or different technology is used widely in
computing to provide cross platform access to competitors' software, for
example. However, there is virtually no chance that the rate of technological
change and innovation will slow down, so the chances of being able to rely on
emulation to solve all access problems long term is very slim. It may be
possible to build emulators for any data, but it almost certainly won't be cost
effective in many cases, and information will be lost because it wasn't worth
the effort of finding or building the required emulator.
All of these options would be aided by steps to minimise the number
of formats and by the use of open standardised systems that reduce the number
of variations needing to be accommodated. The development of data standards in
the digital spatial data field, and standard mark up languages that can be
applied to a great range of documents, are two examples of "technology
proofing" that offer great promise (although their proofing is only relative,
and they will almost certainly need to be migrated on at some stage to maintain
effective accessibility).
Possible solutions: There is probably not going to be a single grand
solution - all of these approaches, and others as well, are likely to play some
part. It is also possible that sustainable solutions will not be found in time
to save large quantities of digital information. Ultimately, the greatest need
will be the commitment to find manageable solutions; however, this commitment
will need information that leads to decisions and action. At this stage, the
highest priorities seem to be trials of migration paths, the development of
cost models, and support for the development and use of "technology proof"
formats and standards that will at least extend the technology cycle and reduce
the frequency of migrations.
Focus questions: where will the resources be found to do this?
Technology for short term, small scale archiving is reasonably
inexpensive and should only be a barrier for the most poorly resourced
institutions. The main short term resource barriers are likely to be the
expertise and labour required to set up and manage archiving sites. It is still
not clear what the longer term costs of preservation are likely to be. On one
side the cost of storage capacity and information management software is going
down; on the other, the information superhighway is a true freeway in that the
amount of traffic seems to increase to take up any increased capacity. The one
clear prospect, however, is that resources will be a major issue affecting the
willingness and ability of institutions to make the necessary long term
commitment to action.
As with digitisation, for many institutions the preservation of digital data is an additional activity in that digital information has not replaced older formats so much as added to the volume and diversity of information that must be managed and preserved.
One particularly important resource is expertise. Managing and
preserving digital data demands a variety of expertise in the areas of
technology and information management, librarian/archivist/curatorial skills,
preservation perspectives, and management skills. Developing and harnessing
this expertise is a major challenge, made more difficult where preservation and
other professionals do not see the roles they each have to play.
Possible solutions: Resource requirements for digital archiving and preservation need to be estimated on the basis of the best available information. A comparison of costs from different trial archiving projects would help. Cost models for longer term commitment will necessarily be tentative, but they may form a useful basis for discussion about the level of resources and the levels of activity that can be sustained.
Effective leadership that manages existing resources and brings together existing expertise to the best effect will also be highly desirable.
From this discussion of barriers and issues relating to digital reformatting and to the preservation of digital information, some general comments are worth emphasising.
It is reasonable to expect that there will always be innovation and change, and people wanting to take advantage of them. Standards will help, but they are unlikely to ever remove the need for technical expertise and for difficult judgments based on understanding of needs and objectives. This might mean that establishing standards for interoperability, with codes of preferred practice, might be more useful than a large scale focus on developing formal standards for every part of every process.
A list of possible recommendations associated with the issues in this
paper is given in Appendix 3. The following are more general
areas of action which may warrant the highest priority:
Dealing with these issues with an eye to their complexities, but also a willingness to find solutions, should do much to ensure that digital technology plays an appropriate part in a national conservation and preservation policy for Australia's movable cultural heritage.
Focus question: how do we decide the best way to digitise for preservation among all the available options?
These are some of the process issues on which it will be critical for managers to make good decisions if a national digitisation strategy is going to address preservation goals:
Some characteristics of the way electronic publishing and record keeping are developing that present challenges in archiving and preserving data:
Some possible recommendations attached to the issues
Issue 1: clarifying the preservation role of digitisation
It is recommended that institutions should undertake experimental digitisation projects and report on all aspects of them.
It is recommended that a forum such as an email discussion list or World Wide Web site be established for reporting and critical evaluation of projects.
Issue 2: Resources and conflicts in digitisation
It is recommended that there be a forum or clearing house set up for reporting and critical analysis of cost models for digitisation projects.
It is recommended that Australian governments commit funds to support a program of experimental digitisation in institutions over the next two years, to be followed by a review of resources required to support an ongoing digitisation program.
It is recommended that institutions identify potential partners (including government) who might be willing and able to support or sponsor ongoing digitisation.
Issue 3: The status of copies
It is recommended that a guidance paper for institutions on copyright law as it applies to digital reformatting be prepared and made available.
It is recommended that institutions and others set up a forum for presenting and discussing models for handling copyright and for protecting the authenticity of copies.
Issue 4: Decisions about the process of digitisation
It is recommended that institutions report on the detail of projects.
It is recommended that guidelines be developed for best practice in digitisation.
Issue 5: Coordination of a national digitisation effort
It is recommended that institutions note and discuss the findings of the DoCA consultancy.
It is recommended that a set of core metadata elements for all digitisation projects be investigated.
Issue 7: Conceptualising the digital preservation process
It is recommended that discussions of digital archiving and preservation include a clear explanation of terms until a widely agreed terminology emerges.
Issue 8: Appraisal for digital preservation
It is recommended that a clearing house for collection development policies and digital archiving selection guidelines be set up, or that existing clearing house structures such as the National Library's Distributed National Collection (DNC) Office be used, and that leading institutions in all sectors involved in archiving digital data compare selection policies.
Issue 9: Roles and responsibilities in digital preservation
It is recommended that institutions with an interest in archiving and preservation of digital data undertake and report on individual and joint trials, and that institutions act to identify likely partners and initiate negotiation of sector-based national preservation networks for digital data.
Issue 10: Facilitating preservation
It is recommended that institutions establish means of providing advice to creators about what will facilitate archiving digital data.
It is recommended that collecting arrangements and technologies that minimise the resource cost while delivering the required digital information to appropriate archiving sites be identified and developed.
It is recommended that institutions work together to develop, test and promulgate appropriate metadata standards, including do-it-yourself elements that can be used by creators.
It is recommended that current review of copyright legislation be finalised to address the legitimate interests of all stakeholders.
Issue 11: Keeping digital data
It is recommended that institutions investigate the costs and challenges in establishing feasible migration paths for digital data, recognising that this can only be tentative until major migration is really needed. They should also further investigate the potential for "technology proofing" standards and formats, and emulation, while keeping a watching brief on other possibilities.
Issue 12: Resources for digital preservation
It is recommended that institutions develop and discuss cost models for digital archiving and preservation action, as a basis for decisions about activities and resource bids to cover them.
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |