Staff Papers
Future Directions in Access and Preservation Technologies and New Electronic Formats
Paper by Colin Webb, Manager, Information Preservation, for the National Conservations and Preservation Strategy Public Forum, October 1996.
Purpose
The purpose of this paper is to address Policy Statement 6 of the National Conservation and Preservation Policy for Movable Cultural Heritage:
The Commonwealth, State, Territory and Local Governments recognise that conservation and preservation are essential to provide ongoing access to Australia's movable cultural heritage for current and future generations.
The paper seeks to provide an overview of recent technological developments which assist in both providing access and protecting original materials, and to address the challenges posed by the need to preserve information found in new electronic formats.
Introduction
This paper is built on three fundamental assumptions.
The first is that a civilised country will support the preservation of its cultural heritage. The Commonwealth's commitment to establishing a National Cultural Network may be seen as an expression of a community desire to explore the role of digital reformatting in doing this. This paper looks at the contribution modern digital reformatting methods might make, and at what is needed to preserve cultural information that exists in digital form.
To explore this, I have chosen to look for significant issues and principles rather than technical detail. My second underlying assumption is that technology will develop to deal with most of the technological barriers. This will still leave us with critical decisions needed to build and guide any strategy.
Many of these decisions are not simple either/or ones - there are complexities that must be understood - but this in itself need not be a barrier. Although we may not be able to see everything that will be needed to ensure preservation for a thousand years, we can at least make a start on what we know needs to be done for the next five, ten or 20 years.
The paper also does not aim to be comprehensive or definitive, but to suggest some of the territory needing further work and debate. In this, my third assumption - that existing collecting institutions will be major players over the short-to-medium term - should be obvious. Undoubtedly there are other players who must be considered, but it seems to make sense to build on existing infrastructure already invested by the community with resources and responsibilities to achieve basically these goals of access and preservation.
There is the prospect of tremendous benefits to be gained, but only if we are willing to understand the complexities, to provide and manage resources, and to commit ourselves to ensuring the survival of important digital information.
Background
Although the paperless office may not have materialised quite as quickly as expected, over the past decade there have been great advances in a number of cross-fertilising digital technologies that have worked in two major directions relevant to access and preservation of cultural heritage materials:
1. Reformatting
Copying original items to produce a version that can be used instead of the originals has been employed for some time as a way of addressing the conflict between preservation and access. The ability to scan analogue images, text and sound recordings, producing digital copies that can be easily manipulated and distributed for use, has been used increasingly by cultural institutions in recent years, although it has a longer history in the records management field.
Its usefulness for improving accessibility is obvious. In theory it makes it possible for Australia's collecting institutions to provide access to their collections to anyone, anywhere, with the necessary technology. It is possible to detect at least four increasingly demanding visions of how it might be used to support preservation:
- by deflecting use from original objects;
- by providing a secure backup for originals in case of loss (similar to photography or microfilm);
- by replacing originals that are already threatened by instability or technological obsolescence (eg brittle newspapers, sound recordings);
- by removing the need for institutions to store original collections at all.
(The validity of these visions is explored further in the next section.)
Once created, digital images become a subset of the broader field of digital information.
2. Digital information
Computers have been used to create and store digital data in Australia since the mid 1950s, with a huge increase in the use of computers in this country over the past few years. Recent innovations in networking and in authoring software (for example), have made it possible for information to be created and distributed in a plethora of digital formats at relatively low cost and without the need for complex programming skills that were once needed. As a result, more and more information is becoming available from what might have otherwise been inaccessible sources. At the same time, digital sound recording and digital photography are also being widely used. Much of what collecting institutions seek to collect is now available in digital form.
As a way of producing, distributing and storing information, this offers many apparent access and preservation benefits including remote and multiple access, speed of retrieval, searchability, a range of different access and storage paths (including server/client networks, optical compact disk, magnetic disk and tape, and so on), seamless access to different kinds of material from different sources, perfect and undetectable manipulation, perfect clones at every copying, and high storage densities compared with paper and film. There are effectively no "originals" that need to be protected, and the more widely the material is used, the greater the chance of survival.
Looking beyond the apparent benefits an ironic conundrum appears - "digital technology" (with all that it means) is touted as the ultimate answer to preservation problems - information that will last forever, that will never wear out, at very little cost; the ability to clone objects so they are available to everyone, everywhere, to use as they wish, and to copy endlessly without loss. At the same time, it may be the most vulnerable of all information systems, the ultimate in ephemera, changing before our eyes, disappearing without trace, present but irretrievable.
In fact, both extremes have some truth, but reality is probably more complex because of the issues, challenges and barriers to maximising the preservation potential.
What Issues and Barriers need to be Dealt with in Relation to Digitisation as a Preservation Strategy?
The following issues are involved in using digital reformatting as part of a national preservation strategy. In many cases these issues apply to national approaches, and also to individual institutions and projects.
Issue 1: clarifying the preservation role of digitisation
Focus question: Is digitisation a preservation strategy? In what way and to what extent?
The previous section presented four hypothetical visions of increasing reliance on digitisation as a component of a preservation strategy. What would be needed to achieve those visions?
- vision 1 - digital copies that deflect use from originals (most institutional digitisation projects to date in Australia probably fit into this category)- this requires digital copies to be of an adequate quality for use by most users, and for the digitisation and its products to be effectively managed, including some effort to protect the considerable investment likely to have been involved;
- vision 2 - digital copies that provide secure backup for originals in case of loss (similar to photography or microfilm) - this requires digital copies to be of an adequate quality for use by most users, and for the digitisation and its products to be effectively managed, including long term archiving so it will be available if needed;
- vision 3 - digital copies that replace originals that are already threatened by instability or technological obsolescence (eg brittle newspapers, sound recordings) - this requires digital copies to be as close a representation of the original as practical, and for the digitisation and its products to be effectively managed, including a comprehensive preservation commitment for guaranteed survival;
- vision 4 - digital copies that provide a completely adequate replacement for objects of all kinds and all values - this requires digital copies that capture all the characteristics considered worth retaining from the original, and that the digitisation and its products be effectively managed, including a comprehensive preservation commitment for guaranteed survival.
Are these achievable? Four basic problems currently affect digital technology:
- media instability: digital information is typically carried by media expected to undergo more rapid chemical and physical deterioration than "traditional" information carriers such as paper or film.
- machine readability: digital media do not present information in a form that can be read by humans without the aid of very specific machines and software. If the appropriate technology is not available, it is not possible to hold the carrier up to the light, or to get hold of a sufficiently powerful magnifier, and read it.
- technological obsolescence: innovation and rapid technological change, so characteristic of the digital information industry and market, virtually guarantee that the appropriate machines to read digital data will not be available for more than a few years after their creation.
- digital image quality: there are both absolute and effective limits to the quality of digital copies. Unlike an analogue image, which is continuous, digital images are created by "sampling" the information available from an analogue original, at a selected sampling rate. The more frequently samples are taken, the better the level of resolution or detail, but the larger the resulting file of data. While it is possible to approach the level of detail found in the original (and in most cases reach it so far as the human eye or ear is concerned), this can be a very expensive process and produces huge digital files that are expensive to store and difficult to transfer. This is one effective limit to image quality, although the limits are constantly being revised by improvements in the technology of sampling, data storage, data compression, and data transfer. On the other hand, many people would argue that the inability of any copy to convey the full information and cultural experience of contact with an original object is an absolute limit to digital image quality, (although this would also be challenged by those who either questioned the worth of such contact, or who foresaw technological developments in "virtual reality" that would make "original objects" redundant.)
These "facts" add up to a powerful challenge to any vision of digitisation as a long term preservation panacea, because they question the ability of digital technology to deliver adequate image quality in a cost effective way, while also calling into question any guarantee of being able to maintain long term accessibility.
However, digitisation is probably best seen as a tool for preservation, rather than a complete answer or no answer at all. In some fields such as audio preservation it will probably be the main tool because analogue technology will desert the stage; elsewhere it will be best seen as one tool in the preservation toolbox. Its net effect is unquantified as it may actually encourage demand for access and display of some originals, and deepen, rather than alleviate, preservation problems.
A major challenge will be to maximise its usefulness without letting it outgrow its useful role. For digitisation is not preservation - to digitise is not to preserve; whatever it may contribute to preserving, it depends on getting many other issues right, and on long term management of the data created. In fact, in creating digital copies we are arguably creating a new preservation problem rather than solving an existing one.
The preservation role of digitisation is not an open and shut case. It requires discussion and debate about what we should aim to achieve, and what is acceptable. The preservation role should not be taken for granted, but needs to be clarified, and some basic indicators agreed on that would help to indicate when the technology, or expectations, had shifted sufficiently to warrant review of the debate.
Such a national debate needs to be informed by similar debate at the level of each sector, each institution and each project. This would help institutions and managers to make responsible and accountable decisions about objectives that will guide projects in the long sequence of management choices and compromises that most projects seem to entail.
(A significant concern clouding this issue is the uncertainty of making such decisions stick. Because digitisation is expensive there may be a great temptation to assume later that preservation goals have been achieved. This kind of wishful thinking is already common with microfilming, where it is easy to assume that materials have been "preserved" by microfilming and can be discarded, although the quality of the filming may never have been adequately checked and any masters may have been poorly stored and are not retrievable. Similar unjustified assumptions could easily arise with digitisation. This may demand responsible and agreed approaches to identifying and managing risks.)
Possible solutions: Some of the preservation limitations of digital reformatting may be overcome by technology development to improve image quality, and by experience in migrating data. In the meantime, the proper role of digitisation could be clarified by shared information - many institutions now have solid, practical experience to sharpen their assumptions - and debate about achievable objectives, informed by access to a wide range of experimental preservation-oriented projects, and a forum for consistent critical evaluation of projects.
Issue 2: Resources and conflicts
Focus question: What resources are required; are they
justified, and where will they come from?
While the number of digitisation projects in Australia and elsewhere
grows almost daily, one consistent message is that it is expensive, both
because of the quantity and kind of materials involved, and because of the need
to deal with the kind of issues being discussed in this paper.
Just how expensive is difficult to quantify, because there are so
many cost variables. While it is easy enough to compare the unit price of
scanning with that of photography or microfilming, these are relatively minor
components of a raft of costs that change with each project and program. At the
same time, quantifying the real and potential benefits of digitisation, and
comparing them with other preservation methods is almost impossible - in some
cases there simply are no equivalent benefits, while in others there are no
equivalent risks. Ultimately, however, such comparisons must be made.
For institutions the most obvious question is funding. Should
digitisation be seen as an additional activity, or as a replacement for some
existing activity? If the latter, what priority should digitisation have,
compared with existing activities and functions? Will, for example, the care of
existing collections have to suffer to pay for digitisation?
On the other hand, if it is an additional activity, can enough
efficiencies be found to fund it from existing budgets, or will outside
resources be needed? And where will they be found?
While resolving the underlying resource issue will not be a
sufficient solution - there are many other issues that need to be resolved as
well as (in some cases before) allocation of resources - it is a necessary one.
Without adequate resources digitisation simply will not happen, and evidence so
far suggests that most institutions in Australia will be not be able to fund
more than very small digitisation programs out of their existing funds, while
income generated by marketed products such as CD-ROMs has generally been poor.
Just as critical as the need for funding per se is the need for some
certainty in longer range funding to facilitate forward planning.
Possible solutions: Some of these issues may be dealt with by access
to more information about the resources required, based on a wider range of
cost models. Resource difficulties may be influenced by a commitment by
governments to fund a program of experimental digitisation to establish
effective models, and by approaches to government and other sources of funds to
support or sponsor ongoing digitisation.
Issue 3: The status of copies
Focus question: How do we protect the rights of copyright
holders, and maintain the authenticity of digital copies?
Reformatting of any kind presents problems concerning the right to
copy and how the copies are used. Copyright and authenticity are two problems
that are exacerbated by the easily distributed, reproduced and manipulated data
that results from digital reformatting. Digitisation programs that do not
explicitly address the interests of copyright holders, and the interests of
users in knowing the authenticity of copies, while providing a level of access
that justifies the investment involved, are likely to compromise any
preservation strategy.
Possible solutions: This issue may be best dealt with through a
combination of improved technology and information. Technology such as
encryption that restricts unauthorised access and manipulation while
facilitating authorised access, will continue to develop to serve commercial
and data security interests, but managers will need to decide what is
appropriate. This would be helped by information and critical appraisal of what
technology is available, but also by a clear uderstanding of copyright law as
it applies to digital reformatting, and by best practice models for negotiation
with copyright holders and for strategies to protect copyright and maintain
authenticity. Although much harder to achieve, some consensus on what is the
right balance between access and restriction is needed.
Issue 4: Decisions about the process of digitisation
Focus question: how do we improve the quality of all decisions
that are involved in each digitisation project?
A comprehensive description of the possibilities and their
implications is beyond the scope of this paper; some of the key process
decisions that must be made have been included in Appendix 1.
However, it is important to recognise that the ability of managers to make good
decisions is critical to developing an effective national preservation-related
digitisation strategy.
Possible solutions: Managers need information to guide decisions that
may have an important impact on the direction and effectiveness of any
digitisation program. Judicious experimenting in pilot projects should provide
more information about what works and what doesn't. The critical review of
models from existing and future programs, and possibly the development of
guidelines, would support good decision making.
Issue 5: Coordination of a national digitisation effort
Focus question: How much coordination, and of what kind, is
needed for a national strategy?
Establishing priorities for a national digitisation program, applying
consistent procedures and standards across all projects, setting up a national
registration system to describe and control digital data, and providing
"seamless" access for users whatever the source of the information, are all
attractive as evidence of a truly national approach. As well as the question of
how such coordination might be achieved, however, there is a more fundamental
question of how much integration would be beneficial and how much diversity
should be valued and encouraged.
Are there middle ways between complete integration and complete ad hocery that might be better than either extreme, retaining the special features of each institution and each sector's approach but producing a synergy that serves national goals? For example:
- while from the point of view of a national strategy it makes sense to concentrate on materials of national significance, there may be an argument for projects for which there is an already identified and enthusiastic audience that will help to guide projects toward effective outcomes. In many cases this may lead in the direction of demonstrated local rather than national significance as a guide to priorities.
- metadata is seen as a crucial element in managing digital information; it seems likely that different institutons will continue to have different metadata requirements, but there may be room for establishing common core elements that should be attached to all digital objects while allowing some latitude in other elements to meet individual needs.
Possible solutions: investigation associated with the current
Department of Communication
and the Arts Digitisation of Collections consultancy on policy issues, and
the establishment of the National Cultural Network, should provide information
and guidance on this issue. However, there may need to be further discussion of
the benefits to be sought and the losses to be avoided in coordinating a
national approach, and of how it might be achieved especially in the key areas
of priorities, metadata and access.
Issue 6: Entry into an effective archiving system
Focus question: are there adequate archiving systems available
that will provide long term protection to the data?
Digital reformatting produces data that must be stored, backed up,
managed, archived and preserved if it is to offer any kind of ongoing access
and preservation benefits. These issues are explored in the next section.
What Issues and Brriers Need to be Dealt with in Order
to Have an Effective National Strategy for the Preservation of Digital
Information?
Introduction
Many of the characteristics, both positive and negative, of digital
data have already been mentioned or implied, particularly the instability of
the media and technology platforms that must be used for access. Added to these
difficulties, there are many other aspects of the way electronic publishing and
record keeping are developing that present preservation challenges, including
the explosion in creating and distributing data; the variety of sources,
formats, and quality; and the dynamic nature of the information. These issues
are explored in more detail in Appendix 2.
Issue 7: Conceptualising the process
Focus question: how can we conceptualise the preservation
process for digital information?
There is not total unanimity about how to describe conservation and
preservation processes for traditional materials, but the terminology of
digital preservation is much less precise, reflecting the diversity of largely
untested understandings of what might be involved. Confusion over terms
currently hampers discussion about objectives, outcomes, and options.
For this paper, I am using the term "archiving" to refer to a "mark
and park" process that basically puts digital information in a safe place where
it will be protected from sudden loss for some time and where decisions can be
made about longer term protection. As such it is a second stage of a larger
process that I am calling "preservation", that ensures long term maintenance
and accessibility. The first stage of preservation is effective creation.
Possible solutions: Sorting out a common language in this field
requires more agreement about the processes that lead to long term
accessibility. While some progress can be made by discussing current concepts
and experience, there may need to be more shared experience before there is
agreement about what is possible.
Issue 8: Appraisal
Focus question: what should be kept, and for how long?
"Appraisal" is a term denoting quite specific procedures in archival
practice: I am using the word in a much more general way here to refer to all
the processes involved in making judgments about what should be kept.
It seems unlikely that all digital data can be kept forever, even if
it should be, (which is a problematic issue in itself). This concept is not
foreign to traditional preservation practice in institutions, although it may
have been masked by relatively slow rates of deterioration and it may have been
inconvenient to draw attention to it. For digital information the issue is
sharpened because the window of opportunity for effective intervention is so
much narrower.
Determining priorities for preservation - and even more importantly,
determining the basis for priorities - will be critical, as a springboard for
action.
However, working out which items warrant preservation action will not
be enough. Judgements are also needed about about the significant elements or
aspects of digital information that should be maintained. For some items
aspects like interactivity, links to other documents, and the look and feel of
the information, may be sacrificed for the sake of preserving some kind of
information content. Decisions about whether to take snapshots of information,
which versions to archive, whether to accumulate or overwrite versions, how far
to pursue links, are all practical questions that ultimately come down to the
issue of what is important enough to justify the effort.
Possible solutions: Establishing the basis for archiving priorities
nationally, or even within sectors, will be difficult. A reasonable start would
be for national institutions to make available for critical discussion any
collection development policies they may have that cover digital materials, or
selection guidelines they may have developed specifically for digital
materials.
Issue 9: Roles and responsibilities
Focus question: who should keep digital information?
Sorting out roles and responsibilities for preserving digital data
will be critical. Given that "benign neglect" offers nothing and active
intervention is required, there is great potential for loss in an environment
where no one has clearly defined their preservation role.
A blurring of formats and roles, and the fear of duplicating effort,
along with an equal fear of being swamped, may well leave important information
uncovered by any effective preservation arrangements.
While there are strong arguments for a network of repositories
committed to specific and complementary preservation roles, it is less clear
who should be involved in such a network, and how it might be made a reality.
What should be covered, and whether some performance monitoring mechanism is
needed to ensure that preservation commitments are being met, will also have to
be sorted out.
Possible solutions: Developing and negotiating an appropriate model
for preservation responsibilities, at least within sectors, should be a very
high priority . This should be based on a recognition of common interest. One
path in this direction would be the development, sharing and discussion of
preservation policies and/or position statements by major institutions. Such an
approach is likely to delay action, however, and a less formal process of trial
archiving projects within and between institutions with a history of
cooperation, might lay a more effective foundation for negotiating national
preservation networks.
Issue 10: Facilitating preservation
Focus question: what is needed to make it easier to maintain
long term access?
There are a number of barriers that need attention in order to facilitate preservation:
- care by creators: those creating digital data need to be empowered to play their essential role - unless they use technologies that facilitate archiving, and maintain information until it has been archived, it will simply not be available for preservation and later access. This becomes an issue of working with creators so that they are aware of the potential implications of not following standards or using formats that make archiving feasible;
- ways of getting important information into an appropriate archiving system: contact with online digital information has been likened to drinking from a fire hose. Legal deposit, archives legislation, and other collecting arrangements have been effective mechanisms in the past. Some have adapted well to a digital environment, but others such as legal deposit laws, may need urgent review to re-establish their usefulness. In their absence, sustainable (and if possible, automated) ways of identifying and finding information to be archived will have to be developed;
- documentation and metadata: as well as assisting automated finding and retrieval, metadata is critically important for efficient management of electronic information. Developing consistent metadata standards that support management and preservation would be a great help;
- copyright law: existing copyright law in Australia hinders the preservation of digital information, which can only be preserved by copying. Changes to legislation should protect intellectual property rights but guarantee a similar level of free access to that provided in the past. Aims to develop satisfactory, technology-neutral legislation that facilitates preservation may take more time than is available to save some material. In the meantime it may be necessary to explore other possibilities with copyright owners to achieve similar ends.
Issue 11: Keeping
Focus question: how should digital data be kept?
Although it may not be a useful distinction to make in all cases, it
may be helpful to look at short term and long term approaches separately.
Short term, the main archiving issues may well be ones of finding a
site willing and able to take responsibility for storing and making the data
accessible in as close to its original context as possible. The best ways of
doing this will probably vary, but the use of large World Wide Web servers
appears to offer promise in providing storage capacity, easy access, and
similar retrieval paths to those used for live sites (even though some live
features such as extensive hyperlinks may be lost). Even in the short term,
however, the need to go off-line as server capacity is exceeded will introduce
retrieval and file management challenges in dealing with data stored on tape in
linear sequence.
Preservation beyond the short term means copying because of the media
instability, machine readability and technological obsolescence problems
already discussed. Short term refreshing - copying to new carriers within the
same technology - will eventually encounter the problem of technology change.
The three main proposed options for dealing with this - migration, emulation,
and maintaining obsolete technologies - are all problematical. All require
ongoing commitment, but the maintenance of a museum of dead technologies seems
to demand an unachievable commitment to maintaining skills as well as equipment
and software. This offers some short term potential while custodians are unable
to do anything else, but it must soon become unsustainable except as a
curiosity that may be useful only in the most exceptional circumstances.
Migration - copying from one technology platform to another - seems
the most feasible option, and it has been long practised in the computing
industry, although it will need to be managed on a scale and with a complexity
not experienced before. Just how it can be managed, at what costs and with what
losses to the information, are still not clear: although in theory digital data
can be copied endlessly without loss, much of the context and structure of
complex modern digital objects may demand more elaborate reconstruction than
they are worth.
Similar objections can be raised to the idea of emulation. The
concept of building software that makes hardware and other software act as if
they were artefacts of obsolete or different technology is used widely in
computing to provide cross platform access to competitors' software, for
example. However, there is virtually no chance that the rate of technological
change and innovation will slow down, so the chances of being able to rely on
emulation to solve all access problems long term is very slim. It may be
possible to build emulators for any data, but it almost certainly won't be cost
effective in many cases, and information will be lost because it wasn't worth
the effort of finding or building the required emulator.
All of these options would be aided by steps to minimise the number
of formats and by the use of open standardised systems that reduce the number
of variations needing to be accommodated. The development of data standards in
the digital spatial data field, and standard mark up languages that can be
applied to a great range of documents, are two examples of "technology
proofing" that offer great promise (although their proofing is only relative,
and they will almost certainly need to be migrated on at some stage to maintain
effective accessibility).
Possible solutions: There is probably not going to be a single grand
solution - all of these approaches, and others as well, are likely to play some
part. It is also possible that sustainable solutions will not be found in time
to save large quantities of digital information. Ultimately, the greatest need
will be the commitment to find manageable solutions; however, this commitment
will need information that leads to decisions and action. At this stage, the
highest priorities seem to be trials of migration paths, the development of
cost models, and support for the development and use of "technology proof"
formats and standards that will at least extend the technology cycle and reduce
the frequency of migrations.
Issue 12: Resources
Focus questions: where will the resources be found to do this?
Technology for short term, small scale archiving is reasonably
inexpensive and should only be a barrier for the most poorly resourced
institutions. The main short term resource barriers are likely to be the
expertise and labour required to set up and manage archiving sites. It is still
not clear what the longer term costs of preservation are likely to be. On one
side the cost of storage capacity and information management software is going
down; on the other, the information superhighway is a true freeway in that the
amount of traffic seems to increase to take up any increased capacity. The one
clear prospect, however, is that resources will be a major issue affecting the
willingness and ability of institutions to make the necessary long term
commitment to action.
As with digitisation, for many institutions the preservation of digital data is an additional activity in that digital information has not replaced older formats so much as added to the volume and diversity of information that must be managed and preserved.
One particularly important resource is expertise. Managing and
preserving digital data demands a variety of expertise in the areas of
technology and information management, librarian/archivist/curatorial skills,
preservation perspectives, and management skills. Developing and harnessing
this expertise is a major challenge, made more difficult where preservation and
other professionals do not see the roles they each have to play.
Possible solutions: Resource requirements for digital archiving and preservation need to be estimated on the basis of the best available information. A comparison of costs from different trial archiving projects would help. Cost models for longer term commitment will necessarily be tentative, but they may form a useful basis for discussion about the level of resources and the levels of activity that can be sustained.
Effective leadership that manages existing resources and brings together existing expertise to the best effect will also be highly desirable.
General Comments
From this discussion of barriers and issues relating to digital reformatting and to the preservation of digital information, some general comments are worth emphasising.
- Digital technology raises many challenges about how we see the world. Arguably there are shifts in things like the relationships between information and how it is carried; ownership and use; information and the way it is presented; creators, publishers, collecting institutions and users. Effective action requires that we recognise where such shifts are taking place but also that we are thoughtful about the extent of the shift and the extent to which we can still use existing models to understand the world.
- At the present time, technology is still reasonably immature. Technological development is important in providing tools to overcome many of the barriers. Innovations that can be used to facilitate preservation often grow out of other, non-preservation oriented innovations. At the same time, innovation is now so widespread that the relatively small market for data archiving is large enough to stimulate specific preservation tools. Either way, an opportunistic approach is needed to recognise potentially useful developments.
- Standards will be an important tool, and many of them are already emerging as either formal or de facto standards (such as SGML and JPEG). However, they tend to be standards for applying a particular product or process, still requiring decisions about which processes and which standards should apply.
It is reasonable to expect that there will always be innovation and change, and people wanting to take advantage of them. Standards will help, but they are unlikely to ever remove the need for technical expertise and for difficult judgments based on understanding of needs and objectives. This might mean that establishing standards for interoperability, with codes of preferred practice, might be more useful than a large scale focus on developing formal standards for every part of every process.
- Management issues - again and again, the issues that crop up are the ability to make decisions, to establish priorities and procedures, to negotiate roles. Although solutions may be technology-enabled, they are frequently not technology-controlled. So many of the issues come back to being able to rely on sound but innovative management practice.
- Control - some issues may be strongly influenced by forces outside the cultural heritage preservation industry, including technological, commercial and political or community pressures. It will be important to distinguish what can be influenced and what cannot.
- In looking for models that work, there are aspects of traditional preservation approaches that still work quite well, such as a recognition that not everything can be kept and a focus on objectives. We need to identify where traditional models have something to teach us and where they are simply inappropriate.
- Costs and benefits - behind most of what I have described as management decisions there needs to be a thorough, if informal, look at the costs and benefits of all options. This can be supported by access to more models and honest, critical review of them.
- Things will change, so responses have to be adaptable - there would be an irony in dealing with the unstable world of digital technology by setting in concrete strategies that would only work in the present circumstances. Just what this means in practical terms will need to be explored.
- A lot is happening already - preservation of digital information is no longer a silent, hidden issue, although it may not always use the jargon of preservation. Most major collecting institutions in Australia are involved in some kind of digitisation activity, and preservation of data is enough of a concern to be the subject of major national conferences, cross-sectoral working groups, and pilot projects. It is a frequent topic of debate on any one of a dozen or more international email discussion lists. The problem now is more one of what we do with all the activity, what we learn from it, how well we can make it serve national as well as individual institutional purposes.
- Roles and responsibilities and cooperation - this presses as the main issue. Is it necessary, or wise, to look for radical departures in structures? In Australia there is a strong and growing experience of cooperation between different institutions and sectors, and between Commonwealth, State, Territory and Local government levels and private institutions. It seems worthwhile exploring the usefulness of the existing infrastructure rather than starting from scratch. However, we may need to find ways to kick start action that will use the infrastructure to sort out responsibilities.
- The basis for exploring that cooperation - although it is not unambiguous, there is evidence that some creators and publishers are keen to participate, and some institutions prepared to take on a digital preservation role. Although there is a lot of work to be done, this may be the kind of support needed for national institutions to take a leading role in negotiating an effective cooperative framework.
Priorities and Recommendations
A list of possible recommendations associated with the issues in this
paper is given in Appendix 3. The following are more general
areas of action which may warrant the highest priority:
- establishing a forum for sharing information and critical review of models
- discussion of objectives and the role of digitisation;
- more experimental digital archiving and preservation trials with critical review;
- the development of cost models, that will be increasingly important in working out what is feasible and what resources are needed;
- sorting out roles and responsibilities;
- resource security and improved planning and commitment;
- technical research and standards;
- sorting out legislation that will enable practical preservation of information through legal deposit and intellectual property rights.
Dealing with these issues with an eye to their complexities, but also a willingness to find solutions, should do much to ensure that digital technology plays an appropriate part in a national conservation and preservation policy for Australia's movable cultural heritage.
Appendix 1
Decisions about the process of digitisation (Issue 4):
Focus question: how do we decide the best way to digitise for preservation among all the available options?
These are some of the process issues on which it will be critical for managers to make good decisions if a national digitisation strategy is going to address preservation goals:
- what should be digitised? - any expectation that all collections can be digitised within the next few years is unrealistic. Consequently the basis for setting priorities becomes a major issue. Different drivers may push priorities in different directions: for example, access-driven programs may have dilemmas over treasures, or popular material, or material that could become popular; preservation-driven programs may have dilemmas over materials at the greatest risk, or of the greatest cultural significance; experimental programs may look for material that will confront the least difficulties in getting runs on the board, or the most difficulties to provide the most learning; space and system management-driven programs may look for originals of such low value, or of such easily captured value, that they could be discarded when copied. Many institutions simply take an opportunistic approach looking for worthwhile projects that may combine any of these.
- are alternative approaches available? - many experimental projects in Australia and overseas have used hybrid approaches, producing both digital copies for access and traditional preservation masters for longer term security, though at a significantly greater cost than a purely digital approach;
- what resolution, bit depth and compression are feasible? - for copyright and practical reasons, many experimental projects have produced high resolution copies for storage and restricted use, and low quality images for networked distribution. Data compression offers possibilities of high resolution at smaller file sizes, but some compression is "lossy" and will not generate an acceptable image quality on decompression;
- what formats, systems and technologies are needed? - different digitisation paths may serve different goals: for example, scanning to produce an image of text may reproduce the look of a page but it will not provide the same word searching capability as conversion to computer-readable text via optical character recognition software;
- what documentation and metadata are required? - digital reformatting probably demands a similar level of documentation to other reformatting systems, but because it must be accessed and handled electronically, having the appropriate metadata attached to the individual file is especially important;
- what data organisation is needed to support access and preservation? - digital data does not reproduce what might look like a natural order in original collections being digitised. For storage and access, logical order has to be imposed by database design, indexing and control systems;
- what access systems will encourage appropriate use? - decisions are required at both the user interface level (the look and feel of how users will gain access), and at the distribution level (whether to use physical carriers like CD-ROM or networked access);
- how will the data be given immediate protection? - some kind of storage and backup system is required;
- what standards should be used? - standards, in the broader sense of preferred approaches, are evolving for many of these issues, but definite and obviously applicable standards are rare. Managers need to choose which standards to apply and how;
- who should manage digitisation? - because it draws on expertise from different functional areas of institutions, and may serve the objectives of different areas, decisions about who "owns" a digitisation program may have a significant influence on the objectives that get the most attention and what compromises are made;
- who should provide services? - decisions are needed about the costs and benefits of outsourcing compared with setting up inhouse operations, and about which services should be considered for outsourcing.
Appendix 2
Some characteristics of the way electronic publishing and record keeping are developing that present challenges in archiving and preserving data:
- the juxtaposition of electronic and non-electronic information - CD-ROMs and floppy disks are often published as part of non-digital titles, electronic and paper records often have to be accommodated in the one record keeping system;
- the variety of formats within multimedia resources - both online and discrete physical products usually consist of a number of formats, often associated with their own software for searching and viewing;
- the exponential growth in the creation and distribution of digital data;
- the fact that much of the data is coming from sources that are not moderated by the traditional conduits or filters such as publishers and record keeping systems; in many cases these creators are mainly interested in short term access and don't have the resources to support anything else;
- the dynamism of the scene - information appears and disappears, new types of publication or activity emerge, and even within the one title or project the look, feel and even the content of the information may change frequently and without notice;
- the quality of the information is very mixed - this is true of the scene as a whole, but it also applies within publications, which may have solid, refereed scholarship side by side with the most ephemeral information;
- much of the networked information is also available in other electronic or traditional formats, while much is unique;
- most networked material contains active hyperlinks to other material from the same or other sites - these links are exposed to changes that may be beyond the control of the creator or archive.
Appendix 3
Some possible recommendations attached to the issues
Issue 1: clarifying the preservation role of digitisation
It is recommended that institutions should undertake experimental digitisation projects and report on all aspects of them.
It is recommended that a forum such as an email discussion list or World Wide Web site be established for reporting and critical evaluation of projects.
Issue 2: Resources and conflicts in digitisation
It is recommended that there be a forum or clearing house set up for reporting and critical analysis of cost models for digitisation projects.
It is recommended that Australian governments commit funds to support a program of experimental digitisation in institutions over the next two years, to be followed by a review of resources required to support an ongoing digitisation program.
It is recommended that institutions identify potential partners (including government) who might be willing and able to support or sponsor ongoing digitisation.
Issue 3: The status of copies
It is recommended that a guidance paper for institutions on copyright law as it applies to digital reformatting be prepared and made available.
It is recommended that institutions and others set up a forum for presenting and discussing models for handling copyright and for protecting the authenticity of copies.
Issue 4: Decisions about the process of digitisation
It is recommended that institutions report on the detail of projects.
It is recommended that guidelines be developed for best practice in digitisation.
Issue 5: Coordination of a national digitisation effort
It is recommended that institutions note and discuss the findings of the DoCA consultancy.
It is recommended that a set of core metadata elements for all digitisation projects be investigated.
Issue 7: Conceptualising the digital preservation process
It is recommended that discussions of digital archiving and preservation include a clear explanation of terms until a widely agreed terminology emerges.
Issue 8: Appraisal for digital preservation
It is recommended that a clearing house for collection development policies and digital archiving selection guidelines be set up, or that existing clearing house structures such as the National Library's Distributed National Collection (DNC) Office be used, and that leading institutions in all sectors involved in archiving digital data compare selection policies.
Issue 9: Roles and responsibilities in digital preservation
It is recommended that institutions with an interest in archiving and preservation of digital data undertake and report on individual and joint trials, and that institutions act to identify likely partners and initiate negotiation of sector-based national preservation networks for digital data.
Issue 10: Facilitating preservation
It is recommended that institutions establish means of providing advice to creators about what will facilitate archiving digital data.
It is recommended that collecting arrangements and technologies that minimise the resource cost while delivering the required digital information to appropriate archiving sites be identified and developed.
It is recommended that institutions work together to develop, test and promulgate appropriate metadata standards, including do-it-yourself elements that can be used by creators.
It is recommended that current review of copyright legislation be finalised to address the legitimate interests of all stakeholders.
Issue 11: Keeping digital data
It is recommended that institutions investigate the costs and challenges in establishing feasible migration paths for digital data, recognising that this can only be tentative until major migration is really needed. They should also further investigate the potential for "technology proofing" standards and formats, and emulation, while keeping a watching brief on other possibilities.
Issue 12: Resources for digital preservation
It is recommended that institutions develop and discuss cost models for digital archiving and preservation action, as a basis for decisions about activities and resource bids to cover them.
