Digital preservation directions statement 2008 to 2012

23 September 2008

1.0 Introduction

1.1 Purpose

This Directions Statement sets out the National Library of Australia’s current priorities in relation to digital preservation for the period 2008 to 2012. These priorities should be considered in the context of the directions and principles articulated in the Library’s Digital Preservation Policy (3rd edition, 2008).

1.2 Goals

The Library’s Digital Preservation Policy defines digital preservation as “the process of maintaining, and if necessary, recovering accessibility to digital information resources.”

The digital preservation objectives stated in the Policy refer to maintaining and preserving access to the Library’s own digital collections, and helping others preserve digital information resources for which they are responsible.

In the term of this Directions Statement, the Library’s goals for its own digital collections are:

  • to take appropriate available preservation action that is needed now; and
  • to significantly increase the Library’s readiness to take longer term preservation action when it will be needed.

With regard to helping others, the Library’s goal is:

  • to help others be aware of, develop and achieve their digital preservation goals
  • in special cases, to play an active role in storing or hosting digital collections belonging to others, with clearly understood agreements on the Library’s preservation responsibilities.

2.0 Vision - Where should the Library aim to be?

2.1 The vision for preserving the Library's digital collections

Over the next 5 years and beyond the Library should aim to achieve the following state:

  • There is good understanding in the Library of preservation issues and of the constraints of preservation strategies, which are taken into account in deciding what to take into the Library’s collections and how it should be managed.
  • Acquisition and processing workflows are in place that lead to well managed collections for the range of digital materials the Library wants to collect.
  • We are collecting appropriate metadata for all digital materials.
  • All digital collection materials, regardless of format, are securely stored and well managed.
  • We have policies in place to assist clear decision making on the Library’s preservation aims, responsibilities, roles and priorities.
  • We are well placed to prevent or respond to threats to the digital collections -
    • infrastructure for monitoring obsolescence risks is in place and in use
    • access dependencies are documented and maintained in well managed ways
    • appropriate preservation responses to maintain viable viewpaths are developed or at an advanced stage of development (including strategies, tools and plans) for all types of digital materials
    • other risks such as data failure are monitored
    • reliable arrangements are in place to mitigate major data security threats such as disasters affecting the Library and the Canberra region.
  • We can provide access to all digital materials via common delivery mechanisms, such as online delivery, subject to legal and ethical constraints on access.
  • The Library’s digital collection management and preservation arrangements are fully documented and the Library operates as a trustworthy repository for its own collections and for the collections of others which it agrees to manage or store.

The timing and degree to which each of these is realised may be influenced by factors including:

  • the introduction of Legal Deposit legislation at a national level for digital materials, which may have an impact on the range and volume of materials to be managed through industrial-scale, automated workflows
  • the appearance of new forms of digital materials the Library may seek to collect
  • the availability of funding resources and skilled staff that can be allocated to acquiring, managing and preserving digital collections
  • the Library’s decisions about the scope of its engagement with communities creating digital content.

3.0 Taking stock

3.1 Emerging developments in digital preservation

Awareness and understanding of the issues and potential solutions for preserving digital materials have grown significantly in the past decade. There is a growing body of knowledge and recommended practices which the Library can draw on in addressing its digital preservation challenges.

The next five years can be expected to bring a consolidation of this body of knowledge, and the filling of some existing gaps. It seems unlikely that comprehensive digital preservation solutions will be available “off the shelf” in this timeframe (if ever), partly because of the different business needs and contexts of major players in the potential market for such products and services. While the Library can expect to make use of many developments to extend its infrastructure for addressing its preservation challenges, there will almost certainly be an ongoing need to invest in refining and adapting some of the tools that are or will be available. On some issues, the Library may need to lead the development of a coordinated comprehensive solution.

3.2 Current progress in NLA

The Library has reasonably good high-level understanding of its digital collections and of their preservation risks. This understanding derives in part from a broad risk assessment of digital collections carried out in 2003-2004, and in part from subsequent work on obsolescence tools and ingest workflows.

The following areas have been identified as priorities for attention:

  1. establishing consultation mechanisms for effective planning of preservation action
  2. acquisition and processing workflows for some materials and some collection areas, including improvements in existing workflows and development of new workflows such as a digital deposit model
  3. empowering other business areas (collection areas, Digitisation) to make informed preservation decisions and to take appropriate action – including advice and training for staff, access to format identification tools, and data capture standards for digitisation
  4. implementing workflows for the transfer of data from fragile physical format carriers to media-independent, managed mass storage, including dealing with the backlog of at risk materials
  5. development of policies including:
    • clarification of copyright and licensing issues
    • levels of preservation
    • acceptable levels of loss
    • responsibility for the preservation of other people’s materials
    • curatorial responsibility for digitisation copies
    • understanding costs of managing digital collection
  6. developing and implementing preservation metadata schemes for the range of materials, to support effective preservation management, including metadata that accurately reflects complex relationships, the maintenance of links between objects and metadata, and the preservation of metadata resources
  7. implementing tools to identify and characterise materials automatically
  8. identifying and maintaining access dependencies (hardware, software, etc) for existing and incoming materials
  9. clarifying safe storage issues to do with fixity checking, backup regimes and storage media
  10. further developing, testing and implementation of systems to monitoring obsolescence  
  11. developing and planning of transformation strategies. Obsolescence problems for some specific formats are expected to occur within the timeframe of this Directions Statement, although for many formats the obsolescence timeframe is likely to be longer. Access to content in some formats such as older versions of Microsoft Word is already endangered.
  12. establishing what is required for the Library to operate as a trustworthy digital repository, and establishment of an audit program with a sustainable methodology.

4.0 Finding a way forward

4.1 Roles

The challenges of preserving the Library’s digital collections can only be addressed through well coordinated action across the Library. Such cooperation involves key partnerships between Digital Preservation, the Library’s IT Division, relevant collection areas and digitisation programs, with strong management support. (Roles are set out in more detail in Appendix A.)

The Digital Preservation Section is required to take a leadership role in understanding the nature and access dependencies of the digital collections across the Library, and in developing and planning effective action to ensure collection materials remain accessible. This involves preventive action (such as specifications; providing a preservation perspective on the development of relevant infrastructure and workflows; empowering staff to make effective preservation decisions; and understanding what is needed to maintain access).

It also involves timely and well managed responses to risks (such as risk monitoring; maintaining access dependencies and planning for their transformation; research and prototyping of tools; and specialised processing).

The Digital Preservation Section has a leadership role in coordinating the development of relevant policy and strategic plans, and in monitoring and reporting on repository performance and preservation capability. The Section is also the frontline in coordinating collaborative efforts with external digital preservation initiatives and in sharing information about the Library’s programs.

The Information Technology Division plays a critical role in providing infrastructure and systems to ingest and safely store the digital collections; to support and implement automated preservation workflows; and to provide a high level of technical expertise.

Collection areas play a critical role in selecting and acquiring digital collection materials; in their initial processing, control and documentation; and in deciding on preservation aims and priorities for materials in their collections, including digitised versions of their analogue collections.

Digitisation programs play a critical role in ensuring that the digital copies they create comply with agreed standards and formats; and in ingesting the copies and their metadata into safe storage.

To ensure coordination of these roles, the Library will re-establish a consultative digital preservation planning committee, the Digital Preservation Planning and Issues Group DPPIG), as a sub-committee of CDMC. The DPPIG will be chaired by the ADG, Collections Management, and consist of representatives of Web Archiving and Digital Preservation Branch, IT Division, relevant collection areas, digitisation programs, and management. The DPPIG will meet quarterly to establish and monitor priorities, approve forward plans, discuss issues requiring joint decisions, and monitor projects.

5.0 Sharing our experience

As previously noted, the Library also has a role in helping others be aware of, develop and achieve their digital preservation goals. Contact and collaboration with other initiatives also helps us stay abreast of developments. Activities that can be undertaken in this regard include:

  • maintaining the PADI subject gateway, and planning to take advantage of developments in information sharing; encouraging useful additions to its scope
  • working with the IIPC, particularly the Preservation Working Group, to assess and report on tools likely to be useful in managing preservation aspects of Web archives
  • participating in development and delivery of training courses in digital preservation
  • mentoring digital preservation programs in key partner organisations such as State Libraries and libraries in the South East Asia and Pacific regions
  • preparing papers and presentations
  • identifying partners to contribute to a national effort and negotiating agreements on responsibilities and roles
  • working with government to establish effective legislative and funding frameworks for digital preservation
  • learning from the experience of others, and sharing information on our approaches and experience
  • participating in development projects with an identifiable priority benefit to the Library.

6.0 Work Plan priorities

Priorities for work in the current period appear in the Division 1 Work Plan for the relevant current year.

7.0 Appendix A

Digital preservation program roles and responsibilities

The Digital Preservation Section is responsible for:

  • understanding the nature of the digital collections across the Library
  • recognising and monitoring significant risks to the digital collections
  • understanding and where cost effective to do so, maintaining access dependencies for formats and carriers in the Library’s collections
  • establishing and specifying recommended practices for long term preservation, including preservation metadata needs, and digitisation standards
  • providing a preservation perspective on the development or acquisition of relevant infrastructure, systems and workflow
  • undertaking groundwork research and prototyping of tools and solutions to develop specifications
  • recommending and planning transformations of access dependencies
  • carrying out specialised processing of digital materials requiring special skills or equipment
  • providing advice and training to empower staff in collection and digitisation areas to deal with first line preservation issues and decisions
  • leading the development of policy and relevant strategic plans
  • monitoring and advising on the Library’s capacity to meet its preservation objectives and to operate as a trustworthy digital repository
  • providing project management and business leadership to digital preservation projects
  • maintaining the Library’s Risk Register entries relating to preservation of digital collections
  • sharing information about the Library’s digital preservation programs and needs, promoting the Library’s role and representing the Library
  • maintaining technical facilities for these responsibilities.

With regard to the Library's digital preservation programs, the Information Technology Division is responsible for:

  • providing infrastructure to manage digital collection
  • ensuring the safe ingest of data to storage
  • ensuring the safe storage and backup of digital collections
  • providing systems for recording and managing specified metadata
  • supporting and maintaining preservation workflow systems
  • providing reports on digital collections for ongoing preservation management and for audit purposes
  • providing technical leadership to digital preservation projects.

With regard to the Library's digital preservation programs, collection areas are responsible for:

  • acquiring digital collection materials in formats that facilitate preservation and with adequate documentation
  • consulting about difficult acquisition or processing decisions involving potential preservation implications
  • processing digital materials to create and maintain manageable digital collections
  • processing physical format digital materials to safe storage using agreed workflows
  • clarifying the preservation aims for digital materials in their collections
  • clarifying priorities if necessary
  • providing agreed metadata
  • programming resources for agreed collaborative projects relevant to the preservation of their digital collections.

With regard to the Library's digital preservation programs, digitisation areas are responsible for:

  • consulting with the Digital Preservation Section about proposed changes to standards, equipment or procedures likely to have an impact on preservation
  • producing digital copies in accordance with agreed standards
  • capturing specified preservation metadata
  • processing digital copies and metadata to safe storage.