Website Archiving Policy

Purpose

This document outlines strategies to ensure that content made accessible through the National Library of Australia’s websites is archived in accordance with the following government policy and guidelines:

The strategies cover content made accessible through:

  • the Library’s public website,
  • the Library’s Intranet
  • websites hosted by the Library, and
  • websites mirrored by the Library.

As well as dealing with individual web-based resources (collections, publications and services) published on these websites they also cover the Library’s websites as records in their own right and the need to be able to reconstruct the content, look and feel of a whole website at a given point in time.

These strategies are based on archiving web-based records as they are published on the Internet. The Library has just started investigating the idea of archiving web pages as they are accessed. Outcomes of this research will be reflected in the next version of this document.

Strategies for the Library's Public Website

‘Low Business Risk’ Approach

In its Policy for keeping web-based records in the Commonwealth Government (Section 3.4) the National Archives of Australia states that:

This policy does not dictate how often agencies should make records of their web resources – for example, how often a snapshot of a website might be taken, or to what extent changes to websites should be tracked over time. Such decisions can only be made intelligently in the context of an agency’s particular business risk environment.

After assessing the business risk, the Library has decided to adopt a ‘low business risk’ approach to archiving the web-based records made accessible through its public website.

Although the content on the Library's public website is subject to frequent and irregular change and the Library aims for high public visibility, there is a low risk that the Library would be called to account, held liable or subject to litigation for the content of its website at any given time. The nature of the Library's business is perceived by the public as benign and contributing to the common good. The Library is not subject to the same degree of public scrutiny or criticism as agencies dealing with the implementation of core government policy in areas that may affect people’s lives on a day-to-day basis, such as defence, health and welfare or immigration.

The primary area of risk when publishing content on the Library’s web site has to do with intellectual property rights. To address this risk, the Library plans to implement separate policies and guidelines in the form of a rights management framework.

Standards-based Record keeping system

The Library will keep its web-based records in a record keeping system that complies with the International Standard ISO 15489, Records Management. The Library’s current record keeping system is Tower Software’s TRIM ContextÒ (http://www.towersoft.com.au/).

Separation of National and Agency Roles

The Library will maintain a separation between its national role in relation to the preservation of Australian government publications as described in Section 2.3 (http://www.naa.gov.au/recordkeeping/er/web_records/policy_chpt2.html#responsibilities) of A Policy for keeping web-based records in the Commonwealth Government and its role as a government agency that creates web-based records.

As a corollary, the Library will maintain a separation between the PANDORA Archive as a physical store for archiving significant Australian publications, the Library’s public website as a means of disseminating its own corporate and commercial online publications and TRIM as a means of archiving versions of these.

The Library will report new and updated versions of its key websites to PANDORA for archiving using the same guidelines and processes that have been developed and promulgated by the Library for other government agencies.

The Library will manage the latest versions of its key websites for persistence whether or not they are selected for the PANDORA Archive.

The Library will keep older versions of key websites in TRIM and retain them as national archives in accordance with the Administrative Functions Disposal Authority (February 2000).

For what constitutes a key website in this context see Section 2.5 below.

Separation of Web Delivery and Record Keeping Roles

The aim of the Library’s website is to operate as a primary means of online service delivery. The website is not an alternative to the appropriate archiving of content on TRIM. This means that, although the Library will manage its website for persistence, over time content may be replaced by later versions at the same web address or by a redirect to another address.

As a corollary, the Library will not use its public website; nor will it use a separate dedicated web server to maintain an online archive of all material posted to its website.

To ensure the persistence of links content that is no longer relevant will be logically archived by removing links to the content and assigning archival headers and footers. However, this logically archived content will not replace the copy already captured in TRIM.

Content Management Framework

The Library has set up a content management framework for its public website that requires content to be managed at ‘key website’ level. A key website is either a collection of information pages or a publication or a service:

  • A Collection is a set of related pages grouped together under a single entry point. Examples include: The Library, Collections, Preservation Activities, Services for Libraries, Australian Library Collections Task Force.
  • A Publication is content that is considered of sufficient significance to catalogue and archive at either collection or item level. Examples include: media releases, staff papers, policy statements, serials, bibliographies, subject guides, directories, conferences, exhibitions and formal e-publications.
  • A Service is a facility that enables users to conduct transactions directly on line. Examples include: NLA Catalogue, Australian Libraries Gateway, Kinetica. Services may differ from publications only to the extent that they are supported by an underlying database and application software. Bibliographies, subject guides and directories may begin life as publications and be redeveloped as services in response to user needs and business priorities.

Each key website has a content owner responsible for ensuring the content is relevant, current and meets NLA Web Policies and Principles.

For key websites that consist of static web pages it will be the responsibility of each content owner to ensure that content they own is captured into TRIM as it is published on the web, along with sufficient metadata to document the date the content was posted to the website, its context and structure and its placement on the website. Procedures for archiving web-based records for dynamically delivered content have yet to be finalised (see Section 2.16).

It will also be the responsibility of each content owner to ensure that content is maintained for as long as it is required, whether for business reasons, to meet accountability requirements or to meet expectations of the Australian community.

Content to be Captured

Content to be captured will include all newly posted files and significant changes to existing files. While the process of capturing content into TRIM is still manual there will be no need to capture minor changes to correct links or spelling mistakes. However, once the Library has fully automated support for the web publication process, all changes including minor changes will be captured.

The National Archives guidelines (section 3.3) state that “where a website publication exactly replicates a printed publication, it is sufficient to maintaint he printed version of the publication as the record that is captured”. However, to keep the business rules and workflows simple, content published on the Library’s website will be captured into the TRIM as part of the web publication process even if it has already been captured in an alternative format.

The Library will treat public gateway or portal sites that contain links to other online resources as requiring long term retention to reflect the Library’s particular role as an information provider over time. However, no attempt will be made to capture or keep the linked external sites unless this is required as part of a sakekeeping arrangement (as, for example, in the case of PADI).

The Library uses its public website to conduct the business of committees and working groups in a password-controlled environment. It also hosts a number of list servers with web-based records that are password-controlled. The Library will apply the same archival strategies to content that is password-controlled as it does to other key websites.

Website Content Change Control System

When business priorities allow it the Library will implement a Website Content Change Control System that:

  • Provides metadata support for the Content Management Framework.
  • Provides workflow support for publishing web-based content.
  • Prevents direct update of the production website.
  • Provides automated support for archiving content as it is published on the web.
  • Generates an activity log that tracks changes over time.

Regular Snapshots

As a general rule, the Library creates its websites in the form of static html pages. Capturing static pages into a TRIM is a relatively simple procedure that ensures that the content and structure of the document itself is preserved; but it does not give a context for the document in relation to the rest of the website. In addition, images, style sheets, server-side scripts and templates may all be needed to regenerate the page in its published form.

As well as archiving web records as they are published on the web, the Library will ensure that regular snapshots are taken of the whole public website in such a way that it can be reconstructed as a published instance.

In accordance with the low business risk approach the Information Management Branch will take such snapshots every six months, in June and December.

In addition, the Information Management Branch will take a snapshot of the public website immediately before implementing a major change to the look and feel or structure of the website. A major change would be a new information architecture that resulted in the merging and replacement of numerous pages, a change in the scope of the web site or a new graphic design.

Content owners will also assess whether there is a need to take snapshots of key websites under their control before implementing a major change to the look and feel or structure of the website. This applies to major key websites such as PictureAustralia as well as smaller websites such as Community Heritage Grants or Document Supply Services.

For key websites that use standard templates, stylesheets, images, headers and footers and server-side scripts, these will be stored at the highest level in the folder structure and captured as part of regular snapshots. For key websites with their own branding these objects will be stored within the directory structure for the key website to ensure that all relevant objects are captured as part of the snapshot.

TRIM can deliver the equivalent of snapshots if all the data objects making up the website are captured, together with appropriate metadata documenting the management history and context of the objects. Snapshots of the website as a whole and of individual key web sites could therefore be compressed and stored in TRIM; however, there may be advantages to storing the snapshots on the Library’s Digital Object Storage System (DOSS). As part of development of the Content Change Control System the appropriate place for storing snapshots will be determined and workflows developed to automate this process. In either case, a metadata record of the snapshot will be held on TRIM.

The snapshots, in combination with routine archiving of new and updated content in TRIM as it is made available on the web will ensure that it is possible to establish the content of the public website or of a given key website to an adequate level for record keeping purposes at any particular point in time.

Once the Website Content Change Control System is in place, a copy of the activity log covering the period between snapshots will also be captured into the TRIM with appropriate metadata.

The closest snapshot in time will be used to establish the context and look and feel of content that has been changed between snapshots.

Snapshots of theLibrary’s public web site will be retained as national archives in accordance with the Administrative Functions Disposal Authority (February 2000).

The use of snapshots will be reviewed against alternative methods for establishing the content of websites as part of implementing the new Website Content Change Control system.

Metadata

Key web sites will be described using the Australian Standard on Metadata (AS 5044) as outlined in the Library’s Metadata Strategies and Guidelines, Available: http://www.nla.gov.au/metadata.html). Web-based records and site snapshots will be described using the guidelines set by the Records Management Unit for all records captured into TRIM.

When business priorities allow it, the Library will extend its Web Site Content Change Control System to enable Australian Government Locator Service (AGLS) and record keeping metadata to be created by content creators as part of the publication process.

The following additional metadata will be recorded in the TRIM for site snapshots:

  • date and time of capture;
  • technical details about the website design;
  • details about the software used to create the content;
  • details of the applications (including search engines) that supplement the web resources;
  • details about the client software needed for viewing the web resources.

Persistent Identifiers

The Library will manage its web-based content for persistence. Once published, the original URL will operate as a persistent identifier for the latest version.

In the short term the Library will ensure the persistence of the URL through policies and guidelines limiting the extent to which files are moved or deleted once they have been published on the web; or through appropriate manual or automatic redirects.

When business priorities allow it, the Library will extend its Web Site Content Change Control System to enable URLs to be resolved to new web addresses when content is moved or replaced by content at another address.

Managing objects separately

At this stage no attempt will be made to manage style sheets, images and other multimedia objects, server-side scripts, etc separately from information pages. These objects will be captured into the TRIM only through snapshots.

When business priorities allow it, the Library will extend its Web Site Content Change Control System to enable such objects to be managed at file level.

As an exception to this rule, images and other multimedia objects that form part of the Library’s digital collections will be managed through the Digital Collections Manager (DCM) and archived separately on the Digital Object Storage System (DOSS). Such objects will be referenced by their persistent identifiers in web content that is captured into TRIM and will not need to be captured in their own right.

Images to be maintained through the Digital Collections Manager will include images that form part of the Library’s internal photographic collection used on the web site and images of the Library’s print publications created for the Online Shop.

Browser software, Search Engines and Plugins

At this stage no attempt will be made to capture copies of browser software, search engine software or plug-ins needed to ensure that a fully functional website can be reconstructed from a snapshot.  In its national role, the Library is monitoring the need to set up such a software repository but this would not be part of its current agency role.  However, the metadata recorded when snapshots are taken will document the versions in use at the time of the snapshot so that links can be made to such a repository in the future.

Transaction logs

The Library maintains website transaction logs for a period of 2-3 years for business management purposes.  These logs are not currently captured into the TRIM. As part of implementing its future Business Intelligence System the Library will explore the most appropriate way of configuring the raw log files to produce comprehensive reports for record keeping purposes.

Security audit trails

Security audit trails will be maintained for all the Library’s web-based services requiring user access management in accordance with Section 4.4 of the Library’s Information Technology Security Policy. This states that: “Where there is a business need to retain the data from a records management perspective (for example, as evidence of unauthorised access to the Library’s financial management system) the Library will adhere to the seven year retention period specified in the Administrative Functions Disposal Authority, Class 2099.”

Web-based Transactions

As well as archiving web-based forms, content owners will ensure that full and accurate records of web-based transactions are captured into TRIM or an alternative database management system in a form that guarantees the authenticity, reliability and accessibility of the records.

Transactions captured into a database management system will be archived in accordance with strategies and guidelines under development for all of the library’s database management systems whether or not the data was gathered via a web-based form.

Dynamically Delivered Content

Web pages are generated dynamically from a combination of databases and style sheets for some of the Library’s services. At this stage, only the static pages making up these key web sites are being archived.

Dynamically delivered content falls into the followng categories:

  • Resource discovery services that are incrementally updated but otherwise remains fairly static (for example, the Catalogue, National Bibliographic Database and Picture Australia).
  • Directory services that may be subject to frequent and irregular change (for example, Australian Libraries Gateway, What’s On).
  • Subject guides that are really publications rather than services but are generated from a database to enable component-level searching and presentation in different views.

The Library has not previously treated its electronic resource discovery services as falling within the ambit of content to be captured into TRIM. Directory services and subject guides have traditionally been included in the national archive both as records and Australian government publications.

A range of different strategies are under consideration by the Library for archiving this content but many of the issues remain unresolved.

In the future it is envisaged that the content owner, the Library’s IT Division and the Library’s Records Management Unit will work together during implementation of the service to ensure that an appropriate object-based or event-based approach is taken to capturing the content for record keeping purposes.

The Library will continue to take an active role in the development of web-based archiving technologies and solutions, with particular emphasis on the requirement to apply such solutions to the archiving of Australian online publications generally.

Disposal

The Library’s web-based records will be retained in accordance with relevant existing disposal authorities for traditional records.

As a general principle, the Library will maintain full and accurate records for as long as they are required – whether for business reasons, to meet accountability requirements or to meet expectations of the Australian community.

Exception Management

Specific key websites may have a higher inherent business risk than has been assessed for the Library’s public website as a whole. Content owners may take more frequent snapshots of the key website in order to address this risk.

There is a need to identify high risk areas on the website and individuals that have most to do with web publishing and, if necessary, to apply tighter or more rigorously managed controls in these cases.

Annual Review

Content owners will conduct an annual review of their key websites, to ensure that content has been archived in accordance with these strategies.

Storage and Preservation Strategies

The National Archives of Australia is ultimately responsible for the preservation of web-based records captured by Australian government agencies into their TRIMs. The Library will assist in the preservation task by implementing appropriate security controls, data administration measures and audit trails and by adopting media and open standards that meet longevity and migration requirements.

Strategies for the Library’s Other Websites

Intranet

Individual documents on the Library’s Intranet recording transactions or publications not created or kept in any other form will be captured and retained in TRIM, together with relevant metadata.

The Library will also take snapshots of the Library’s Intranet before making significant changes to its structure or content.

Records which document the processes involved in planning, designing, producing and maintaining the Library’s key websites will be captured and retained according to relevant classes in the Administrative Functions Disposal Authority.

Hosted Websites

The strategies outlined for the Library’s public website will also be adopted for hosted websites.

Mirrored Websites

The Library will keep records which document the processes involved in mirroring a website but will not take responsibility for capturing an keeping such websites in TRIM.

Roles and Responsibilities

The NLA Website Committee is responsible for:

  • determining the level of business risk informing the policy,
  • promulgating the policy,
  • monitoring its application, and
  • reviewing it on an annual basis.

Information Management Branch is responsible for:

  • maintaining this document,
  • setting up systems and processes to ensure that content can be captured into TRIM as it is posted to the web, and
  • ensuring that snapshots are taken of the Library’s websites at the frequencies specified.

Content owners are responsible for:

  • ensuring that these strategies are applied to the key websites that they own.

Content creators are responsible for:

  • ensuring that individual web-based records are captured into TRIM as it is posted to the web. (Until systems and processes are fully automated, Information Management Branch may do this on their behalf).

The Records Management Unit is responsible for:

  • implementing and maintaining a a TRIM that complies with the International Standard AS ISO 15489, Records Management,
  • exercising appropriate disposal authorities and other federal records management legislation,
  • assessing and testing the performance of the TRIM, and
  • providing content owners with special records management advice.

The Information Technology Division is responsible for:

  • developing and enhancing the Web Content Change Control System,
  • developing and implementing methods for archiving dynamically-generated content.
  • Date last reviewed: 23 January 2004.

Web Record Archiving Workflow

Diagram showing the web record archiving workflow for different levels of devolution of the authoring process

Back to the top