Current issues in web archiving in Australia

A paper by Paul Koerbin, Digital Archiving Branch, National Library of Australia, at the Open Publish Conference 2005

Keywords

Web archiving, PANDORA, PANDAS, National Library of Australia

Introduction

This paper discusses some of the issues currently posing a challenge for web archiving in Australia. These issues can be broadly categorised as curatorial, technical and legal. The perspective from which these issues are considered is from that of a major national collecting library, the National Library of Australia, which began its web archiving initiative in 1996 and for many years now has been undertaking web archiving as a routine library activity. In other words, it is the perspective of an experienced, hands-on practitioner with a responsibility to collect and preserve Australia’s documentary heritage. The result of the National Library’s web archiving initiative is known as PANDORA, Australia’s Web Archive [1].

With the PANDORA Archive the National Library Australia has adopted an approach to archiving that is highly selective and applies procedures that put considerable emphasis on achieving high quality archiving as well as on the provision of access to the archived content. While web archiving is an established process within the National Library it remains one that is constantly challenging in regards to achieving quality web archiving outcomes on a sufficiently large scale in an efficient way.

The amount and variable quality of the content of web publishing, the technical complexity of many online resources, and the legal constraints within which this material can be acquired for preservation purposes pose significant difficulties for web archiving.

Web Publishing

In discussing web archiving it is necessary to understand what we mean by a web publication. Since the web is an open medium for disseminating information there are obviously many more types of information or data in more formats and media that are publicly available over web than we have had to deal with in the print publishing world. On the web we do find information packaged in a manner that clearly resembles print publishing, such as discrete PDF type publications or many e-journals that at least resemble the issuance methods of print journals. But so much more of what is available on the web is more difficult to define in terms of traditional publishing. Organisations, including governments, find they are easily able to put more of their administrative records online and do so. Traditionally this is information that may be of interest to organisational or government archives but not necessarily libraries. Then there is the huge body of personal, self published resources, from vanity websites to blogs. There is Internet art, music downloads, pod casts, chat rooms, forums, shops, web sites of all manner and mixture of information and file formats, databases and so on. These resources are inter-connected by hyperlinks, content is re-used, commented upon and directly referenced so that they are not necessarily definable as discrete publications; or, perhaps more correctly, they are able to be defined or delineated in many ways depending on the particular interest or perspective of the user.

For the purpose of guiding us in our web archiving work we have attempted to define a web publication. However, for this to be a practical working definition, it is necessarily broad.

A publication is information, regardless of its format or method of delivery, that is made available to the general public, or to an identified public, either free of charge or for a fee [2].

This definition would seem to include practically everything available via the Internet. However, if someone has made data available on a public medium such as the web, it is problematic to consider it anything other than published. Attempting to define a publication in this manner provides us with the broad scope of what may be of potential interest to us for web archiving and preservation – in a sense it clears away the obstacle of a restrictive, inconsistent, unsustainable or unworkable definition – but it does not in itself provide us with direction in terms of what we actually select.

Web Archiving - PANDORA Australia’s Web Archive

By web archiving we mean, of course, more than simply storing a copy of something somewhere, such as a backup on a CD or DVD or digital tape. In talking of web archiving we mean a number of activities that manage the collection, acquisition, preservation of and access to resources published on the World Wide Web [3].

As previously mentioned, the National Library has adopted selective web archiving as its primary approach. A selective approach allows us to achieve outcomes with limited resources. It necessarily involves prioritising what we do and accepting that there is much we would like to do but cannot given the scale of web archiving we undertake. Selective archiving does have the advantages of being more readily adaptable to the resources available to do the work and allowing us to attempt the more costly aspects of the archiving process [4]. These include activities such as quality analysis of harvested content and manual post-harvest work to ensure that a display (or access) copy of a harvested resource is as complete and as functional as possible. Selective archiving also allows us to seek necessary permissions from publishers in order to undertake the harvesting of their web publications and, significantly, make them accessible through the PANDORA Archive.

This approach to web archiving is one in which online resources are essentially considered as discrete documents. What has been referred to as a ‘bibliocentric’ approach [5]. This is perhaps not surprising given that the National Library has great experience in collecting and maintaining printed documents of all types. This is an important distinction to be aware of because the characteristic structure of the web does tend to work counter to this approach and some future researchers may find an archive built in this manner restrictive for their specific research objectives.

So while the dominant characteristic of the web is its interconnectedness, the present methods we use in selecting and archiving discrete online publications effectively abandons such linkages [6]. On the web the discreteness of the publications is not always self-evident; so it is the selection process that determines the parameters of the publications – its extent or discreteness, even its title – for the purpose of archiving and principal metadata description (that is, the catalogue record).

The argument may then be made as to why not adopt an archiving approach to retain these links. Some organisations undertaking web archiving do adopt this approach, and indeed the National Library has just completed its first attempt at a whole Australian domain harvest in June and July 2005 [7]. However, this is not a straightforward solution for a number of reasons: the scale of such harvesting means it is costly to undertake; providing useful access requires large scale search facilities, again costly to set up and maintain; providing access is problematic since, in the absence of legal deposit, the publications cannot be made generally available without the publisher’s permission; and, undertaking quality assurance checking and post-harvest patching or fixing is not practical. Nevertheless, there is good reason to attempt such large scale crawls periodically in order to pick up content that would not otherwise be selected for archiving and to obtain some sort of snapshot view of the web at a given time – albeit that such a snapshot, in the case of the Australian domain crawl, has a chronological dimension of five weeks.

Curatorial Issues

Digital curation, upon which web archiving is dependent, is the active process of management of the resources over time [8]. As such, all the challenges that confront us in undertaking web archiving may be seen as curatorial issues. These challenges include, for example, the significant issues associated with the long-term preservation of and access to the bitstream. However, for the purpose of this paper I wish to focus just on issues of identification, selection and responsibilities.

Identification and selection

Anything published online in Australia, and indeed anything related to Australia and published outside Australia – the location being virtually meaningless in the online world [9] – is potentially a candidate for inclusion in the PANDORA Archive. Given the large amount of candidate resources, identification and selection processes are consequently significant issues. In practice selection means prioritising in order to archive those titles considered of greatest long-term research value. This is of course a contentious procedure. In our selection processes, content is the driving factor, so publications with substantial original content are a priority. We must recognise, however, that this necessarily excludes much of what the web is about – popular, personal and commercial culture for example. We do make some effort to archive representative works but it will inevitably be inadequate for some research purposes.

Naturally other stakeholders will have different perspectives and different priorities that will not necessarily be met or be inadequately met by PANDORA as it currently functions, in its content and dimensions. For example, we are aware of not really addressing the curatorship of Internet art [10]. Researchers themselves may provide challenging concepts as to how the web may be viewed and analysed, which would require sophisticated and smart crawling and versatile user defined access paths to properly fulfil [11].

Other selective approaches such as thematic selection may be pursued but these still require considerable input of intellectual resources. In fact the PANDORA approach to selection is flexible and does incorporate thematic selection; for example in the identification and selection of sites and other web publications in relation to election campaigns. In undertaking thematic archiving within the selective approach we are also able to be highly reactive; so, for example, our attention and focus can turn to an event (e.g. Canberra bushfires, Bali bombing) in a timely manner. However, the need for more systematic, indeed automatic, identification and selection processes is critical if we are to increase the scale of selective archiving within the constraint of limited resources.

The selection process for PANDORA also adopts, when considered appropriate, what has been described as the ‘Arizona model’ [12]. This selection process involves determining an   aggregation of resources – related by logical hierarchy and common provenance – to be selected, harvested, preserved and described as an archival entity. An example may be the selection of the complete ‘publications’ section of a government web site, rather than treating the publications individually. Where the difficulty remains in respect to PANDORA when adopting an ‘Arizona model’ selection is in the description, since we generally only catalogue the high level resource. This sort of selective archiving would seem to have inherent efficiencies; and while search engines allow access to the deeper content, the curatorship of such content requires the implementation of an efficient system to describe the content of such an aggregation of resources.

In regard to identification and selection therefore, we need to meet the challenge of efficiently identifying and selecting on a larger scale. One way in which this might be achieved includes using metadata repositories that can be re-used for this purpose. For example the National Bibliographic Database may be used to identify new government online publications and these records may be re-used to register selections within the PANDORA management system [13]. There are many problems to resolve with such an approach, not the least the quality and reliability of the original metadata source. And most importantly, it does require cooperation by the original metadata creators – the authors and/or publishers – to make this work most effectively. The web archive curator also needs to work to engage the document and metadata creators in the archival process so that they can provide requisite content and/or metadata to assist in the identification, selection and acquisition of resources.

Many organisations undertaking web archiving, whatever their initial approach may have been, now recognise that adopting complimentary, or hybrid approaches, including selective, domain and thematic web archiving offer the likelihood of the best outcomes [14].

Responsibilities

The PANDORA Archive, while established and led by the National Library of Australia, operates with shared curatorial responsibilities, at least in terms of selection and acquisition. So, all mainland state libraries and the Northern Territory Library assume responsibility for online resources of primary interest to their own jurisdictions. In addition, three PANDORA partner agencies have curatorial responsibilities based on their topical expertise: ScreenSound Australia for film, radio and some music resources; the Australian War Memorial for Australian military history and veterans’ issues; and the Australian Institute for Aboriginal and Torres Strait Islander Studies for resources relating to indigenous issues and interests.

There are other initiatives in Australia undertaking responsibility for the archiving (and perhaps long-term preservation) of material of specific jurisdiction or forms. In regard to web archiving mention should be made of the State Library of Tasmania’s long established Our Digital Island [15] web archive and more recent STORS [16] electronic deposit system. Institutional digital repositories are increasingly visible especially in the higher education sector where they can manage e-prints, digital theses and other electronic publishing in which they are stakeholders. The ARROW Project [17] is working to identify and test the software solutions to support best-practice management of institutional digital repositories.

The PANDORA Archive is currently only able to manage a small, albeit targeted and valuable, part of the Australian web domain, so that the vast majority of the content of the Australian web domain is not curated for long-term preservation and access. There is therefore considerable scope and great need for stakeholders with an expertise and stake in specific aspects of web publishing – Internet art, to use an example already mentioned – to take initiative in establishing the curatorial requirements, processes and projects.

Technical Issues

Perhaps the National Library’s greatest contribution to the technical challenges of web archiving has been in the development of a workflow management system or nascent curator tool (known as PANDAS, a contraction of PANDORA Digital Archiving System). This system was developed in-house by the National Library and first implemented in June 2001 [18]. There is no doubt that this system has provided workflow efficiencies and has made possible the partnership or shared curatorial arrangements employed for PANDORA. It remains one of the few such tools specifically designed to manage remote web archiving workflows and the only one yet to be made generally available for evaluation. While it was designed specifically for the workflows developed by the National Library, the software has been used by the UK Web Archiving Consortium for their initial web archiving project [19].

PANDAS manages the essential workflows associated with web archiving, such as recording selection decisions and other administrative metadata, managing the scheduling and harvesting processes, the quality assurance processes, the preparation of archived titles for display and the maintenance of access restrictions. It interacts with other deployed software such as the actual harvesting robot.

The challenge remains, nevertheless, to develop a customisable curator tool that manages multiple workflows in a more flexible manner so that it is adaptable to various curatorial or archiving approaches. Such development is beyond the resources and business objectives of the National Library. However, PANDAS as a functional working web archiving workflow management system is well placed to inform development of a more robust curator tool being undertaken by the International Internet Preservation Consortium [20]. As such, our current development of PANDAS is focused on enhancing the workflow management capabilities of the system and preparing the application for an open source environment.

The National Library is not an information technology research organisation and therefore is not in a position to develop or enhance all the requisite tools for web archiving. The focus on the in-house development of the workflow management tool is driven by the Library’s operational and business needs. For harvesting and indexing software we are obviously dependent upon other sources. The Internet Archive has developed the Heritrix crawler as open source, extensible, archival-standard web crawling software [21] and at some stage in the near future we expect to move to using this software. The importance of the development of robust crawling mechanisms is that crawling (as distinct from depositing) web resources has many advantages. Crawling regimes can be scheduled; processes for long term preservation purposes, such as the automatic extraction of preservation metadata or storage in archival file formats, can be applied as part of the harvesting process; it requires, in most cases, little or no work on the part of the publisher; and, unlike the deposit of material, the web archive curator has more control over the acquisition process, at least in terms of timeliness.

The problem with harvesting from the web is that the crawl robots have their limitations. Links cannot always be understood and followed by the harvester and content may not be gathered. Even when content may be gathered the process of downloading the files may result in lack of functionality of some elements. Many of these problems may be able to be fixed post-harvest, but currently it is a manual (and therefore costly) process [22]. Therefore, on the technical side there is the challenge of improving the crawl robots to be better able to parse and harvest content for the Archive; and the challenge to develop automated quality checking of the completeness and functionality of the harvested content. And this is not even to touch upon the technical issues in relation to the preservation of the bitstream and the longer-term functional access to the content.

Legal Issues

The National Library has a statutory obligation under the National Library Act 1960 [23] to develop and maintain a national collection of library material, including a comprehensive collection of library material relating to Australia and the Australian people; and, moreover, to make this material available to the Australian public. The legal deposit provision in the Copyright Act 1968 – Section 201 [24] requires the delivery of library materials to the National Library, a requirement that helps the Library identify, collect and preserve as comprehensive a collection as possible. However, the definition of library material in the Act does not cover electronic resources and so the advantages for the National Library of legal deposit are not applicable in regard to acquiring web publications.

The legal situation is certainly one of the factors why a selective approach to archiving has been adopted. Only by being selective could we work on a scale that would allow us to obtain the necessary permissions and meet our objective to make the archived resources accessible. When undertaking large scale, whole (or partial) domain archiving, which allows for the possibility of more efficient harvesting and  broader archival coverage, the legal situation is clearly a significant factor inhibiting use of the content acquired in this process since it is not practical to contact each individual copyright owner. So, while larger scale harvesting may have significant advantages and efficiencies in strictly web archiving terms, this approach obviously becomes a less attractive proposition to commit the limited resources available for web archiving to, when access to such archival content is so restricted.

We obviously do all we can to respond in an appropriate and responsible manner to the intellectual property (and commercial) interests of the copyright owners, while still striving to achieve the objectives of archiving, preservation and long-term and universal access. The PANDORA Archive does not claim or assume any copyright in the resources it archives – it remains with the authors and publishers of the intellectual content – and this is noted in our standard copyright statement on the PANDORA web site [25]. Moreover we also capture any explicit copyright statement included within the resource we archive and highlight this on the access page to the archived resources. When a copyright owner requires that access to the archived resource be restricted we are able to accommodate this by applying appropriate restriction profiles. We do aim to make such restricted access as short as possible to achieve the best outcomes for both the publisher and the general public. The restriction period and conditions are negotiated with the publisher and most restriction periods are in the order of one to three years; however, in exceptional cases we have closed access for the full period of copyright. In restricting access we are currently, for technical reasons, limited to restricting the entire archived publication, even if the requirement is only to restrict a portion of that resource. We do have enhancement plans for our management system to improve this functionality to allow us to restrict specific files in the archive rather than entire instances of the resource.

It is also important to understand that when a resource has been archived in PANDORA we do not edit the intellectual content – that is, delete or edit text, images etc – an act which we would consider to constitute tampering with the provenance of the material.  With the extension of existing exemptions for libraries and archives to cover the digital environment through the Copyright Amendment (Digital Agenda) Act 2000 [26], we are able to allow Library officers access to the resource to apply preservation and maintenance processes and actions, whether that resource is under an access restriction or freely accessible.

The need for legal deposit to be extended to digital material including online resources is considered to be an essential condition and a high priority for the effective extension of the National Library’s web archiving activity.


[1] http://pandora.nla.gov.au/ (accessed on 13 July 2005)

[2] The full definition can be found in the National Library of Australia’s selection guidelines for online resources at: http://pandora.nla.gov.au/selectionguidelines.html#pubdefinition (accessed on 13 July 2005)

[3] It may be argued that other resources delivered via the Internet may or should be included in such activities, but it is the World Wide Web that is our principle concern here. Grey areas certainly exist and the PANDORA Archive does include content in the form of PDF files delivered via email rather than the web HTTP protocol. We are not considering the archiving of emails generally or other forms of communication such as SMS in the current purview of the PANDORA Archive.

[4] For a detailed analysis of the cost of selective archiving for PANDORA see: Phillips, Margaret. 2005. ‘Selective Archiving of Web Resources: A Study of Acquisition Costs at the National Library of Australia’. In RLG DigiNews, 9 (3), June 15 2005. http://www.rlg.org/en/page.php?Page_ID=20666#article0 (accessed on 13 July 2005)

[5] Cobb, Judith, Pearce-Moses, Richard, & Surface, Taylor. ECHO DEPository Project. http://www.ndiipp.uiuc.edu/pdfs/IST2005paper_final.pdf (Viewed on 13 July 2005).

[6] Access tools may be used to re-establish such links, for example by supporting a URL query API. Although this is currently not done in the PANDORA Archive it is one of the enhancements being pursued in the current redevelopment process.

[7] For a summary of the different approaches adopted by various organisations see: http://www.nla.gov.au/padi/topics/92.html. (accessed on 13 July 2005)

[8] Rusbridge, Chris et al. 2005. The Digital Curation Centre: A Vision for Digital Curation. 2005. http://www.dcc.ac.uk/docs/DCC_Sardinia_paper_final.pdf (accessed on 13 July 2005)

[9] Though not actually meaningless in respect of web archiving, particularly in regard to legal issues (see the Legal Issues section below).

[10] A forum on this issue was hosted on the Empyre discussion list in February 2005. See: http://www.subtle.net/empyre/ or http://nla.gov.au/nla.arc-44333. (accessed on 13 July 2005)

[11] For example, the ‘web sphere analysis’ concept described by Steven M. Schneider in various documents at: http://people.sunyit.edu/~steve/ (accessed on 13 July 2005).

[12] Cobb, Judith, Pearce-Moses, Richard, & Surface, Taylor. ECHO DEPository Project. http://www.ndiipp.uiuc.edu/pdfs/IST2005paper_final.pdf (accessed on 13 July 2005).

[13] This is the PANDAS workflow management system, discussed further under the Technical Issues section of the paper. See also, footnote 18 below.

[14] This is well illustrated in the Danish web archiving strategy. See: Christensen-Dalsgaard, Birte. 2004. ‘Web Archive Activities in Denmark’. In RLG DigiNews, 8 (3) June 15, 2004. http://www.rlg.org/en/page.php?Page_ID=17661#article0 (accessed on 13 July 2005)

[15] http://odi.statelibrary.tas.gov.au/ (accessed on 13 July 2005)

[16] http://www.stors.tas.gov.au/ (accessed on 13 July 2005)

[17] http://www.arrow.edu.au/ (accessed on 13 July 2005)

[18] For a detailed description of PANDAS see: Koerbin, Paul. 2004. The PANDORA Digital Archiving System (PANDAS): Managing Web Archiving in Australia: A Case Study. http://www.nla.gov.au/nla/staffpaper/2004/koerbin2.html (accessed on 13 July 2005)

[19] http://www.webarchive.org.uk/ (accessed on 13 July 2005)

[20] http://www.netpreserve.org/about/index.php (accessed on 13 July 2005)

[21] http://crawler.archive.org/  (accessed on 13 July 2005)

[22] A survey of many of the conditions found on web sites that influence the harvesting of content and the quality of an archival crawl was completed for the IIPC. See: Marill, Jennifer, Boyko, Andrew & Ashenfelder, Michael. 2004. Web Harvesting Survey. http://www.netpreserve.org/publications/iipc-r-001.pdf (accessed on 13 July 2005)

[23] http://www.austlii.edu.au/au/legis/cth/consol_act/nla1960177/s6.html (accessed on 13 July 2005)

[24] http://www.austlii.edu.au/au/legis/cth/consol_act/ca1968133/s201.html#library_material (accessed on 13 July 2005)

[25] http://pandora.nla.gov.au/pandcpyr.html (accessed on 13 July 2005)

[26] http://www.austlii.edu.au/au/legis/cth/consol_act/caaa2000294/ (Viewed 13 July 2005). Also, a fact sheet on the Act for libraries, archives, galleries and museums has been produced by the Commonwealth Department of Communications, Information Technology and the Arts and is available at: http://www.dcita.gov.au/ip/...fact_sheet (accessed on 13 July 2005)