The Australian Government Web Archive
Collecting the government's online documentary heritage goes large scale
11 February 2015
In March 2014 the National Library of Australia made public a new web archiving service called the Australian Government Web Archive or, more succinctly, the AGWA. While the release of this service was made without great fanfare at the time, over the past year a considerable amount of new content, including retrospectively collected content, has been added, greatly enhancing the substance and value of the Archive.
Given the Library's long experience with web archiving and the continuing operation of the flagship PANDORA web archive, established in 1996, it may be useful to explain why the AGWA was established and why it was established as a service distinct from the PANDORA Archive.
One reason, though perhaps the least interesting for a general audience, relates to the technology that underlies the Library's web archiving programs. While PANDORA has been and still is a great success, it has limits which, perhaps counterintuitively, are derived from that very success in being one of the earliest established web archives in the world. Very early in the Library's web archiving program, last century in fact, we began developing our own infrastructure and specialised workflow tool, charmingly known as PANDAS, to manage PANDORA. This was before open-source, standards-based web archiving tools started to emerge and became available in the middle of the first decade of this century. The workflows, procedures and tools we developed for PANDORA became well established within the standard operations of the Library, with the result that we are rather tightly bound, somewhat hamstrung and, I might even venture, constrained by the infrastructure that was vital to the success of the web archiving program. One of the ways this is most evident is that the PANDORA infrastructure, while good for a selective small scale operation, does not lend itself to large scale bulk harvesting of web content.
So, one motivation for the AGWA was to prototype a new web archiving infrastructure within the Library using the now standard tools such as the Heritrix web harvester, WARC files and Open Wayback for delivery. To manage this we developed a basic interface to integrate the delivery of archival websites within a live website experience. This sounds simple, perhaps, but in fact it is a major challenge for a web archive service to deliver the time dimension of archived websites seamlessly and clearly to the user.
The development of AGWA, however, is not merely or even primarily an exercise in coming to grips with specific technologies and a new infrastructure. In fact there is a more strategic imperative for the AGWA—how to adequately deal with the challenges in respect to collecting government publications as they are increasingly published online?
For reasons that are perhaps self-evident, governments are turning to and relying more and more upon online publishing over print. Print is by no means dead, as my colleague the government publications manager will remind me. But the advantages to government of online publishing are great and include:
- the ability to publish quickly
- the ability and incentive to publish more (perhaps not always an advantage) including copious amounts of 'grey literature'
- the ability to disseminate more widely more effectively
- and, of course, the compelling need to publish more cost effectively (more cheaply, if you prefer).
The result of this is that there is much more government publishing available online and of course, like all online content, it remains highly vulnerable to sudden and virtual disappearance. Moreover, the drive for efficiencies and online publishing has meant that the old established mechanisms for ensuring print publications make their way to deposit libraries, such as government printers, clearinghouses and deposit schemes, have either disappeared or are greatly diminished and diminishing. Assumptions that may have been reasonable in the past, that print publications as tangible artefacts will in sufficient numbers make their way inevitably into library collections no longer hold true. Most worringly, the compelling reasons for online publishing are usually not accompanied by consideration of long term preservation and access.
Other factors driving the development of the AGWA and the timing of that development were the drive for open government articulated in the work of the Government 2.0 Taskforce, chaired by Dr Nicholas Gruen, and its 2009 report Engage: Getting on with Government 2.0. Even more important was Sir Peter Gershon's Review of the Australian Government’s Use of Information and Communication Technology in 2008 that recommended that agencies be allowed to obtain ‘opt-out’ whole-of-government ICT arrangements where there was a genuine business case to do so. This made it possible for the Library, in May 2010, to obtain approval for whole-of-government arrangements to collect, preserve and make accessible government websites without the considerable administrative burden of having to seek prior permission for each website or document, as was the case prior to this. A triumph of common sense and uncomplicated efficiencies.This simple but critical administrative arrangement made feasible the efficient large scale bulk harvesting of government websites for preservation and long term access using the AGWA infrastructure.
Government publication represents the great cornerstone of the 'grey literature' that articulates, informs and influences public policy. As such, there is a great public interest in ongoing access to the historic documents of government that have a bearing on social, economic, cultural, commercial, legal and other developments. As government publishing moves to online as the primary means of dissemination we need to meet the challenge to collect and preserve this content to continue the great collections of government print material we have.
Currently the AGWA includes content dating back to 2005 amounting to around 144 million files or 15 terabytes of data. At present the AGWA only includes Commonwealth Government websites which are collected through bulk harvests of nearly 1000 seed URLs. The scheduling of the harvests is not routinely established yet but harvests are currently being conducted roughly three times per year.
So, search for historic Commonwealth Government online content – using URLs or keywords – via the Library's AGWA website.
This post appears in our Web Archiving blog. To subscribe to future posts, add http://www.nla.gov.au/blogs/web-archiving to your RSS reader. To subscribe to all National Library of Australia blog posts, use http://www.nla.gov.au/blogs