National Library Home > News & Events
Gateways

First Whole Australian Web Domain Harvest

June and July 2005 saw the first ever large-scale harvest of the Australian web domain by the National Library of Australia. The objective was to collect, for preservation purposes, as much of the .au domain as possible within a limited crawl duration. Another objective was to gain more understanding of the issues associated with this approach to digital archiving.

This large-scale harvest follows ten years of selective archiving by the Library through PANDORA, Australia’s Web Archive <http://pandora.nla.gov.au/>. Since 1996, when PANDORA began, some 10 000 titles amounting to around 20 000 archived instances have been archived in PANDORA. These titles are made publicly available from the PANDORA website because permission to archive and provide access to them is obtained from the publishers.

While this achievement is significant, the Library has long recognised that this represents only a very small part of publishing activity on the Australian web domain. Large scale, automated harvesting to supplement the selective archiving done for PANDORA is an important step towards the preservation of Australia’s online documentary heritage.

At the request of the Library, the San Francisco based Internet Archive <http://www.archive.org/> conducted the harvest. The not-for-profit agency is experienced in large-scale web harvesting for preservation purposes and, together with the National Library of Australia, is a member of the International Internet Preservation Consortium <http://www.netpreserve.org/>.

The harvest crawl ran continuously for six weeks and resulted in the capture of more than 185 million unique documents (or files) from more than 800 000 hosts, amounting to 6.69 terabytes of data. The collection will be maintained by the National Library of Australia, but public access to the content will not be initially possible for copyright reasons. Nevertheless, providing access to this valuable resource and building upon this collection with further large-scale harvesting remain objectives of the National Library.

This first Australian domain harvest provides a valuable benchmark and foundation for the Library to develop future web archiving strategies. It also provides the basis for the development of a significant collection of Australia’s documentary heritage for future generations.

Paul Koerbin

Project Officer Australian Domain Harvest
Digital Archiving Branch
National Library of Australia