ISSN 1443-0568 | no. 67 | February 2004
Deep Web Archiving
The National Library and its PANDORA partners have been selecting and archiving Australian Internet publications for eight years now. With the permission of their publishers, publications and websites are copied into the Archive by sending a harvesting robot to the required site and harvesting the specified files. This method of acquisition depends on the robot being able to navigate html links through the publication to gain access to all of the component pages and their associated images, sound files, and such. Publications and sites of this type are said to be part of the ‘surface Web’ because search engines and harvesting robots can trawl them.
An increasing number of publications and websites, however, do not rely on html links, but are dynamically driven, underpinned by database software. Typically, in this situation, someone wanting information must enter search terms in a query box or make a selection from a drop-down menu. It requires a degree of intelligence that harvesting robots do not yet have: they cannot gain access to the intellectual content of these sites and bring the files back to the Library’s server. Publications and sites of this type are said to be part of the ‘deep Web’ because search engines and harvesting robots cannot penetrate them. One study [1] has estimated that the deep Web may be as much as 500 times as large as the surface Web.
The inability to harvest deep Web sites has prevented national libraries around the world from collecting them, and has consequently placed this significant part of our documentary heritage at risk of loss.
This year the Library has earmarked some funds to conduct research to address this problem. It has also joined the International Internet Preservation Consortium [2] (IIPC) and is taking a leading role in the IIPC Deep Web Archiving Working Group in order to cooperate with other national libraries in finding a solution.
Currently, the Bibliothèque nationale de France and the National Library of Australia are actively working together on this project. The former has taken responsibility for developing tools for acquiring databases, while the latter is developing tools to enable access to stored data.
Solutions will be shared with other Consortium members, and ultimately other national libraries as well. It is anticipated that this work will be an important contribution to the knowledge required to preserve documentary heritage in these formats.
For more information, contact Margaret Phillips, Director Digital Archiving.
mphillips@nla.gov.au
[2] The International Internet Preservation Consortium is a consortium of national libraries and the Internet Archive formed to work collaboratively to identify, develop, and facilitate implementation of solutions for selecting, harvesting, collecting, preserving and providing access to Internet content.
| Contents list for this issue of Gateways |