ISSN ISSN 1443-0568 | no. 66 | December 2003
|
Deep Web Archiving PANDORA, Australia’s Web Archive, is a collection of Australian online publications that the National Library and its partners have been building over the last eight years. It now contains around 5000 titles, many of which are archived on a regular basis to keep up to date with changing content. Until now, most of these titles have been copied to the Archive, with the permission of publishers, by using a Web ‘crawler’ or ‘robot’. The Digital Archiving System developed by the Library sends the crawler out to the specified Web address of the publisher’s site, copies the files that make up the publication, and brings them back to storage space on the PANDORA server. Staff check the files to ensure the publication is complete and then move it into the Archive for public access. In a small number of cases, publishers send the files to us, for instance, as an email attachment, and they are transferred into the Archive. There is a significant body of publications and websites, however, that crawler technology cannot penetrate. This part of the Web is sometimes referred to as the ‘deep web’ because search engines such as Google and Yahoo cannot navigate their way down through its pages to retrieve the information contained there. Access to deep web sites relies on human intelligence. For instance, it may require key words to be entered into a search engine query box to search the data contained in a database, or the selection of an option from a drop-down box. Today’s crawlers and search engines cannot yet mimic this human behaviour. One study 1 has estimated that the deep web may be 400 – 550 times larger than the surface web. For national libraries, which are responsible for documenting the published output of their respective nations, there is a pressing need to find a solution to archiving at least some of this material and ensuring long-term access to it. The National Library has commenced a research project that is investigating how the content of the databases which drive such interactive websites can be managed for long-term preservation. The key challenge is finding ways of acquiring, storing, organising, and providing access to this data when the software and hardware used by the publisher to create and maintain this database will become obsolete over time. Fortuitously, just as the Library began to embark on this research, it was invited to join the International Internet Preservation Consortium (IIPC). This is a consortium of 11 national libraries with experience in digital archiving, and the Internet Archive. The main objective of the Consortium is to work collaboratively to identify, develop and facilitate implementation of solutions for selecting, harvesting, collecting, preserving and providing access to Internet content. The activity of the Consortium is organised through six working groups that address specific areas of Internet archiving, which require solutions. One of these groups is the Deep Web Archiving Group, which the National Library of Australia has been asked to lead. Monica Berko, Director of Applications Branch, is the Library’s representative. The other main participant is the Bibliothèque Nationale de France. France
and Australia are sharing the task of developing solutions, with France
investigating automated ways of identifying deep web sites and methods
of acquiring the data, while Australia is researching how to manage the
data and provide access to it once a library has custody of it. 1 Berman, Michael K. (2001) ‘The Deep Web: Surfacing Hidden Value’ in The Journal of Electronic Publishing volume 7, issue 1. Online. Available at <www.press.umich.edu/jep/07-01/bergman.html>. See paragraph four for reference to the BrightPlanet study. |
| Contents list for this issue of Gateways |
Please address any comments and enquiries to The Editor, Gateways: mbrooks@nla.gov.au