National Library of Australia - Home page
 

 

 

Web archiving tools

Limit to: Web crawlers

Web archiving comprises several processes. These processes include resource identification and selection, seeking and recording permission to archive, harvesting, quality assurance testing, storage and making the archived resources accessible. The following resources facilitate these processes, and range from web archiving management systems to tools for specific tasks, such as crawlers.

Articles | Books | Events | Organisations and Websites | Online tutorial | Policies, Strategies & Guidelines | Projects and Case Studies | Bibliographies | Glossaries | Journals & Newsletters | Discussion Lists | Surveys | Tools | Evaluations

Articles

Creating Preservation-Ready Web Resources
Corporation for National Research Initiatives In: D-Lib magazine (Date Created: Jan 2008) (United States of America)
This article describes a simple model for archiving "everyday websites" at the time of dissemination using the web server to prepare the site's resources for preservation.
http://www.dlib.org/dlib/january08/smith/01smith.html
Safekept
Dominos : a New Web Crawler's Design
Djeraba, Chabane; Hafri, Younès (Date Created: 2004)
This article describes 'Dominos', a high performance crawler developed as a real-time distributed system running on a cluster of machines in order to facilitate the French Web legal deposit. Various general web crawling strategies are described, including repetitive crawling, targeted crawling, and deep web crawling and the differences between Dominos and other web crawlers are outlined. A detailed technical picture of the functions and processes of the Dominos crawler and results from performance testing are included.
http://www.iwaw.net/04/Hafri.pdf
This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
Safekept
Introduction to Heritrix : an Open Source Archival Quality Web Crawler
Avery, Dan; Kimpton, Michele; Mohr, Gordon; Ranitovic, Igor; Stack, Michael (Date Created: 2004)
This article provides an introduction to Heritrix, the open source archival web crawler developed by the Internet Archive. Major components of the crawler are described, its features and limitations are discussed and its performance is evaluated. Future plans, including collaboration with other institutions, are also noted.
http://www.iwaw.net/04/Mohr.pdf
This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
Safekept
Language Engineering Techniques for Web Archiving
Coch, Jose; Masanès, Julien (Date Created: 2004) (France)
This article describes the WATSON project and the use of language engineering techniques for facilitating both web archiving and archive mining by researchers. These techniques include pre-filtering of less important sites, categorisation of sites and characterisation of sites via a WATSON working station, which shows key words, key sentences and metadata information.
http://www.iwaw.net/04/Coch.pdf
This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
Lazy Preservation : Reconstructing Websites by Crawling the Crawlers
Bollen, Johan; McCown, Frank; Nelson, Michael L. ; Smith, Joan A. (Date Created: Sep 2006) (United States of America)
This paper was presented at the Workshop on Web Information and Data Management (WIDM 2006) in November 2006. The paper describes the results of two experiments, the caching behavior of search engines and reconstructing websites using the Warrick web-repository crawler, which was developed at Old Dominion University. The authors stress that this approach should not be a substitute for a digital preservation infrastruture and policy but could be used as a last resort approach.
http://www.cs.odu.edu/~fmccown/pubs/lazyp-widm06.pdf
mod_oai : An Apache Module for Metadata Harvesting
Harrison, Terry L.; Liu, Xiaoming; McFarland, Nathan; Nelson, Michael L.; Van de Sompel, Herbert (Date Created: 2005) (United States of America)
This article discusses mod_oai, an Apache 2.0 module that optimizes web content harvesting by building Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) capability into the Apache server. The article includes a comparison between web crawling and OAI-MPH harvesting and presents the results of experimentation using the authors' university web site as a testbed.
http://arxiv.org/ftp/cs/papers/0503/0503069.pdf
Safekept
Test Bed Taxonomy for Crawler
Boyko, Andrew (Date Created: Jul 2004)
This paper reports on the problems and issues that automated archival web crawlers encounter in the surface web within the context of harvesting, content retrieval and link detection and presentation. The 15-page document covers issues such as non-typical URIs, cookies and forms, as well as synthetic and non-HTML content and links.
http://netpreserve.org/publications/iipc-r-002.pdf
Version 1
Safekept
Historical Towards Continuous Web Archiving: First Results and an Agenda for the Future
Masanès, Julien In: D-Lib magazine (Date Created: Dec 2002) (France)
As part of its Web archiving activities, the Bibliothèque nationale de France (BnF) has investigated various ways in which the crawler programs used for the automatic collection of Web pages can be improved. In particular, the BnF initiative has tried to implement and evaluate sophisticated crawler programs that use the link structure of the Web to provide an automatic estimation of a Web page's importance. This paper describes the result of experiments whereby the automatic ranking of a sample set of Web sites (using Xyleme's ranking algorithm) was compared with manual evaluations of the same sites by librarians. There was a sufficient level of correlation from these to suggest that automatic importance evaluation could help national libraries build large-scale collections of Web resources.
http://www.dlib.org/dlib/december02/masanes/12masanes.html
New Web Curator Tool
Beresford, Philip In: Ariadne (Date Created: Jan 2007) (New Zealand; United Kingdom)
This paper describes the development of the Web Curator Tool, a Web harvesting management system by the National Library of New Zealand, British Library and Sytec.
http://www.ariadne.ac.uk/issue50/beresford/
Year of selective Web Archiving With the Web Curator at the National Library of New Zealand
Joe, Susanna; Lala, Vanita; Lee, Gillian; Paynter, Gordon In: D-Lib magazine (Date Created: May 2008) (New Zealand)
This article describes the Web Curator Tool and its use for selective archiving at the National Library of New Zealand in its first year.
http://www.dlib.org/dlib/may08/paynter/05paynter.html

Top

Organisations and Websites

mod_oai
Old Dominion University (Regularly Updated) (United States of America)
Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
http://www.modoai.org/

Top

Projects and Case Studies

mod_oai
Old Dominion University (Regularly Updated) (United States of America)
Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
http://www.modoai.org/
New Web Curator Tool Project
(Regularly Updated) (New Zealand; United Kingdom)
This is the Sourceforge web page for the Web Curator Tool Project, a Web harvesting management system. The project is a collaboration between the National Library of New Zealand and the British Library and was initiated by the International Internet Preservation Consortium (IIPC). Software was developed by Sytec Resources.
http://webcurator.sourceforge.net/

Top

Tools

Heritrix
Purpose: Heritrix is an open-source, extensible, web-scale, archiving quality web crawler. It was developed by the Internet Archive with the Nordic National Libraries. More information on latest developments and updates can be found at : http://freshmeat.net/projects/heritrix/? .
Download: http://sourceforge.net/projects/archive-crawler . branch_id=46804&release_id=194813
http://crawler.archive.org/
Warrick
McCown, Frank (Date Created: 2006) (United States of America)
This tool is designed to assist recovery of a lost web site from an archived or cached version in the Internet Archive, Yahoo, Google or Livesearch. It was built as part of a research project at Old Dominion University.
http://warrick.cs.odu.edu/
New Web Curator Tool Project
(Regularly Updated) (New Zealand; United Kingdom)
This is the Sourceforge web page for the Web Curator Tool Project, a Web harvesting management system. The project is a collaboration between the National Library of New Zealand and the British Library and was initiated by the International Internet Preservation Consortium (IIPC). Software was developed by Sytec Resources.
http://webcurator.sourceforge.net/

Top


Related Topics: Appraisal Tools | Digital Repository Management Tools | Electronic Journal Archiving Tools | XML Tools

Broader Topic: Digital Preservation Tools

Home | About | Search | Browse topics | Feedback | Contributions | What's New?
NATIONAL LIBRARY OF AUSTRALIA,
Canberra, ACT 2600, AUSTRALIA
Telephone + 61 2 6262 1111; Facsimile +61 2 6257 1703;
Telephone Typewriter Number: 1800 026 372