Web crawlers
A web crawler is a program that browses the web in an automated and systematic manner. Crawlers can be configured to do various type of crawls. The most significant is the collection of as much as possible from the web, such as that used by the Internet Archive, and crawling specific parts of the web, for example, a specific langage or domain such as .au. Most automatic crawlers follow the robot.txt rules where web site owners can indicate which pages can be crawled. A limitation of web crawlers is that they may not cover deep web resources such as databases.
Articles
| Books
| Events
| Organisations and Websites
| Online tutorial
| Policies, Strategies & Guidelines
| Projects and Case Studies
| Bibliographies
| Glossaries
| Journals & Newsletters
| Discussion Lists
| Surveys
| Tools
| Evaluations
-

Dominos : a New Web Crawler's Design
Djeraba, Chabane; Hafri, Younès
(Date Created: Sep 2004)
- This article describes 'Dominos', a high performance crawler developed as a real-time distributed system running on a cluster of machines in order to facilitate the French Web legal deposit. Various general web crawling strategies are described, including repetitive crawling, targeted crawling, and deep web crawling and the differences between Dominos and other web crawlers are outlined. A detailed technical picture of the functions and processes of the Dominos crawler and results from performance testing are included.
- http://www.iwaw.net/04/Hafri.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
-

Introduction to Heritrix : an Open Source Archival Quality Web Crawler
Avery, Dan; Kimpton, Michele; Mohr, Gordon; Ranitovic, Igor; Stack, Michael
(Date Created: 2004)
- This article provides an introduction to Heritrix, the open source archival web crawler developed by the Internet Archive. Major components of the crawler are described, its features and limitations are discussed and its performance is evaluated. Future plans, including collaboration with other institutions, are also noted.
- http://www.iwaw.net/04/Mohr.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
- Lazy Preservation : Reconstructing Websites by Crawling the Crawlers
Bollen, Johan; McCown, Frank; Nelson, Michael L. ; Smith, Joan A.
(Date Created: Sep 2006)
(United States of America)
- This paper was presented at the Workshop on Web Information and Data Management (WIDM 2006) in November 2006. The paper describes the results of two experiments, the caching behavior of search engines and reconstructing websites using the Warrick web-repository crawler, which was developed at Old Dominion University. The authors stress that this approach should not be a substitute for a digital preservation infrastruture and policy but could be used as a last resort approach.
- http://www.cs.odu.edu/~fmccown/pubs/lazyp-widm06.pdf
- mod_oai : An Apache Module for Metadata Harvesting
Harrison, Terry L.; Liu, Xiaoming; McFarland, Nathan; Nelson, Michael L.; Van de Sompel, Herbert
(Date Created: 2005)
(United States of America)
- This article discusses mod_oai, an Apache 2.0 module that optimizes web content harvesting by building Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) capability into the Apache server. The article includes a comparison between web crawling and OAI-MPH harvesting and presents the results of experimentation using the authors' university web site as a testbed.
- http://arxiv.org/ftp/cs/papers/0503/0503069.pdf
-

Test Bed Taxonomy for Crawler
Boyko, Andrew
(Date Created: Jul 2004)
- This paper reports on the problems and issues that automated archival web crawlers encounter in the surface web within the context of harvesting, content retrieval and link detection and presentation. The 15-page document covers issues such as non-typical URIs, cookies and forms, as well as synthetic and non-HTML content and links.
- http://netpreserve.org/publications/iipc-r-002.pdf
- Version 1
-

Towards Continuous Web Archiving: First Results and an Agenda for the Future
Masanès, Julien
In: D-Lib magazine
(Date Created: Dec 2002)
(France)
- As part of its Web archiving activities, the Bibliothèque nationale de France (BnF) has investigated various ways in which the crawler programs used for the automatic collection of Web pages can be improved. In particular, the BnF initiative has tried to implement and evaluate sophisticated crawler programs that use the link structure of the Web to provide an automatic estimation of a Web page's importance. This paper describes the result of experiments whereby the automatic ranking of a sample set of Web sites (using Xyleme's ranking algorithm) was compared with manual evaluations of the same sites by librarians. There was a sufficient level of correlation from these to suggest that automatic importance evaluation could help national libraries build large-scale collections of Web resources.
- http://www.dlib.org/dlib/december02/masanes/12masanes.html
Top
Organisations and Websites
|
- mod_oai
Old Dominion University
(Regularly Updated)
(United States of America)
- Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
- http://www.modoai.org/
Top
Projects and Case Studies
|
- mod_oai
Old Dominion University
(Regularly Updated)
(United States of America)
- Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
- http://www.modoai.org/
Top
- Warrick
McCown, Frank
(Date Created: 2006)
(United States of America)
- This tool is designed to assist recovery of a lost web site from an archived or cached version in the Internet Archive, Yahoo, Google or Livesearch. It was built as part of a research project at Old Dominion University.
- http://warrick.cs.odu.edu/
Top
-
CODA-WEBB: Slutrapport
Lindbäck, Lena
(Date Created: 05 May 2009)
(Sweden)
- This is the final report of the CODA WEBB project (60p) in Swedish which is in 3 parts: Strategy for archiving a website; A literature review of tools for web crawling; and a Review of "open repositories" to publish, store and give access to digital documents. Web crawlers evaluated for the LDB Centre Test platform project were HTTrack, Website Copier, Heritrix and PageNest. with Heritrix being selected.Open repository tools evaluated for the test platform were EPrints, DSpace, Fedora, Greenstone and DAITSS, with Fedora being selected.
- https://ldb.project.ltu.se/main.php/ldb.project.ltu.se/main.php/projec
ts/portalproject/docs/Publikationer/Svenska%20publikationer/CODA-WEBB. pdf?fileitem=7475400
Top
Broader Topic:
Web archiving tools
NATIONAL LIBRARY OF AUSTRALIA, Canberra, ACT 2600, AUSTRALIA
Telephone + 61 2 6262 1111; Facsimile +61 2 6257 1703; Telephone Typewriter
Number: 1800 026 372
|