Web archiving tools
Limit to:
Web crawlers
Web archiving comprises several processes. These processes include resource identification and selection, seeking and recording permission to archive, harvesting, quality assurance testing, storage and making the archived resources accessible. The following resources facilitate these processes, and range from web archiving management systems to tools for specific tasks, such as crawlers.
Articles
| Books
| Events
| Organisations and Websites
| Online tutorial
| Policies, Strategies & Guidelines
| Projects and Case Studies
| Bibliographies
| Glossaries
| Journals & Newsletters
| Discussion Lists
| Surveys
| Tools
| Evaluations
- Creating Preservation-Ready Web Resources
Corporation for National Research Initiatives
In: D-Lib magazine
(Date Created: Jan 2008)
(United States of America)
- This article describes a simple model for archiving "everyday websites" at the time of dissemination using the web server to prepare the site's resources for preservation.
- http://www.dlib.org/dlib/january08/smith/01smith.html
-

Dominos : a New Web Crawler's Design
Djeraba, Chabane; Hafri, Younès
(Date Created: 2004)
- This article describes 'Dominos', a high performance crawler developed as a real-time distributed system running on a cluster of machines in order to facilitate the French Web legal deposit. Various general web crawling strategies are described, including repetitive crawling, targeted crawling, and deep web crawling and the differences between Dominos and other web crawlers are outlined. A detailed technical picture of the functions and processes of the Dominos crawler and results from performance testing are included.
- http://www.iwaw.net/04/Hafri.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
-

Introduction to Heritrix : an Open Source Archival Quality Web Crawler
Avery, Dan; Kimpton, Michele; Mohr, Gordon; Ranitovic, Igor; Stack, Michael
(Date Created: 2004)
- This article provides an introduction to Heritrix, the open source archival web crawler developed by the Internet Archive. Major components of the crawler are described, its features and limitations are discussed and its performance is evaluated. Future plans, including collaboration with other institutions, are also noted.
- http://www.iwaw.net/04/Mohr.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
-

Language Engineering Techniques for Web Archiving
Coch, Jose; Masanès, Julien
(Date Created: 2004)
(France)
- This article describes the WATSON project and the use of language engineering techniques for facilitating both web archiving and archive mining by researchers. These techniques include pre-filtering of less important sites, categorisation of sites and characterisation of sites via a WATSON working station, which shows key words, key sentences and metadata information.
- http://www.iwaw.net/04/Coch.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004
- Lazy Preservation : Reconstructing Websites by Crawling the Crawlers
Bollen, Johan; McCown, Frank; Nelson, Michael L. ; Smith, Joan A.
(Date Created: Sep 2006)
(United States of America)
- This paper was presented at the Workshop on Web Information and Data Management (WIDM 2006) in November 2006. The paper describes the results of two experiments, the caching behavior of search engines and reconstructing websites using the Warrick web-repository crawler, which was developed at Old Dominion University. The authors stress that this approach should not be a substitute for a digital preservation infrastruture and policy but could be used as a last resort approach.
- http://www.cs.odu.edu/~fmccown/pubs/lazyp-widm06.pdf
- mod_oai : An Apache Module for Metadata Harvesting
Harrison, Terry L.; Liu, Xiaoming; McFarland, Nathan; Nelson, Michael L.; Van de Sompel, Herbert
(Date Created: 2005)
(United States of America)
- This article discusses mod_oai, an Apache 2.0 module that optimizes web content harvesting by building Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) capability into the Apache server. The article includes a comparison between web crawling and OAI-MPH harvesting and presents the results of experimentation using the authors' university web site as a testbed.
- http://arxiv.org/ftp/cs/papers/0503/0503069.pdf
-

Test Bed Taxonomy for Crawler
Boyko, Andrew
(Date Created: Jul 2004)
- This paper reports on the problems and issues that automated archival web crawlers encounter in the surface web within the context of harvesting, content retrieval and link detection and presentation. The 15-page document covers issues such as non-typical URIs, cookies and forms, as well as synthetic and non-HTML content and links.
- http://netpreserve.org/publications/iipc-r-002.pdf
- Version 1
-

Towards Continuous Web Archiving: First Results and an Agenda for the Future
Masanès, Julien
In: D-Lib magazine
(Date Created: Dec 2002)
(France)
- As part of its Web archiving activities, the Bibliothèque nationale de France (BnF) has investigated various ways in which the crawler programs used for the automatic collection of Web pages can be improved. In particular, the BnF initiative has tried to implement and evaluate sophisticated crawler programs that use the link structure of the Web to provide an automatic estimation of a Web page's importance. This paper describes the result of experiments whereby the automatic ranking of a sample set of Web sites (using Xyleme's ranking algorithm) was compared with manual evaluations of the same sites by librarians. There was a sufficient level of correlation from these to suggest that automatic importance evaluation could help national libraries build large-scale collections of Web resources.
- http://www.dlib.org/dlib/december02/masanes/12masanes.html
-
Web Curator Tool
Beresford, Philip
In: Ariadne
(Date Created: Jan 2007)
(New Zealand; United Kingdom)
- This paper describes the development of the Web Curator Tool, a Web harvesting management system by the National Library of New Zealand, British Library and Sytec.
- http://www.ariadne.ac.uk/issue50/beresford/
- Year of selective Web Archiving With the Web Curator at the National Library of New Zealand
Joe, Susanna; Lala, Vanita; Lee, Gillian; Paynter, Gordon
In: D-Lib magazine
(Date Created: May 2008)
(New Zealand)
- This article describes the Web Curator Tool and its use for selective archiving at the National Library of New Zealand in its first year.
- http://www.dlib.org/dlib/may08/paynter/05paynter.html
Top
Organisations and Websites
|
- mod_oai
Old Dominion University
(Regularly Updated)
(United States of America)
- Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
- http://www.modoai.org/
Top
Projects and Case Studies
|
- mod_oai
Old Dominion University
(Regularly Updated)
(United States of America)
- Mod-oai is a joint project of the Old Dominion University Digital Library Research Group, the Los Alamos National Laboratory and is funded by the Mellon Foundation. The aim of the project is to make the content from Apache Web servers accessible via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Web servers and Web crawlers would be able to selectively harvest from data repositories, for instance data gathering could be limited to files of a certain mimetype. The website provides an overview of the project including links to relevant organisations and describes members of the project team.
- http://www.modoai.org/
-
Web Curator Tool Project
(Regularly Updated)
(New Zealand; United Kingdom)
- This is the Sourceforge web page for the Web Curator Tool Project, a Web harvesting management system. The project is a collaboration between the National Library of New Zealand and the British Library and was initiated by the International Internet Preservation Consortium (IIPC). Software was developed by Sytec Resources.
- http://webcurator.sourceforge.net/
Top
- Heritrix
- Purpose: Heritrix is an open-source, extensible, web-scale, archiving quality web crawler. It was developed by the Internet Archive with the Nordic National Libraries. More information on latest developments and updates can be found at : http://freshmeat.net/projects/heritrix/? .
Download: http://sourceforge.net/projects/archive-crawler .
branch_id=46804&release_id=194813
- http://crawler.archive.org/
- Warrick
McCown, Frank
(Date Created: 2006)
(United States of America)
- This tool is designed to assist recovery of a lost web site from an archived or cached version in the Internet Archive, Yahoo, Google or Livesearch. It was built as part of a research project at Old Dominion University.
- http://warrick.cs.odu.edu/
-
Web Curator Tool Project
(Regularly Updated)
(New Zealand; United Kingdom)
- This is the Sourceforge web page for the Web Curator Tool Project, a Web harvesting management system. The project is a collaboration between the National Library of New Zealand and the British Library and was initiated by the International Internet Preservation Consortium (IIPC). Software was developed by Sytec Resources.
- http://webcurator.sourceforge.net/
Top
Related Topics: Appraisal Tools | Digital Repository Management Tools
| Electronic Journal Archiving Tools
| XML Tools
Broader Topic:
Digital Preservation Tools
NATIONAL LIBRARY OF AUSTRALIA, Canberra, ACT 2600, AUSTRALIA
Telephone + 61 2 6262 1111; Facsimile +61 2 6257 1703; Telephone Typewriter
Number: 1800 026 372
|