National Library of Australia - Home page
 

 

 

Web archiving

1. Purpose of page

The emergence of the World Wide Web as a global information resource has created special challenges for the archiving of digital materials. The Web's ever-expanding size, the dynamic and ephemeral nature of its content, and how this is to be captured, stored and made accessible for the long-term are some of the key questions being addressed by electronic archiving programs.

This page serves as an introduction to some of the major archiving initiatives that have been established by national libraries around the world to preserve their country's Web heritage. The main models that have emerged for archiving Web content are listed, together with descriptive 'snapshots' of the key projects and collaborative initiatives. Links are also provided to PADI resources which provide further discussion of Web archiving issues. Information about related topics such as legal deposit, selection and preservation can be found on the relevant PADI Topic page, or by searching the PADI database.

The following countries have well-established national Web archiving programs in place or are actively pursuing relevant issues:


2. Web archiving models

In many countries, national libraries have taken the leading role in researching and experimenting with Web archiving processes, in some cases supported by a legal deposit mandate for collecting electronic resources. In addition, a number of international partnerships have also been successful in exploring and testing digital archiving theory and practice. Several models have developed for content collection which can be broadly categorised as follows:

Whole domain
The whole domain (or comprehensive) approach involves the collection of Web sites and online resources using harvesters to automatically retrieve material in broad sweeps of national Web space, such as that undertaken by the Swedish Kulturarw3 (Cultural Heritage Cubed) and the Finnish EVA project. The US-based Internet Archive is the most ambitious example of this model, an initiative which is seeking to preserve the entire content of the global Web.

Selective
Selective approaches are aimed at archiving defined portions of Web space or particular kinds of resources according to specified criteria. Selection may be based on the significance or quality of resources, their theme or topic, or by targeting a related set of Web sites. Rather than attempting to archive all content, the selective harvesting model, such as that adopted by Australia's PANDORA archive, entails the collection of specified site or document 'snapshots' at scheduled intervals.

Thematic
Another form of selective archiving involves the collection and preservation of Web content relating to a particular theme or event. The Library of Congress's Minerva project has used a thematic approach in selecting electronic publications to archive as part of its Election 2002 and Winter Olympics projects.

Deposit
In some countries publishers deposit online material based on legal or voluntary deposit codes. At this stage the deposit of Web resources by content owners is not well-established in any country, though several experimental systems are underway. A successful voluntary deposit scheme for journals in electronic form operates in The Netherlands through agreements concluded with publishers, whilst in Sweden the deposit of static and dynamic Internet resources is a legislative requirement.

Combined approaches
A growing number of Web archiving programs are concluding that no one archiving model is entirely satisfactory for preserving national online heritage. Instead, countries such as France and Denmark have found that by employing a combination of comprehensive, selective and thematic collection techniques, the optimum coverage of material is achieved.

Each of the foregoing archiving models have benefits and difficulties according to the particular Web context to which they are applied. Other strategies are also emerging for archiving Web content, including the 'by discipline' approach of some universities and research institutes, such as the DACHS (Digital Archives for Chinese Studies) project based at the University of Heidelberg. In an innovative approach, the Virtual Remote Control (VRC) program at Cornell University is monitoring changes to Web sites over time, and capturing those at risk of the loss of information as a last option.

A useful discussion on the relevant advantages and disadvantages of the various Web archiving strategies can be found in the study Collecting and Preserving the World Wide Web : a Feasibility Study Undertaken for the JISC and Wellcome Trust. Further introductions to Web archiving theory and practice are listed under Web archiving resources - Overviews.

Some Web archiving case studies have been collected on this site:

Australia | Austria | Canada | Czech Republic | Denmark | Finland | France | Germany | Japan | Lithuania | The Netherlands | New Zealand | Norway | Sweden | United Kingdom | U.S.A | Collaborative initiatives


3. Case studies of archiving approaches

Australia

The National Library of Australia has been involved in digital archiving since 1996, being one of the first national libraries to undertake investigations in this area. The major focus of its program has been the creation of the PANDORA archive, which is an archive of selected Australian online publications deemed to be of national significance and long term research value. The system operates on a model of selecting, collecting and storing Web resources from the Australian domain, in cooperation with one territory and six state libraries, the Australian War Memorial, AIATSIS (the Australian Institute of Aboriginal and Torres Strait Islander Studies) and ScreenSound Australia. To support the system the NLA has also developed the PANDAS (PANDORA Digital Archiving System) software to assist in the automatic collection, description and provision of access to the archive.

In Australia, legal deposit legislation does not yet cover electronic publications, though a voluntary deposit scheme is in place with a number of publishers. Working within the Library's resource constraints, emphasis has been placed on creating a representative, rather than comprehensive archive, and guidelines have been created for the selection of online materials for preservation. Within the NLA, members of the Library's Digital Archiving Branch regularly scan the Internet for new publications that comply with selection guidelines, and harvesting software is employed for capture of resources. Selection is also undertaken by participating institutions.

Fundamental to PANDORA's philosophy has been the provision of access to archive content which is controlled in accordance with rights permission negotiated with each publisher. A proportion of titles are regularly re-harvested to preserve changes to content and some minor migration experiments have been made. The underlying strategy has been to store material in original formats, whilst maintaining access to another copy through available technologies.

Key documents describing aspects of the Library's archiving program in more detail are available from its Digital Archiving page. Links to more specific information about Australian Web archiving initiatives are also listed under Web archiving resources - Australia.

Updated May 2004

Austria

A July 2000 amendment to Austrian media law extended legal deposit requirements to digital offline material. Networked digital resources were not included in the provisions and in response to the legislative gap, the AOLA (the Austrian Online Archive) project was established in 1999 to archive the Austrian Web space. A joint initiative of the Austrian National Library and the Department of Software Technology at the Vienna University of Technology, harvester software is used to collect the .at domain and other sites of Austrian interest through periodic snapshots of publicly available Web sites.

An initial AOLA pilot study was followed by the official first phase of the project in 2001. A partially successful attempt at gathering a snapshot of the Austrian Web space was made with the NEDLIB harvester in May of that year, but this was later aborted due to crawler instability. In the following month, a second effort using an adapted version of the Kulturarw3 Project's Combine harvester was successful, as was a further crawl in 2002. A Data Warehouse was established to analyse the characteristics of the harvested data.

At present the project is discontinued due to a lack of funding. A resumption of captures is envisaged for 2005. A combined selective and a free harvesting approach to collection is seen as the most effective strategy for the future. The archive is not yet publicly accessible, and important areas for further work include the selection of appropriate storage systems, the development of a suitable legal framework, and the choice of optimum methods for ensuring future access and preservation. The migration techniques employed for preserving resources at present are seen as a short-range solution, to which emulation methods will be added in the long-term.

Links to more specific information about Austrian Web archiving initiatives are listed under Web archiving resources - Austria.

Updated August 2004

Canada

In 1994-95, the National Library of Canada conducted an Electronic Publications Pilot Project (EPPP) to examine the issue of the deposit of online electronic publications. The final project report made recommendations on important issues such as selection, copyright, and long-term preservation of e-publications. As an outcome of the EPPP, Libraries and Archives Canada (as the NLC is now known) has continued to collect online books and periodicals on a voluntary basis through its Electronic Collection Web site. The E-Collection, as the archive is popularly known, contains publications from commercial publishers, non-trade publishers, individuals, and federal and provincial governments. At present, unrestricted access is provided to approximately 80 percent of the archive through the LAC Web site, the residual content remaining restricted due to publishers' concerns regarding unfettered public access to their publications in the E-Collection.

In the absence of specific legal deposit legislation, a selective and distributed model for collecting networked online Canadiana has been adopted by the LAC, as part of the broader objective to collect and archive all online resources originating in Canada or Canadiana published in other domains. Detailed work has been undertaken on defining collecting scope, and formal collection guidelines were issued in 1998. As part of the selection process resources are manually assessed and a collecting status is assigned to each one according to a three-tier scale. 'Archived' resources are stored on the LAC's Web site and carry a permanent commitment on the part of the LAC to preserve them. 'Served' resources are hosted by the LAC but are not endorsed with the promise of preservation, responsibility for which may be taken up by other host organisations. Those non-hosted resources for which the LAC has made an ongoing commitment to preserve a link to are assigned a 'Linked' status.

Links to more specific information about Canadian Web archiving initiatives are listed under Web archiving resources - Canada.

Updated May 2004

Czech Republic

The archive of the Czech online-born documents (known as WebArchiv) was originally launched within the R & D project funded by the Ministry of Culture in 2000 and since then it has been implemented in the National Library of the Czech Republic and funded almost exclusively through various grants. There are two other institutions involved in the project: Moravian Library, which is responsible for IT issues, and Institute of Computer Science of Masaryk University as external co-operator.

The aim of the WebArchiv project is to protect the Czech web cultural heritage. There is implemented a comprehensive solution in the field of archiving of the national web, i.e. bohemical online-born documents, including the Czech National Bibliography. Tools and methods for collecting, archiving and preserving web resources as well as providing long-term access to them are applied. Both large-scale automated harvesting of the entire national web and selective archiving are being carried out, including thematic collections on a given subject. At present these methods are tested and are a subject of further research. To run all operations in a routine way, two conditions must be met: sufficient long-term funding and legal background, primarily the legal deposit legislation.

Concerning legal issues, the current state of legislation in this field is not convenient. Legal Deposit Act doesn‘t cover online-born documents, new applicable legislation is being prepared. Nevertheless, since June 2006 the Copyright Law updated in harmony with the Directive 2001/29/EC will come into force; this means that it will be possible to open access to the whole archive from the library premises for the purpose of research or private study. Besides, agreements with publishers are made for some important resources applying selection criteria, which authorize the library to make these archived resources online accessible.

Collecting online documents is an automated process carried out by a set of software tools that harvest, index and save data in the archive according to preassigned parameters. Nowadays, most of the software tools used are open source products developed by the International Internet Preservation Consortium (IIPC) with the rest developed locally. So Heritrix tool is being used for web crawling. Harvested files including relevant metadata are saved in standardized archival formats supported by the IIPC consortium. Next to the server, the data is stored on the redundant disk array (RAID) with expected transfer to National Library’s new large data storage facility in the near future. At present (May 2006) data of approximate size of 2 TB (i.e. 26 million unique files) are stored in the archive. Another one server is used to give online access to a limited dataset whose content is covered by agreements with its original publishers. Fulltext indexing is performed by open-source system Nutch and is accessed through Nutchwax and WERA tools. The recent changes in Czech copyright law will demand for a fulltext index of the whole archive and require a new server dedicated to fulltext indexing.

International standards are being used for description and identification (MARC21, Dublin Core, ISSN and URN) and archiving (ARC). Selected online-born documents are catalogued in an ALEPH library system which supports Z39.50 (both client and server levels) and OAI-PMH protocols (both repository and harvesting levels with the profile for MARC21 and qualified Dublin Core); records are thus also registered in the Czech National Bibliography.

Links to more specific information about Czech Web archiving initiatives are listed under Web archiving resources - Czech Republic.

Updated May 2006

Denmark

Danish legal deposit legislation enacted in 1997 extended deposit requirements to physical format digital publications and static online documents and was openly-worded to apply to other future electronic formats. Based on this mandate the Royal Library (Kongelige Bibliotek) has been selectively collecting Web resources since 1998, with physical format electronic products, static online publications and e-journals being downloaded to the online Pligtaflevering registration system upon notification from publishers. Under Danish law publishers must supply details of electronic products being submitted to the system, from which the Royal Library extracts Internet addresses for the materials for downloading. Royal Library staff than select, check, catalogue and transfer the collected materials to an archival server.

In a parallel initiative in 2001, the Danish netarchive.dk was established by the Royal Library in collaboration with the State Library (Aarhus) and Center for Internet Research, University of Aarhus, as a testbed for various archiving strategies, and to assess the usefulness of the collected material for researchers. The collection of Internet coverage of the 2001 Danish municipal elections was used as a case study for the first phase of the project, which ran to July 2002. Legal impediments prevented direct capture of content from the Danish Web, and access and harvesting frequency agreements were instead negotiated with owners, whose sites were collected using appropriate software.

Work on the second phase of the project has continued with the aim of refining the processes identified in the pilot and making recommendations for requisite amendments to Danish legal deposit regulations. Once specific legal deposit legislation has been enacted for networked material, a combined selective and harvesting approach to collecting has been recommended as a future path. This would involve the collection of material via three main channels: an annual harvest of Internet coverage of one major event or theme (an 'event-based' harvest), the ongoing selective harvesting of sites of key interest, and the automatic collection of quarterly snapshots of the Danish Web. It is envisaged that the State and University Library would take responsibility for the first two methods, with the Royal Library administering the latter. Present interest is focused on the release of an official response to the recommendations of a netarchive.dk report, which was presented to the Danish Ministry of Culture in April 2004.

The Royal Library has also been active in the Nordic Web Archive project.

Links to more specific information about Danish Web archiving initiatives are listed under Web archiving resources - Denmark.

Updated May 2004

Finland

In Finland, the EVA archival system to collect, register and store Internet publications and ensure their long-term preservation, ran as a joint project of libraries, publishers and expert organisations within the country, initiated in 1997. The program was led by the Helsinki University Library (National Library of Finland) in association with three Finnish universities, the Edita publishing house and CSC Scientific Computing, with the objective of collecting publicly available static Web pages and other inline material in the Finnish domain. Specific aims of the project have included the formulation of criteria for electronic document selection, the creation of a direct deposit system for publishers, and the development of a reliable long-term archive.

Building on the work of two earlier stages, the third phase of the EVA project was completed in 2001, and the resultant archival system is based on a whole domain approach to collection. Following a successful first sweep of the Finnish Web in June 2002, snapshots of the .fi Web space are now gathered periodically. In addition, partial harvests are carried out in connection with special events such as national elections. Cooperation has been obtained in capturing the national domain from the Finnish portal Info Center Finland, which forwards Finnish server locations in other domains than .fi for loading into the project harvester.

Finland has also been an active partner in the Nordic Web Archive and NEDLIB projects, with programming of the NEDLIB harvester prepared by the CSC to project specifications. Further development of the NEDLIB system was jointly undertaken by the CSC and the Helsinki University Library (HUL). Up to Spring 2004 the NEDLIB harvester was used, but it will be replaced by the Heritrix harvester, developed jointly by a consortium consisting of the Internet Archive and a number of national libraries.

The national library's right to harvest Web content will be granted in the new Copyright Act, to be approved by the Parliament in late 2004 - early 2004. The new Copyright Act and Legal Deposit Act have been developed in parallel, so that these laws refer to one another. The new Legal Deposit Act, which covers electronic resources (on-line and off-line) plus radio and TV programs, is yet to be passed from the Ministry of Education to the Parliament.

Links to more specific information about Finnish Web archiving initiatives are listed under Web archiving resources - Finland.

Updated July 2004

France

An extension of the French legal deposit law to networked digital material should be voted by the French Parliament in 2004. It will allow BnF and INA (Institut National de l'Audiovisuel, in charge of TV and Radio preservation) to harvest sites on-line as well as request a deposit from publishers, when online harvesting is impossible and make the collection publicly available on site. At present the experimental archive is not publicly accessible.

Ahead of any legislative amendments, the Bibliothèque nationale de France (BnF) has been active since 2000 in developing a combined methodology including:

  1. Automatic large scale domain crawls several times a year
  2. Continuous crawl of automatically selected sites (10% of the total)
  3. Deposit of deep web sites that can't be harvested on-line
  4. Thematic event-based collection for very ephemeral sites (that crawlers would take too long to find)

In collaboration with INRIA (the French National Institute for Research in Computer Science and Automatic Control), the BnF has been experimenting in Web collection techniques using automatic harvesting and evaluation methods. A test of automatic selection parameters based in-linking ranking (like Google) has been made.

The initial phase of the program, which ran until June 2001, focused on content gathering, involving the collection of 16 audiovisual sites with small robots to assess the limits of these crawlers. In 2002 two harvests were made: the first a thematic collection of sites relating to the French elections (1900 sites), and the second a comprehensive crawl of the .fr Web.

In a second pilot on methods for archiving hidden Web sites, which ran from 2002 to June 2003, 100 selected Web owners were approached to deposit their Web sites in the BnF for permanent archiving. A portable extraction tool (DeepArc) was developed to enable simple extraction of database to XML by the producers. This tool, currently being tested by IIPC members will be released in open-source soon. It is envisaged that harvester technology may be employed as a discovery tool to identify sites in the Deep Web of interest for collection, which would then be transferred to the BnF after a technical negotiation with site owners.

Aware of the necessity of international collaboration in the domain of Web Archiving, BnF has launched the International Web Archiving Workshop (IWAW) series, and the Web archive discussion list and actively participated in the creation of the International Internet Preservation Consortium (IIPC) that BnF coordinates.

Links to more specific information about French Web archiving initiatives are listed under Web archiving resources - France.

Updated July 2004

Germany

In 1997, whilst awaiting updating of its legal deposit legislation to include online publications, the German National Library (DDB) commenced a voluntary deposit trial for electronic materials. The project began by collecting electronic dissertations and theses and was extended in 2000 when the SpringerLink publishing house began depositing electronic journals. The scope of the program was broadened in the same year with the issue of guidelines for the deposit of networked digital publications, and the development of an automatic collection process for online materials in 2001.

A cooperative agreement between the DDB and the German publishers' union was reached in March 2002 to formalise the voluntary deposit scheme. Under the terms of the arrangement, publishers deposit their electronic resources in the Deposit.DBB.DE archive server, to the contents of which the DDB provides in-house access. Dynamic publications and databases are able to be transferred to the system subject to certain transferability criteria, including their ability to operate with standard software but independently of other servers.

An experiment in collecting dynamic Web sites was also conducted using the German government Web site (Die Bundesregierung) and HTTrack harvester technology. Crawls were performed in December 2000 and February 2001, from which it was concluded that the most efficient and effective strategy for Germany would be a legally-enforced deposit model rather than a resource intensive 'pull' approach. The studies have also highlighted the importance of ensuring the usability of archived material at an early stage in the archival process so that access problems are not obscured until it is too late for them to be resolved.

Links to more specific information about German Web archiving initiatives are listed under Web archiving resources - Germany.

Updated May 2004

Japan

In 2000 amendments were made to Japanese legal deposit law to encompass CD-ROMs and other physical format materials. The National Diet Library submitted a request to the Legal Deposit System Research Council in 2002 requesting that either legal deposit legislation be extended to online digital information, or that a legal framework be developed for their collection by the NDL.

Whilst awaiting the outcome of the Council's deliberations, the NDL has initiated three-year trial projects and studies, one of which is WARP (Web Archiving Project), an experimental investigation into the gathering and archiving of Web resources, and another is the research of the country's long-term digital preservation needs.

Through the WARP project, the NDL is collecting snapshots of Japanese Web sites and testing storage and preservation solutions, with the agreement of selected participating organisations. WARP is also being used to collect and archive electronic periodicals as the basis for an online periodicals collection. In June 2004, 600 Japanese Web sites and more than 1100 electronic journals had been collected.

Future research interest is focused on the compilation of comprehensive selection guidelines for the preservation of Japanese digital heritage, the exploration of comprehensive harvesting techniques, and the development of rendering and searching techniques.

Links to more specific information about Japanese Web archiving initiatives are listed under Web archiving resources - Japan.

Updated July 2004

Lithuania

The Martynas Mažvydas National Library of Lithuania is the legal deposit institution for Lithuanian publications. Electronic resources were included in a 1996 general deposit decree, though the exact extent of the provision was not defined. To guide the NLL in administering its national collecting responsibilities for online materials, selection criteria were compiled to include the collection of resources from the .lt domain, other documents of special quality and of acknowledged Lithuanian interest, and commercial and government e-publications. In accordance with the guidelines, physical format materials are to be deposited in a traditional manner, whilst static electronic documents will be gathered using harvesting methods. Dynamic resources and other public database content are to obtained by way of depository agreements concluded with publishers.

To support the collection of online Lithuanian material the Archive of Electronic Resources was established in November 2002 as a subsystem of the LIBIS (Lithuanian Integrated Library Information System) program. Once fully operational, it is envisaged that the AER will operate as a deposit system for online publications, based upon the NEDLIB model.

In addition, the NLL has been a participant in the NEDLIB project and the NEDLIB harvester was employed in the first sweep of the Lithuanian Web in October 2002. A second crawl in November 2002 involved the selective capture of approximately 60 registered e-periodicals. Many of these comprised dynamically generated Web pages, which would be overlooked by existing harvesting techniques. To address this issue, agreements for the deposit of e-publications were forwarded in March 2003 to the principal Lithuanian publishers. Regular harvests of the Lithuanian Web are now conducted every six months and rights clearances for each resource are being simultaneously obtained, together with metadata elements in Dublin Core standard.

Links to more specific information about Lithuanian Web archiving initiatives are listed under Web archiving resources - Lithuania.

Updated May 2004

The Netherlands

In 1995 the Depot van Nederlandse Electronische Publicaties (DNEP) project was established in The Netherlands to construct a system for the voluntary deposit of digital publications. The country is one of the few in the world that does not have legal deposit legislation, and lodgement of Dutch resources in the National Library of the Netherlands (Koninklijike Bibliotheek) is effected through voluntary agreements with publishers.

After an extended period of experimentation, the KB entered into a project with IBM in 2000 to develop a deposit archive for Dutch electronic publications based on the OAIS framework. The resultant e-Depot was introduced at the KB in 2002 where it has become the official archive of the major Dutch publishing houses Kluwer and Elsevier Science, and other members of the Dutch Publishers Association. Deposit of electronic publications in the e-Depot is not restricted to publications with Dutch imprint, but is based on agreement with publishers. While Elsevier and Kluwer were Dutch publishers in origin, new arrangements have been drawn up with international publishers that do not have a specific connection with the Netherlands, like the open access publisher BioMed Central and Blackwell Publishing. Now fully operational, work is continuing on the development of a Long-Term Preservation Module to provide access to stored data, manage metadata, and enable migration and emulation strategies to be effected. The Dutch research initiatives have been extensively documented, and a number of reports and a project overview have been produced by the working group.

In another collaboration, the KB was the lead partner in the NEDLIB project, which ran between 1998 and 2000. Through this program the KB trialled harvesting of the Dutch domain using the NEDLIB harvester and participated in the development of guidelines for online deposit systems.

Links to more specific information about Dutch Web archiving initiatives are listed under Web archiving resources - The Netherlands.

Updated July 2004

New Zealand

The National Library of New Zealand has been formally researching Web archiving since 2002 (although earlier tests were undertaken during the 1999 election).

Harvesting experiments have been undertaken using HTTrack technology and have targeted the 2002 national election, the America's Cup regatta held in Auckland and government web sites. Over 400 discrete objects and full web sites have been captured and stored since December 2002. In important supporting research, the Library has developed a preservation metadata model, the full version of which was issued in July 2003.

The National Library of New Zealand Act 2003 expanded the scope of the Library's legal deposit mandate to include online and offline electronic resources including both publicly available and 'deep-web' Web resources. In anticipation of the legislative amendments, the NLNZ began actively developing its processes for the selection, acquisition and archiving of physical format and online electronic documents into its existing workflows.

In 2003 a selection database was created for recording selection decisions and in early 2004 the Library began researching available web archiving software including the National Library of Australia's PANDAS technology.

In May 2004, following a major review of the NLNZ's digital library directions, substantial funding was announced for the establishment of a New Zealand digital archive to safeguard the national output of online and offline electronic resources which will include web archiving activities.

The National Library of New Zealand's web archiving approach will comprise a combination of snapshot harvesting of the whole .nz domain at defined intervals, selective harvesting of specific web sites or parts of web sites, and event harvesting of resources associated with topics such as the national elections and the America's Cup regatta.

Links to more specific information about New Zealand Web archiving initiatives are listed under Web archiving resources - New Zealand.

Updated July 2004

Norway

The National Library of Norway started the Paradigma Project in August 2001 and it is scheduled to run until 31 December 2004. The main goal of the initiative is to ensure the satisfactory legal deposit of Norwegian digital documents, and this includes the development of the technology, methodology and routines for the selection, collection, description and identification of all types of digital documents - including dynamic Internet documents.

The Library is also working to give users access to its Internet archive in compliance with the Legal Deposit Act, for purposes such as research and documentation. Considered extremely modern when it came in to effect in 1989, the current Legal Deposit Act, covers all generally available Norwegian documents stored on any medium: including paper, microforms, photographs, combined documents, sound fixations, films, video, digital documents and broadcasting programs. Documents published abroad for Norwegian publishers and those specially adapted for a Norwegian public are also covered.

Based on the Legal Deposit Act and recommendations from the Paradigma Project, the National Library has started the general harvesting of all generally available digital documents from the Norwegian Web space ('.no'). The first harvesting round was carried out in December 2002-January 2003, and it resulted in approximately 3.1 million URLs. The second harvesting round in August 2003 resulted in approximately 4.1 million URLs. The third harvesting round is currently underway, and in time, documents found on domains such as '.com', '.org' and '.net', will also be harvested.

An extremely challenging issue is the deposit of resources from the Deep Web, such as Internet newspapers, streaming media, documents from web cameras, interactive media and e-materials of all types stored in databases. The Paradigma Project has started the daily collection of approximately 65 Internet newspapers, and it will be downloading several entire newspaper databases in the near future, thus complementing the daily 'snapshots'. The National Library of Norway is also discussing Deep Web problems within the framework of the International Internet Preservation Consortium.

The Paradigma Project has recommended selection criteria for digital documents based on the Legal Deposit Act and the Library's collection strategy. These criteria will influence the Library's manual routines as well as the design of its Internet archive's system architecture. The Project plans to implement a three-phase selection process, so that librarians receive technical help to find the few documents that qualify to be catalogued at some level. The first selection phase finds and collects the Norwegian and Sami documents from the Internet. The second phase gives librarians the opportunity to automatically produce ranked lists based on specific queries. These lists are based on the use of vectors containing metadata that has been automatically extracted from the collected documents. In the third phase, librarians choose specific documents from the ranked lists for manual registration at some level, using the selection criteria mentioned above.

The Paradigma Project is currently investigating metadata formats for the description of digital documents. Although many digital documents may deserve a bibliographic description, the Paradigma Project estimates that far less than 1 per cent of the material collected from the Norwegian Internet domain may ever be subject to cataloguing at some level. In contrast, 100 per cent of the Internet documents will be fully indexed with FAST indexing software after harvesting. This will allow the Library's staff and users to search the Internet archive - both via free text as well as other indices. In addition to cataloguing some documents, and indexing all documents, the Project will be harvesting existing embedded metadata as well as the Internet documents they describe, and the National Library is planning a future service that will allow publishers to generate and deliver metadata with their documents at the time of deposit.

Giving users access to the legal deposit Internet archive is a complex matter, and the National Library must find satisfactory solutions in spite of the many, and sometimes conflicting, regulations found in the Legal Deposit Act, the Copyright Act and the Personal Data Act. The Paradigma Project is currently trying to find answers to questions like: Which users can receive access to different types of digital materials? Can they access the collections from computers outside the National Library?

The Library plans to give users access to its Internet archive via the Nordic Web Archive's (NWA) Access Tool. Today, free text searching with Boolean operators, a search for a certain URL, and the presentation of document history via a timeline are standard options. The Paradigma Project will adapt this tool's interface to accommodate several special user functions, and because it is desirable to present the archived digital documents and metadata in an organised and structured way, IFLA's Functional Requirements for Bibliographic Records (FRBR) model will be used in the Internet archive's design.

Links to more specific information about Norwegian Web archiving initiatives are listed under Web archiving resources - Norway.

Updated June 2004

Sweden

From its inception in September 1996, the Swedish Kulturarw3 (Cultural Heritage Cubed) project has explored methods for the collection, preservation and provision of access to Swedish online documents. A project of the Swedish Royal Library, Kulturarw3 is engaged in the comprehensive collection of static and dynamic Internet resources from the Swedish Web.

In a significant technological advance, Sweden was the first country to investigate the use of harvester technology for archiving Web content. The initial harvest in 1997 was limited to the .se domain but subsequent crawls have been expanded to include other Swedish server addresses. Since 2002 the Royal Library has been authorised to collect Swedish Web sites on the Internet, and two to three harvests are carried out per year. The Royal Library is responsible for storage and long-term preservation of the collected resources, which may only be accessed on-site at the Library's premises. From 2003, access to the archive content has been provided using surfing software akin to the Internet Archive's Wayback Machine. Research effort is now engaged in developing a tool to enable free text searching.

The Royal Library has also been active in the Nordic Web Archive project.

Links to more specific information about Swedish Web archiving initiatives are listed under Web archiving resources - Sweden.

Updated June 2004

United Kingdom

Research and experimentation in digital archiving in the United Kingdom can be characterised as beginning with pilot initiatives involving individual institutions followed by collaboration nationally and internationally.

In 2001 the British Library carried out a six-month experiment to select and capture (but not make publicly accessible) 100 UK Websites. The Domain.uk project, as it was called, was based on obtaining explicit agreement of Website publishers and included in its scope a broad range of UK Websites of historical and cultural significance. It also included some key current events such as the General Election of 2001 and the foot and mouth disease epidemic of the same year. The success of Domain.uk led to the setting up of the British Library's current Web Archiving Programme, which is focussed on the following interlinked strands: the definition of a collection development policy for Web archiving; participation, as a lead partner in the UK Web Archiving Consortium (UKWAC); and participation in the International Internet Preservation Consortium (IIPC), led by the Bibliotheque nationale de France.

In parallel with these developments, legal deposit legislation within the UK has now become subject to the Legal Deposit Libraries Act 2003, which reached the statute book in February 2004. The Act extends legal deposit to non-print and currently comprises enabling legislation. A governmental Advisory Panel is to be set up to oversee the various stages of secondary legislation which are likely to lead to Regulations by format, e.g., offline, e-books, e-journals, Websites. Pending secondary legislation, the British Library Web archiving initiatives will be based entirely on a voluntary, rights-cleared basis.

Links to more specific information about British Web archiving initiatives are listed under Web archiving resources - United Kingdom.

Updated July 2004

United States

The Minerva (Mapping the Internet the Electronic Resources Virtual Archive) project was established by the Library of Congress in 2000 as an experiment to provide experience in the archiving of 35 selected Web sites. Snapshots of three sites for closer study were downloaded using HTTrack software and catalogued onto LC's library management system. A Web site was created to test access, and legal issues were examined in cooperation with the US Copyright Office. In a parallel study, daily snapshots of approximately 200 sites relating to the 2000 presidential election were gathered by the Internet Archive. Based on the results of these pilots, recommendations were made regarding future larger-scale archiving initiatives.

Today Minerva has developed into an ongoing thematic digital archiving program, which focuses on the harvesting of topic based public Web content. More than 35,000 sites have been archived since 2000, including the 9/11 and 9/11 Anniversary, Election 2000 and 2002, and the 2002 Winter Olympics sites, in association with the Internet Archive which acts as the contractor for crawling and capture. LC has also explored the legal aspects of collecting Web content and it has been interpreted that the Library is entitled to collect and make accessible networked materials, though this is not explicitly stated in legal deposit legislation.

It is recognised that both comprehensive and selective harvesting will need to be employed by LC in future if the volume and complexity of the United States Web is to be adequately preserved. Present directions suggest that this will best be achieved through partnerships with organisations such as the Internet Archive. As part of this strategic path, the Library of Congress is also leading a cross-sector cooperative online archiving venture known as the National Digital Information Infrastructure and Preservation Program (NDIIPP). The NDIIPP will coordinate existing digital archiving research programs across the public and private sectors in the United States to minimise replication, allocate responsibilities and to share knowledge and experience.

Links to more specific information about United States Web archiving initiatives are listed under Web archiving resources - U.S.A.

Updated May 2004

Collaborative initiatives

International Internet Preservation Consortium

(IIPC) In July 2003 the International Internet Preservation Consortium (IIPC) was created as a collaboration between the national libraries of France, Australia, Canada, Denmark, Finland, Iceland, Italy, Norway, Sweden, the British Library, the Library of Congress and the Internet Archive. Led by the Bibliothèque nationale de France, the Consortium has been formed to support and foster development in the collection, archiving and provision of access to international Internet content in the long-term. Encouraging the use of common tools and methods to facilitate interoperability, and the raising of awareness of Internet preservation issues the are other key objectives of the Consortium.

Six working groups have been established to consider issues relating to a common technical framework, researcher requirements, access tools, metrics and the evaluation of Web crawlers, the Deep Web and content management. Until the expiry of the initial three-year agreement, membership of the IIPC is limited to the founding partners, but it is hoped that more national libraries will join the Consortium after mid-2006.

Links to more specific information about the IIPC are listed under Web archiving resources - IIPC.

Updated July 2004

Internet Archive

A San Francisco based non-profit commercial venture, the Internet Archive is comprehensively harvesting and preserving publicly available materials from the global Web, and has amassed the largest collection of public Web pages in the world since it began its pioneering activity in 1996. The core of the Internet Archive's collections are received from Alexa Internet, a commercial crawler and research company which donates collected data to IA immediately following each crawl period. Broad snapshot crawls are made every two months, and the content of the archive was made publicly accessible in 2001 via the 'Wayback Machine'.

This whole domain approach is based on periodic harvesting, in which snapshots of Web sites of all levels of quality and subject scope are collected and stored for future reference. As many sites as possible are collected but not all of these in their entirety, currently a total of 45 million sites. In tandem some narrow crawls are made of selected sites to collect their entire content.

More recently, the Internet Archive has also entered into partnerships with other institutions to develop special collections according to the institution's profile. The IA has also be active in trying to enter into collaborations with national institutions in order to collect and build interoperable national heritage collections, using IA technical expertise and data collection based on each country's specifications.

Links to more specific information about the Internet Archive are listed under Web archiving resources - Internet Archive.

Updated July 2004

NEDLIB

Funded by the European Commission, the NEDLIB (Networked European Deposit Library) Project ran between 1997 and 2000 with the purpose of developing harvesting software specifically for the collection of Web resources for a European deposit library. Harvesting technology for indexing Web content had been in place since the mid-1990s, but NEDLIB was one of the first crawlers to be specifically developed for archiving purposes. The new software was created by the Finnish IT Center for Science (CSC), to a specification formulated by the multiple project partners (a consortium of eight European national libraries, a national archive, two IT companies and three publishing houses).

In January 2000, the first version of the NEDLIB harvester was released and tested by project members and the national libraries of Estonia and Iceland. An updated system was tested on the Icelandic Web in January 2001, and with further modifications on the Finnish domain in 2002. The NEDLIB initiative revealed that whilst it is a fairly simple undertaking to design a basic harvester, it is a very complex task to develop software that is efficient, robust, and that can be programmed to capture every nuance of selection. Although the project is now defunct, the NEDLIB site is managed by the Koninklijke Bibliotheek (the National Library of the Netherlands). The NEDLIB harvester can still be found at http://web.archive.org/web/20051217014839/www.csc.fi/sovellus/nedlib/. The latest version was published in September 2002. The application is no longer supported or developed; in Finland, it will be replaced by Heritrix in Summer 2004.

Links to more specific information about the NEDLIB project are listed under Web archiving resources - NEDLIB.

Updated July 2004

Nordic Web Archive

The Nordic Web Archive is a cooperative project of the national libraries of Denmark, Finland, Iceland, Norway and Sweden for the sharing of experience in Web archiving. The major focus of the collaboration has been the development of an interoperable toolset for providing long-term access to Web archives. Funding was received in August 2000 for the first phase of the program, which is administered from the National Library of Norway.

Digital archives exist in all the participating countries, the content of which is transferred to the central NWA. A software package known as the NWA toolset has been developed using expertise from each partner library to enable navigation and searching of archived content in both time and space.

The second phase of the NWA project began in late 2002, and is focused on the enhancement of software stability, the need for additional user functionality, and the re-evaluation of the user interface.

Links to more specific information about the Nordic Web Archive are listed under Web archiving resources - Nordic Web Archive.

Updated May 2004

UK Web Archiving Consortium

Comprising six leading UK institutions, the UK Web Archiving Consortium (UKWAC) will work, with the permission of rightsholders, on an experimental system for archiving selected key UK Websites - ensuring that invaluable scholarly, cultural and scientific resources remain available for future generations.

The UKWAC - comprising The British Library, Joint Information Systems Committee of the Higher and Further Education Councils (JISC), The National Archives, The National Library of Wales, The National Library of Scotland and the Wellcome Trust - will run for an initial period of two years, during which approximately 6,000 websites will be collected and archived.

Consortium members will obtain the permission of website owners to archive selected sites whilst working collaboratively to explore how to develop compatible selection policies and to investigate the complex technical challenges involved in collecting and archiving Web material.

Each consortium member will select and 'capture' content relevant to its subject and/or domain. For example, The British Library will archive sites reflecting national culture and events of historical importance. These could include Web pages focusing on key events in national life, museum Web pages, e-theses, selected blogs to support research material and Web-based literary and creative projects by British subjects.

Wellcome will preserve a record of medicine on the Web whilst The National Archives will focus on archiving selected material from six main clusters of government departments. The Scottish and Welsh National Libraries will collect material reflecting the culture and history of Scotland and Wales and JISC will preserve Websites from leading-edge, innovative ICT projects in UK Higher and Further Education.

Infrastructure costs, such as software, hardware, and ongoing technical development and support will be shared equally amongst the Consortium members. UKWAC will use HTTrack - the open source Web crawler - to acquire films for storage. The software to carry out the archiving processes - PANDORA Digital Archiving System (PANDAS) - has already been developed and tested by the National Library of Australia and its partners for archiving Australian Websites and making them accessible through PANDORA.

UKWAC members have selected Magus Research Limited to help extend the PANDAS software for UK needs and provide the shared hardware and technical support they require.

Updated July 2004

4. Resources

Web archiving resources - Overviews

Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/Webarchive.html

Archiving the World Wide Web
http://www.clir.org/pubs/reports/pub106/Web.html

Management of Networked Electronic Publications : A Table of Status in Various Countries
http://www.collectionscanada.ca/obj/r7/f2/r7-100-e.pdf

Best Practices for Digital Archiving : An Information Lifecycle Approach
http://www.dlib.org/dlib/january00/01hodge.html

Digital Preservation and Permanent Access to Scientific Information : The State of the Practice
http://www.dtic.mil/cendi/publications/04-3dig_preserv.html

Web Archiving : Bibliography
http://www.ifs.tuwien.ac.at/~aola/links/WebArchiving.html

Digital Electronic Archiving : The State of the Art and the State of the Practice
http://web.archive.org/web/20040715012747/http://www.icsti.org/Dig_Archiving_Report_1999.pdf

ERPANET Workshop - The Long Term Preservation of Databases : ERPANET Workshop Report
http://www.erpanet.org/events/2003/bern/Bern_Report_final.pdf

Collecting and Preserving the World Wide Web : a Feasibility Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_feasibility.pdf

Legal Issues Relating to the Archiving of Internet Resources in the UK, EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf

The Last Page of the Internet : the Importance of Preserving the Dynamic Aspects of the Internet
http://www.deflink.dk/upload/doc_filer/doc_alle/1023_NBR.doc


Web archiving resources - Australia

PANDORA - Preserving and Accessing Networked Documentary Resources in Australia
http://pandora.nla.gov.au/index.html

PANDORA, Australia's Web Archive, And The Digital Archiving System That Supports It http://www.digicult.info/downloads/dc_info_issue6_december_20031.pdf

Digital Archiving http://www.nla.gov.au/initiatives/digarch.html

Archiving the Web : the PANDORA Archive at the National Library of Australia http://www.nla.gov.au/nla/staffpaper/2001/cathro3.html

Report On Senior Executive Fellowship to Research Digital Archiving in National Libraries http://www.nla.gov.au/nla/staffpaper/2002/elect.html

Digital Preservation : The Australian Experience http://www.nla.gov.au/nla/staffpaper/dw001004.html

Ensuring Long-Term Access to Online Publications http://www.press.umich.edu/jep/04-04/phillips.html

Digital Archiving - Developing Policy And Best Practice Guidelines At The National Library Of Australia http://web.archive.org/web/20010303103129/www.icsti.org/icsti/2000workshop/gatenby.html

Legal Issues Relating to the Archiving of Internet Resources in the UK, EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf

Collecting and Managing Web Resources for Long-Term Access : Web Harvesting and Guidelines to Support Preservation (ICABS Actions 3.3 and 3.4) http://www.ifla.org/IV/ifla70/papers/026e-Gatenby.pdf


Web archiving resources - Austria

AOLA : Austrian On-Line Archive
http://www.ifs.tuwien.ac.at/~aola/

Uncovering Information Hidden in Web Archives: a Glimpse at Web Analysis Building on Data Warehouses
http://www.dlib.org/dlib/december02/rauber/12rauber.html

Austrian On-Line Archive : Current Status and Next Steps
http://bibnum.bnf.fr/ecdl/2001/austria/austria.pdf

Long-Term Preservation of Digital Material - Building an Archive to Preserve Digital Cultural Heritage From the Internet
http://www.ifs.tuwien.ac.at/~aola/publications/thesis-ando.pdf

Part of Our Culture is Born Digital : On Efforts to Preserve it for Future Generations
http://www.inst.at/trans/10Nr/rauber10.htm

Archiving the Internet : Challenges, Projects, and the Austrian Perspective
http://www.ifs.tuwien.ac.at/~aola/publications/slides_reichenau_01.pdf


Web archiving resources - Canada

Networked Electronic Publications : Policy and Guidelines
http://epe.lac-bac.gc.ca/100/200/301/nlc-bnc/networked_epubs-ef/9/8/index-e.html

National Library of Canada Electronic Collection
http://collection.nlc-bnc.ca/e-coll-e/index-e.htm

Electronic Publications Pilot 1996/97 : Final Report
http://epe.lac-bac.gc.ca/100/200/301/statcan/electronic-e/0009711-534-RIE.pdf

Electronic Publications Pilot Project (EPPP) : Summary of the Final Report
http://epe.lac-bac.gc.ca/100/200/301/nlc-bnc/eppp_summary-e/ereport.htm

Networked Electronic Publications Policy
http://web.archive.org/web/20041123172033/http://www.collectionscanada.ca/bulletin/015017-9905-07-e.html


Web archiving resources - Czech Republic

WebArchiv : Archive of the Czech Web
http://en.webarchiv.cz

Archiving the Czech Web : Issues and Challenges
http://bibnum.bnf.fr/ecdl/2003/index.html

Research Intent - Digital Libraries
http://wwwdata-old.muni.cz/research/cez_item.asp?ID=23

Digital Cultural Heritage and the Cooperation of National Memory Institutes
http://en.webarchiv.cz/files/dokumenty/konference/Zabickabanskabystrica2006.doc



Web archiving resources - Denmark

Netarchive.dk : ECDL Workshop, October 19th 2002
http://bibnum.bnf.fr/ecdl/2002/Kb/Kb.html

netarchive.dk - the Pilot Project
http://www.netarchive.dk/pilot-index-en.htm

netarchive.dk - 2nd phase 2004
http://netarchive.dk/index-en.php

netarchive.dk - the Pilot Project : Project Reports
http://www.netarchive.dk/rap/index-en.htm

Legal Deposit From the Internet in Denmark - Experiences With the Law From 1997 and the Need for Adjustments
http://www.deflink.dk/upload/doc_filer/doc_alle/1023_BNH.doc

Legal Deposit In Denmark - The New Law and Electronic Products http://web.archive.org/web/20070608091825rn_1/liber-maps.kb.nl/articles/dupont11.htm


Web archiving resources - Finland

EVA : The Acquisition and Archiving of Electronic Network Publications
http://www.lib.helsinki.fi/eva/english.html

EVA : Elektronisen Verkkoaineiston Hankinta ja Arkistointi : Raportteja
http://www.lib.helsinki.fi/eva/raportit.html

Harvesting the Finnish Web Space : Practical Experiences
http://bibnum.bnf.fr/ecdl/2001/finland/sld001.htm

EVA - The Acquisition and Archiving of Electronic Network Publications In Finland
http://www.lib.helsinki.fi/tietolinja/0199/evaart.html

Experiences and Conclusions From a Pilot Study : Web Archiving of the District and County Elections 2001
http://netarchive.dk/publikationer/webark-final-rapport-2003.pdf

Legal Deposit Act (Draft)
http://www.minedu.fi/export/sites/default/OPM/Julkaisut/2003/liitteet/opm_132_Legal_Deposit_Act.pdf?lang=en


Web archiving resources - France

Towards Continuous Web Archiving : First Results and an Agenda for the Future
http://www.dlib.org/dlib/december02/masanes/12masanes.html

A First Experience in Archiving the French Web
ftp://ftp.inria.fr/INRIA/Projects/verso/gemo/GemoReport-229.pdf

The BnF's Project for Web Archiving
http://bibnum.bnf.fr/ecdl/2001/france/sld001.htm

International Web Archiving Workshops (IWAW)
http://www.iwaw.net

Web Archive (discussion list)
http://listes.cru.fr/wws/info/web-archive


Web archiving resources - Germany

Archive Server DEPOSIT.DBB.DE
http://deposit.ddb.de/index_e.htm

Collecting,Indexing and Archiving Net Publications
http://www.d-nb.de/eng/wir/ueber_dnb/netzpubl.htm

Submission of online publications to Die Deutsche Bibliothek
http://deposit.ddb.de/netzpub/web_abgabe_np_gesamt_e.htm

Collection of German online resources by Die Deutsche Bibliothek
http://bibnum.bnf.fr/ecdl/2001/germany/sld001.htm

Long-term archiving of digital documents : what efforts are being made in Germany?
http://caliban.ingentaselect.com/vl=9314169/cl=57/nw=1/fm=docpdf/rpsv/cw/
alpsp/09531513/v16n3/s9/p207

NESTOR : Network of Expertise in Long-term STOrage of Digital Resources - A Digital Preservation Initiative for Germany
http://www.dlib.org/dlib/april04/dobratz/04dobratz.html

NESTOR : Kompetenznetzwerk Langzeitarchivierung : Network of Expertise in Digital Preservation
http://www.langzeitarchivierung.de/index.php?newlang=eng

Long-term Preservation of Electronic Theses and Dissertations
http://edoc.hu-berlin.de/etd2003/liegmann-hans/HTML/index.html


Web archiving resources - Japan

WARP
http://warp.ndl.go.jp/

Digitizing, Archiving and Preserving Japanese Cultural Heritage
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287396&reqid=118767#feature1

Web Resources as Cultural Heritage: International Symposium on Web Archiving
http://www.ndl.go.jp/en/publication/proceedings/index.html

Legal Deposit System Council
http://www.ndl.go.jp/en/aboutus/deposit_council_book.html

Electronic Publications and the Legal Deposit System in Japan
http://www.ndl.go.jp/en/publication/ndl_newsletter/107/076.html

Electronic Publications and National Bibliography in Japan http://www.ndl.go.jp/en/publication/ndl_newsletter/112/123.html


Web archiving resources - Lithuania

Harvesting and Archiving of Electronic Resources in Lithuania : Towards Virtual Library
http://www.inforum.cz/inforum2003/prispevky/Jodelis_Remigijus.pdf


Web archiving resources - The Netherlands

e-depot and Digital Preservation
http://www.kb.nl/dnp/e-depot/e-depot-en.html

KB/IBM Long Term Preservation Study
http://www.kb.nl/hrd/dd/dd_onderzoek/reports/6-webpublications.pdf

Digital Preservation in Practice: The e-Depot at the Koninklijke Bibliotheek
http://puck.emeraldinsight.com/vl=2423084/cl=15/fm=html/nw=1/rpsv/cw/
mcb/03055728/v34n1/s3/p21

Legal Deposit of Digital Materials
http://www.kb.nl/kb/hrd/dd/dd_links_en_publicaties/publicaties/LIBER-Oltmans-input.doc

Digital Archiving : A Necessary Evil or New Opportunity?
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6W63-4BV52PK-1&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=3ab0c901795f97b1a891e20529da4e85

Digital Preservation and Permanent Access: The UVC for Images
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/uvc-ist.pdf

The Road to E-Deposit at the Koninklijke Bibliotheek
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/eskb-roadtoe-deposit3_20aug02.pdf

Permanent Archiving of Electronic Publications : Research & Practice
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/
summerschoolticer2003.pdf

Preservation Functionality in a Digital Archive
http://portal.acm.org/citation.cfm?id=996350.996416&coll=portal&dl=ACM&type=series&idx
=996350=Proceedings&WantType=Proceedings&title=International
%20Conference%20on%20Digital%20Libraries

Preserving electronic publications
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/isu350.pdf


Web archiving resources - New Zealand

National Library to Capture New Zealand's Digital Heritage
http://www.natlib.govt.nz/bin/media/pr?item=1085885702

National Library of New Zealand (Te Puna Matauranga o Aotearoa) Act 2003
http://www.natlib.govt.nz/files/Act03-19.pdf

National Library to lead electronic harvesting
http://www.natlib.govt.nz/bin/media/pr?item=1064531843

National Library of New Zealand - Digital Library Development Review
http://www.natlib.govt.nz/files/ross_report.pdf

National Library of New Zealand Preservation Metadata Schema
http://www.natlib.govt.nz/en/whatsnew/4initiatives.html#meta

Metadata Standards Framework - Preservation Metadata (revised)
http://www.natlib.govt.nz/files/4initiatives_metaschema_revised.pdf

National Library of New Zealand Preservation Metadata Data Model
http://www.natlib.govt.nz/files/nlnz_data_model.pdf


Web archiving resources - Norway

Paradigma : Preservation of Digital Material
http://www.nb.no/paradigma/eng_index.html

The Paradigma Project
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287396&reqid=118767#feature2

The Paradigma Web Harvesting Environment
http://bibnum.bnf.fr/ecdl/2003/index.html

Identification of Network Accessible Documents : Problem Areas and Suggested Solutions
http://bibnum.bnf.fr/ecdl/2003/index.html

The Paradigma Project and its Quest for Metadata Solutions and User Services http://www.ifla.org/IV/ifla70/papers/009e-Nuys.pdf


Web archiving resources - Sweden

Kulturarw3
http://www.kb.se/english/find/internet/websites/

New Decree for Kulturarw3
http://web.archive.org/web/20020803234652/http://www.kb.se/Info/Pressmed/Arkiv/2002/020605_eng.htm

The Collection of Swedish web pages at the Royal Library - The Web Heritage of Sweden
http://www.ifla.org/IV/ifla68/papers/111-163e.pdf

Kulturarw3, the Swedish WWW-Archive : Or, to Preserve the Swedish World Wide Web
http://bibnum.bnf.fr/ecdl/2001/sweden/sld001.htm

The Kulturarw3 Project - The Royal Swedish Web Archiw3e - An Example of 'Complete' Collection of Web Pages
http://www.ifla.org/IV/ifla66/papers/154-157e.htm

The Kulturarw3 Project - The Royal Swedish Web Archiw3e
http://web.archive.org/web/20060623011336/http://www.kb.se/kw3/articles/article000605.pdf


Web archiving resources - United Kingdom

Domain UK : Britain on the Web
http://bibnum.bnf.fr/ecdl/2002/uk/uk.html

Digital Preservation Activities in the UK : Building the Infrastructure
http://www.ifla.org/IV/ifla69/papers/129e-Jones.pdf

Web Archiving : a Feasibility Study for JISC and the Wellcome Trust
http://www.jisc.ac.uk/index.cfm?name=project_webarchiving

Legal Issues Relating to the Archiving of Internet Resources in the UK, EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf

Historic Change in Legal Deposit Law Saves Electronic Publications for Future Generations - Bill to Extend Legal Deposit to UK Non-Print Materials Receives Royal Assent
http://www.bl.uk/cgi-bin/press.cgi?story=1382

UK Government Web Archive
http://www.nationalarchives.gov.uk/preservation/archivedwebsites.htm


Web archiving resources - United States

MINERVA : Mapping the Internet Electronic Resources Virtual Archive
http://www.loc.gov/minerva/

Collections Policy Statement : Web Site Capture & Archiving
http://lcweb.loc.gov/acq/devpol/webarchive.html

Web Preservation Project Final Report : A Report to the Library of Congress
http://web.archive.org/web/20041106155137/http://www.loc.gov/minerva/webpresf.pdf

Preserving Our Digital Heritage : Plan for the National Digital Information Infrastructure and Preservation Program
http://www.digitalpreservation.gov/library/pdf/ndiipp_plan.pdf

National Digital Information Infrastructure and Preservation Program
http://www.digitalpreservation.gov/

Editor's Interview : National Digital Information Infrastructure and Preservation Program
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287092&reqid=118767#feature1

National Digital Preservation Initiatives: An Overview of Developments in Australia, France, the Netherlands, and the United Kingdom and of Related International Activity
http://www.clir.org/pubs/abstract/pub116abst.html

Web Preservation Projects at Library of Congress
http://www.archimuse.com/mw2003/papers/grotke/grotke.html

Collecting and Preserving the Web : the Minerva Prototype
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287468&reqid=118767#feature1

Building Thematic Web Collections: Challenges and Experiences from the September 11 Web Archive and the Election 2002 Web Archive
http://bibnum.bnf.fr/ecdl/2003/index.html

Legal Issues Relating to the Archiving of Internet Resources in the UK, EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf


Collaborative initiatives

Web archiving resources - IIPC

Netpreserve.org : International Internet Preservation Consortium
http://netpreserve.org/about/index.php

Web archiving resources - Internet Archive

Internet Archive
http://www.archive.org/

TechNotes -- The Internet Archive: An End to the Digital Dark Age
http://www.cswe.org/publications/jswe/03-2technotes.htm

Saving the Web
http://bibnum.bnf.fr/ecdl/2002/ia/ia.html

Archiving the Internet
http://web.archive.org/web/20030605113917re_/www.archive.org/
sciam_article.html

The Internet Archive, An Interview with Brewster Kahle
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287741&reqid=118767#interview

Web archiving resources - NEDLIB

NEDLIB : Networked European Deposit Library
http://nedlib.kb.nl/

Collecting and Preserving the Web: Developing and Testing the NEDLIB Harvester
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287468&reqid=118767#feature2

NEDLIB Local Situation
s http://nedlib.kb.nl/results/local_situations_v2.htm

Long-Term Preservation of Electronic Publications: the NEDLIB Project
http://www.dlib.org/dlib/september99/vanderwerf/09vanderwerf.html

Setting up a Deposit System for Electronic Publications, the Nedlib Guidelines
http://nedlib.kb.nl/results/NEDLIBguidelines.pdf

Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html

Web archiving resources - Nordic Web Archive

NWA : Nordic Web Archive
http://nwa.nb.no

Nordic Web Archive
http://www.lib.helsinki.fi/tietolinja/0100/nwa.pdf

Nordic Web Archive
http://bibnum.bnf.fr/ecdl/2003/index.html

Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html


Articles | Books | Events | Organisations and Websites | Online tutorial | Policies, Strategies & Guidelines | Projects and Case Studies | Bibliographies | Glossaries | Journals & Newsletters | Discussion Lists | Surveys | Tools | Evaluations

Articles

2007 Australian Federal Election on the Internet
Crook, Edgar (Date Created: Dec 2007) (Australia)
This paper describes the challenge of archiving web sites for the Australian 2007 Federal Election in the PANDORA web archive.
http://www.nla.gov.au/nla/staffpaper/2007/documents/Election2007.pdf
Historical 2nd ECDL Workshop on Web Archiving
Day, Michael In: Cultivate Interactive (Date Created: Nov 2002)
A report from the 2nd ECDL Workshop on Web Archiving held in Rome on the 19th September 2002 organised by the Bibliothèque nationale de France (BnF) and the Vienna University of Technology.
http://www.cultivate-int.org/issue8/ecdlws2/
3rd ECDL Workshop on Web Archiving
Day, Michael In: Ariadne (Date Created: 30 Oct 2003)
A report on the 3rd ECDL Workshop on Web Archives held in Trondheim in August 2003, describing work presented on identifiers, web archiving tools and experiences in web archiving from various countries.
http://www.ariadne.ac.uk/issue37/ecdl-web-archiving-rpt/
4th International Web Archiving Workshop
Day, Michael
The 4th International Web Archiving Workshop was held at the University of Bath in September as part of ECDL 2004. This report features a paragraph is on each paper.
http://www.ariadne.ac.uk/issue41/ecdl-web-archiving-rpt/
Historical Access to Web Archives: the Nordic Web Archive Access Project
Brygfjeld, Svein Arne (Date Created: 22 Aug 2002) (Scandinavia)
Presented at the 68th IFLA General Conference and Council held in Glasgow, 18-24th August, 2002, this paper describes a prototype access system for access to large-scale web archives, as developed by the Nordic Web Archive Access Project. The project is an initiative of the National Libraries of Denmark, Finland, Iceland, Norway and Sweden.
http://www.ifla.org/IV/ifla68/papers/090-163e.pdf
Also available in French at http://www.ifla.org/IV/ifla68/papers/090-163f.pdf
Architecture of DAMP : a System for Harvesting and Archiving Web Publications
Milinovic, Miroslav ; Topolscak, Nebojsa (Date Created: Jul 2005) (Croatia)
This article discusses the DAMP project (Digital Archive for Web Publications). DAMP is used to archive Croatian websites. It was developed by The University Computing Centre based at the University of Zagreb (SRCE) and the National and University Library (NUL) in Zagreb. The article discusses the first version of DAMP, including its architecture, the gathering process and future developments such as the instigation of an authentication and authorisation system.
http://widwisawn.cdlr.strath.ac.uk/Issues/Vol3/issue3_3_1.html
WIDWISAWN, Vol. 3, no. 3 (July 2005)
Archive Server DEPOSIT.DBB.DE
Barth, Kristina; Michel, Michaela (Regularly Updated) (Germany)
The German National Library's (Die Deutsche Bibliothek) storage system for the national collection of born digital and born again digital publications. At present German legal deposit law only requires the deposit of static electronic documents, but networked materials are also accepted into the system via voluntary arrangements with publishers. Subject to copyright limitations, the content of the DEPOSIT.DBB.DE archive can be searched and accessed via the Library's gateway and online catalogues.
http://deposit.ddb.de/index_e.htm
Archiving and Accessing Web Pages : the Goddard Library Web Capture Project
Allen, Robert B. ; Anderson, Nikkia; Hodge, Gail; Senserini, Alessandro ; Smith, Daniel In: D-Lib magazine (Date Created: Nov 2004) (United States of America)
This is a report on capturing intranet web pages by the NASA Goddard Space Flight Center. The aim of the Web Capture project was to ensure the long term preservation of this information, and to extract and index metadata in a format that is searchable. The article discusses how this was achieved including descriptions of system flow, web site selection, and the capture process. Technical limitations of the process are also discussed.
10.1045/november2004-hodge
http://www.dlib.org/dlib/november04/hodge/11hodge.html
Archiving the Canadian Web : Experiences at Library and Archives Canada
Lilleniit, Roselyn (Date Created: Aug 2007) (Canada)
This article describes current practices for selection, acquisition, management and accessibility of Canadian web sites at Library and Archives Canada and outlines future plans.
10.1300/J123v53n01_11
http://www.haworthpress.com/store/ArticleAbstract.asp?ID=101558
Available in The Serials Librarian, Vol 53, No 1/2, 20 Aug 2007, pp. 139-149
Archiving the Greek Web
Eirinaki, Magdalini; Jevtuchova, Darija; Lampos, Charalampos; Vazirgiannis, Michalis (Date Created: 2004) (Greece)
This paper describes a project to archive Greek orientated web materials using the domain name (.gr), the Greek language and Greek content as selection criteria. It describes the methodology for archiving, including details on the web crawler and semantic analysis, and presents the results of the experiments.
http://www.iwaw.net/04/Lampos.pdf
Safekept
Historical Archiving the Internet
Kahle, Brewster In: Report on the WWW 8 Conference (Date Created: Apr 1996)
Brewster Kahle discusses his rationale for an Internet Archive of public documents. He gives insights into some of the logistical issues including those of data gathering, cost, access and collecting mandate. It is intended that the Archive will provide resources for research and data analysis.
http://web.archive.org/web/20030605113917re_/www.archive.org/sciam_art
icle.html
Archiving the Slovenian Web : Recent Experiences
Grobelnik, Marko; Kavcic-Colic, Alenka (Date Created: 2004) (Slovenia)
This paper reports on a two year project ending in September 2004 which aimed to develop a national repository for the long term preservation of the Slovenian web. The two main goals of the project were to develop a methodology to archive the Slovenian web and to develop a web crawler to facilitate this.
http://www.iwaw.net/04/Kavcic.pdf
This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004.
Historical Archiving the Web - Some Legal Aspects
Kavcic-Colic, Alenka (Date Created: 21 Aug 2002)
A paper presented at the 68th IFLA General Conference and Council held in Glasgow, 18-24 August, 2002, discussing some legal aspects of archiving publications from the web. Copyright and legal deposit legislation are discussed with regard to rights implications for harvesting, public access and long-term preservation.
http://www.ifla.org/IV/ifla68/papers/116-163e.pdf
Also available in French at http://www.ifla.org/IV/ifla68/papers/116-163f.pdf
Archiving the Web : European Experiences
Hakala, Juha In: Tietolinja News (Date Created: Oct 2003)
This paper was presented at CONSAL XII, 20-23 October 2003, Brunei, and is an updated version of Collecting and Preserving the Web: Developing and Testing the NEDLIB Harvester, published in 2001. It outlines the outcomes of the NEDLIB Harvester Project for the archiving of Web resources. Some of the key issues in using this form of technology for preserving materials on the Web are reviewed. New developments are discussed including the Nordic Web Archive II project and the establishment of the International Internet Preservation Consortium (IIPC) to develop standards and tools for web archiving.
http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html
Historical Archiving the Web : Political Party Web Sites in the Netherlands
Druiven, Henk; Keyzer, André; Voerman, Gerrit; den Hollander, Frank (Date Created: Jun 2003) (Netherlands)
This article describes the Dutch project Archipol, which archives Dutch political Web sites, and examines why this information may be important to researchers in the future. Also discussed are various international initiatives to archive similar Web sites.
http://search.epnet.com/direct.asp?an=9972863&db=aph
Available in Information Services and Use, vol. 23, issue 1 (2003), ISSN 0167-5265; and by subscription online at www.ebscohost.com
Safekept
Historical Archiving the Web: The PANDORA Archive at the National Library of Australia
Cathro, Warwick; Webb, Colin; Whiting, Julie (Date Created: Jun 2001) (Australia)
This paper describes the National Library of Australia's PANDORA web archive. It includes description of the Library's framework for digital collection management, the reasons for a selective approach to web archiving, the archiving processes, persistent identifiers, access control, preservation plans and opportunities for global cooperation and information sharing.
http://www.nla.gov.au/nla/staffpaper/2001/cathro3.html
Safekept
Historical Archiving the World Wide Web
Lyman, Peter In: Building a National Strategy for Digital Preservation: Issues in Digital Media Archiving (Date Created: Apr 2002) (United States of America)
A paper in the series of six environmental scans commissioned as part of the US National Digital Information Infrastructure and Preservation Program (NDIIPP), this article examines archiving the web from several perspectives. As well as technical issues, such as those involved in defining the digital object, authenticity and provenance and the technologies needed for preservation, the article considers the roles of librarians and computer scientists in managing digital material and organisational issues such as copyright and the responsibility to preserve.
http://www.clir.org/pubs/reports/pub106/web.html
The collected papers are also available in PDF format via: http://www.clir.org/pubs/abstract/pub106abst.html
Archiving Web Site Resources : a Records Management View
Kelly, Brian; Pennock, Marueen (Date Created: 2006)
This brief paper forms part of the poster exhibition at the 15th International World Wide Web Conference in 2006. The paper describes a records management approach to the archiving and maintenance of web sites so that enough metadata is collected on the resource to ensure legal and archival requirements. The authors propose several ways to facilitate this including collaboration between record creators and managers, education and training.
http://www.dcc.ac.uk/docs/WWW2006_Archiving_Web_Site_Resources.pdf
Poster presented at the WWW 2006 conference, Edinburgh Scotland
Archiving Web Sites for Preservation and Access
Guenther, Rebecca (Date Created: Apr 2007) (United States of America)
This article examines the technical challenges in harvesting and managing web archives using as an example the Library of Congress' Minerva project.
http://web.ebscohost.com/ehost/detail?vid=12&hid=17&sid=a15a30f8-9aa2-
45e7-86ee-d138a9fb323a%40sessionmgr108
Journal of Archival Organization, April 2007, Vol 4 No1-2, p 141-166 and by subscription through EBSCOhost
Safekept
Historical Archiving Websites
Boudrez, Filip; Van den Eynde, Sofie (Date Created: Jul 2002) (Belgium)
A report from the Flemish DAVID Project examining the need for archiving of web sites, approaches for collection and management, and a range of legal issues including copyright. The report is available in PDF format.
http://www.expertisecentrumdavid.be/davidproject/teksten/Rapporten/Rep
ort5.pdf
Archiving, Indexing and Accessing Web Materials: Solutions for Large Amounts of Data
Cowart, Charles; Minor, David; Moore, Reagan; Zhu, Bing (Date Created: 23 Jun 2007) (United States of America)
This article describes two different technologies applied by the San Diego Super Computer Centre for archiving, indexing and accessing web archived materials for two projects: Library of Congress/San Diego Supercomputer Center and the National Science Digital Library.
http://iwaw.net/07/IWAW2007_minor.pdf
Paper presented at IWAW 2007.
Australian Web Domain Harvests: A Preliminary Quantitative Analysis of the Archive Data
Koerbin, Paul (Date Created: 15 Apr 2008) (Australia)
This report provides a preliminary quantitative analysis of content from three Australian web domain harvests conducted between 2005 and 2007. It reports on the size of the Australian web harvests; number and percentage of hosts and URLs by 2nd level domain; MIME types and changes between the 2006 and 2007 harvests.
http://pandora.nla.gov.au/documents/auscrawls.pdf
Safekept
Historical Austrian Online Archive Processing: Analyzing Archives of the World Wide Web
Aschenbrenner, Andreas; Rauber, Andreas; Witvoet, Oliver (Date Created: Sep 2002) (Austria)
A paper describing the use of OLAP technologies (On-Line Analytical Processing) to build a data warehouse of a web archive. Data warehousing allows exploration and analysis of technical aspects of the archive, such as the use of operating systems, scripts and file types, from which inferences of change over time may be drawn.
http://www.ifs.tuwien.ac.at/ifs/research/pub_pdf/rau_ecdl02.pdf
Safekept
Blog Today, Gone Tomorrow? Preservation of Weblogs
Entlich, Richard In: RLG DigiNews (Date Created: 15 Aug 2004)
Blogs (weblogs) are a recent development on the Internet and they are becoming more popular as a communication tool. The article outlines the nature and ephemeral life of weblogs and how content can be lost, and also looks at the nascent efforts to preserve them.
http://digitalarchive.oclc.org/da/ViewObject.jsp?objid=0000070519&reqi
d=4139
Characteristics of .au Websites: An Analysis of Large-Scale Web Crawl Data from 2005
Ackland, Robert; Bailey, Peter; Spink, Amanda (Date Created: 02 Jul 2007) (Australia)
This paper provides a preliminary analysis of .au domain websites collected by the Commonwealth Scientific and Industrial Research Organisation (CSIRO) in 2005. It is a first step in the planned ".au Census" project.
http://ausweb.scu.edu.au/aw07/papers/refereed/spink/paper.html
Paper presented at AusWeb07, the thirteenth Australasian World Wide Web Conference, Coffs Harbour NSW 30 June - 4 July 2007
Safekept
Collecting and Managing Web Resources for Long-Term Access : Web Harvesting and Guidelines to Support Preservation (ICABS Actions 3.3 and 3.4)
Gatenby, Pam (Date Created: 2004) (Australia)
This paper outlining the National Library of Australia’s recent contributions to the advancement of understanding of international Web archiving issues, was presented at the 70th IFLA Conference in August 2004. The NLA’s work on web harvesting and the preservation of digital materials as part of its ICABS responsibilities are detailed, together with its involvement in the International Internet Preservation Consortium’s (IIPC) Deep Web Working Group.
http://www.ifla.org/IV/ifla70/papers/026e-Gatenby.pdf