Web archiving
1. Purpose of page
The emergence of the World Wide Web as a global information resource
has created special challenges for the archiving of digital materials. The
Web's ever-expanding size, the dynamic and ephemeral nature of its content, and
how this is to be captured, stored and made accessible for the long-term are
some of the key questions being addressed by electronic archiving programs.
This page serves as an introduction to some of the major archiving
initiatives that have been established by national libraries around the world
to preserve their country's Web heritage. The main models that have emerged for
archiving Web content are listed, together with descriptive 'snapshots' of the
key projects and collaborative initiatives. Links are also provided to
PADI resources which provide further
discussion of Web archiving issues. Information about related topics such as
legal deposit,
selection and
preservation can be
found on the relevant PADI Topic page, or by
searching the PADI
database.
The following countries have well-established national Web archiving
programs in place or are actively pursuing relevant issues:
2. Web archiving models
In many countries, national libraries have taken the leading role in
researching and experimenting with Web archiving processes, in some cases
supported by a legal deposit mandate for collecting electronic resources. In
addition, a number of international partnerships have also been successful in
exploring and testing digital archiving theory and practice. Several models
have developed for content collection which can be broadly categorised as
follows:
Whole domain The whole domain (or comprehensive)
approach involves the collection of Web sites and online resources using
harvesters to automatically retrieve material in broad sweeps of national Web
space, such as that undertaken by the Swedish
Kulturarw3 (Cultural Heritage
Cubed) and the Finnish
EVA project. The
US-based Internet Archive is the most
ambitious example of this model, an initiative which is seeking to preserve the
entire content of the global Web.
Selective Selective approaches are aimed at
archiving defined portions of Web space or particular kinds of resources
according to specified criteria. Selection may be based on the significance or
quality of resources, their theme or topic, or by targeting a related set of
Web sites. Rather than attempting to archive all content, the selective
harvesting model, such as that adopted by Australia's
PANDORA archive, entails the
collection of specified site or document 'snapshots' at scheduled intervals.
Thematic Another form of selective archiving
involves the collection and preservation of Web content relating to a
particular theme or event. The Library of Congress's
Minerva project has used a thematic
approach in selecting electronic publications to archive as part of its
Election 2002 and Winter Olympics projects.
Deposit In some countries publishers deposit
online material based on legal or voluntary deposit codes. At this stage the
deposit of Web resources by content owners is not well-established in any
country, though several experimental systems are underway. A successful
voluntary deposit scheme for journals in electronic form operates in
The Netherlands through agreements concluded with
publishers, whilst in Sweden the deposit of static and
dynamic Internet resources is a legislative requirement.
Combined approaches A growing number of Web
archiving programs are concluding that no one archiving model is entirely
satisfactory for preserving national online heritage. Instead, countries such
as France and Denmark have found
that by employing a combination of comprehensive, selective and thematic
collection techniques, the optimum coverage of material is achieved.
Each of the foregoing archiving models have benefits and difficulties
according to the particular Web context to which they are applied. Other
strategies are also emerging for archiving Web content, including the 'by
discipline' approach of some universities and research institutes, such as the
DACHS (Digital Archives
for Chinese Studies) project based at the University of Heidelberg. In an
innovative approach, the Virtual Remote Control
(VRC) program at Cornell University is monitoring changes to Web sites over
time, and capturing those at risk of the loss of information as a last option.
A useful discussion on the relevant advantages and disadvantages of the
various Web archiving strategies can be found in the study
Collecting
and Preserving the World Wide Web : a Feasibility Study Undertaken for the JISC
and Wellcome Trust. Further introductions to Web archiving theory and
practice are listed under Web archiving
resources - Overviews.
Some Web archiving case studies have been collected on this site:
Australia | Austria |
Canada | Czech Republic |
Denmark | Finland |
France | Germany |
Japan | Lithuania |
The Netherlands | New Zealand
| Norway | Sweden |
United Kingdom | U.S.A |
Collaborative initiatives
3. Case studies of archiving approaches
Australia
The National Library of Australia has been involved in digital archiving
since 1996, being one of the first national libraries to undertake
investigations in this area. The major focus of its program has been the
creation of the PANDORA
archive, which is an archive of selected Australian online publications deemed
to be of national significance and long term research value. The system
operates on a model of selecting, collecting and storing Web resources from the
Australian domain, in cooperation with one territory and six state libraries,
the Australian War Memorial, AIATSIS (the Australian Institute of Aboriginal
and Torres Strait Islander Studies) and ScreenSound Australia. To support the
system the NLA has also developed the PANDAS (PANDORA Digital Archiving System)
software to assist in the automatic collection, description and provision of
access to the archive.
In Australia, legal deposit legislation does not yet cover electronic
publications, though a voluntary deposit scheme is in place with a number of
publishers. Working within the Library's resource constraints, emphasis has
been placed on creating a representative, rather than comprehensive archive,
and guidelines have been created for the selection of online materials for
preservation. Within the NLA, members of the Library's Digital Archiving Branch
regularly scan the Internet for new publications that comply with selection
guidelines, and harvesting software is employed for capture of resources.
Selection is also undertaken by participating institutions.
Fundamental to PANDORA's philosophy has been the provision of access to
archive content which is controlled in accordance with rights permission
negotiated with each publisher. A proportion of titles are regularly
re-harvested to preserve changes to content and some minor migration
experiments have been made. The underlying strategy has been to store material
in original formats, whilst maintaining access to another copy through
available technologies.
Key documents describing aspects of the Library's archiving program in
more detail are available from its
Digital Archiving
page. Links to more specific information about Australian Web archiving
initiatives are also listed under Web archiving resources - Australia.
Updated May 2004
Austria
A July 2000 amendment to Austrian media law extended legal deposit
requirements to digital offline material. Networked digital resources were not
included in the provisions and in response to the legislative gap, the
AOLA (the Austrian Online
Archive) project was established in 1999 to archive the Austrian Web space. A
joint initiative of the Austrian National Library and the Department of
Software Technology at the Vienna University of Technology, harvester software
is used to collect the .at domain and other sites of Austrian interest through
periodic snapshots of publicly available Web sites.
An initial AOLA pilot study was followed by the official first phase of
the project in 2001. A partially successful attempt at gathering a snapshot of
the Austrian Web space was made with the NEDLIB harvester in May of that year,
but this was later aborted due to crawler instability. In the following month,
a second effort using an adapted version of the Kulturarw3 Project's Combine
harvester was successful, as was a further crawl in 2002. A Data Warehouse was
established to analyse the characteristics of the harvested data.
At present the project is discontinued due to a lack of funding. A
resumption of captures is envisaged for 2005. A combined selective and a free
harvesting approach to collection is seen as the most effective strategy for
the future. The archive is not yet publicly accessible, and important areas for
further work include the selection of appropriate storage systems, the
development of a suitable legal framework, and the choice of optimum methods
for ensuring future access and preservation. The migration techniques employed
for preserving resources at present are seen as a short-range solution, to
which emulation methods will be added in the long-term.
Links to more specific information about Austrian Web archiving
initiatives are listed under Web archiving resources - Austria.
Updated August 2004
Canada
In 1994-95, the National Library of Canada conducted an
Electronic
Publications Pilot Project (EPPP) to examine the issue of the deposit of
online electronic publications. The final project report made recommendations
on important issues such as selection, copyright, and long-term preservation of
e-publications. As an outcome of the EPPP, Libraries and Archives Canada (as
the NLC is now known) has continued to collect online books and periodicals on
a voluntary basis through its
Electronic
Collection Web site. The E-Collection, as the archive is popularly known,
contains publications from commercial publishers, non-trade publishers,
individuals, and federal and provincial governments. At present, unrestricted
access is provided to approximately 80 percent of the archive through the LAC
Web site, the residual content remaining restricted due to publishers' concerns
regarding unfettered public access to their publications in the E-Collection.
In the absence of specific legal deposit legislation, a selective and
distributed model for collecting networked online Canadiana has been adopted by
the LAC, as part of the broader objective to collect and archive all online
resources originating in Canada or Canadiana published in other domains.
Detailed work has been undertaken on defining collecting scope, and formal
collection guidelines were issued in 1998. As part of the selection process
resources are manually assessed and a collecting status is assigned to each one
according to a three-tier scale. 'Archived' resources are stored on the LAC's
Web site and carry a permanent commitment on the part of the LAC to preserve
them. 'Served' resources are hosted by the LAC but are not endorsed with the
promise of preservation, responsibility for which may be taken up by other host
organisations. Those non-hosted resources for which the LAC has made an ongoing
commitment to preserve a link to are assigned a 'Linked' status.
Links to more specific information about Canadian Web archiving
initiatives are listed under Web archiving resources - Canada.
Updated May 2004
Czech Republic
The archive of the Czech online-born documents (known as WebArchiv) was originally launched within the R & D project funded by the Ministry of Culture in 2000 and since then it has been implemented in the National Library of the Czech Republic and funded almost exclusively through various grants. There are two other institutions involved in the project: Moravian Library, which is responsible for IT issues, and Institute of Computer Science of Masaryk University as external co-operator.
The aim of the WebArchiv project is to protect the Czech web cultural heritage. There is implemented a comprehensive solution in the field of archiving of the national web, i.e. bohemical online-born documents, including the Czech National Bibliography. Tools and methods for collecting, archiving and preserving web resources as well as providing long-term access to them are applied. Both large-scale automated harvesting of the entire national web and selective archiving are being carried out, including thematic collections on a given subject. At present these methods are tested and are a subject of further research. To run all operations in a routine way, two conditions must be met: sufficient long-term funding and legal background, primarily the legal deposit legislation.
Concerning legal issues, the current state of legislation in this field is not convenient. Legal Deposit Act doesn‘t cover online-born documents, new applicable legislation is being prepared. Nevertheless, since June 2006 the Copyright Law updated in harmony with the Directive 2001/29/EC will come into force; this means that it will be possible to open access to the whole archive from the library premises for the purpose of research or private study. Besides, agreements with publishers are made for some important resources applying selection criteria, which authorize the library to make these archived resources online accessible.
Collecting online documents is an automated process carried out by a set of software tools that harvest, index and save data in the archive according to preassigned parameters. Nowadays, most of the software tools used are open source products developed by the International Internet Preservation Consortium (IIPC) with the rest developed locally. So Heritrix tool is being used for web crawling. Harvested files including relevant metadata are saved in standardized archival formats supported by the IIPC consortium. Next to the server, the data is stored on the redundant disk array (RAID) with expected transfer to National Library’s new large data storage facility in the near future. At present (May 2006) data of approximate size of 2 TB (i.e. 26 million unique files) are stored in the archive. Another one server is used to give online access to a limited dataset whose content is covered by agreements with its original publishers. Fulltext indexing is performed by open-source system Nutch and is accessed through Nutchwax and WERA tools. The recent changes in Czech copyright law will demand for a fulltext index of the whole archive and require a new server dedicated to fulltext indexing.
International standards are being used for description and identification (MARC21, Dublin Core, ISSN and URN) and archiving (ARC). Selected online-born documents are catalogued in an ALEPH library system which supports Z39.50 (both client and server levels) and OAI-PMH protocols (both repository and harvesting levels with the profile for MARC21 and qualified Dublin Core); records are thus also registered in the Czech National Bibliography.
Links to more specific information about Czech Web archiving initiatives
are listed under Web archiving resources - Czech Republic.
Updated May 2006
Denmark
Danish legal deposit legislation enacted in 1997 extended deposit
requirements to physical format digital publications and static online
documents and was openly-worded to apply to other future electronic formats.
Based on this mandate the Royal Library (Kongelige Bibliotek) has been
selectively collecting Web resources since 1998, with physical format
electronic products, static online publications and e-journals being downloaded
to the online Pligtaflevering
registration system upon notification from publishers. Under Danish law
publishers must supply details of electronic products being submitted to the
system, from which the Royal Library extracts Internet addresses for the
materials for downloading. Royal Library staff than select, check, catalogue
and transfer the collected materials to an archival server.
In a parallel initiative in 2001, the Danish
netarchive.dk was
established by the Royal Library in collaboration with the State Library (Aarhus) and Center for Internet Research, University of Aarhus, as a testbed for various archiving strategies, and to
assess the usefulness of the collected material for researchers. The collection
of Internet coverage of the 2001 Danish municipal elections was used as a case
study for the first phase of the project, which ran to July 2002. Legal
impediments prevented direct capture of content from the Danish Web, and access
and harvesting frequency agreements were instead negotiated with owners, whose
sites were collected using appropriate software.
Work on the second phase of the project has continued with the aim of
refining the processes identified in the pilot and making recommendations for
requisite amendments to Danish legal deposit regulations. Once specific legal
deposit legislation has been enacted for networked material, a combined
selective and harvesting approach to collecting has been recommended as a
future path. This would involve the collection of material via three main
channels: an annual harvest of Internet coverage of one major event or theme
(an 'event-based' harvest), the ongoing selective harvesting of sites of key
interest, and the automatic collection of quarterly snapshots of the Danish
Web. It is envisaged that the State and University Library would take
responsibility for the first two methods, with the Royal Library administering
the latter. Present interest is focused on the release of an official response
to the recommendations of a netarchive.dk report, which was presented to the
Danish Ministry of Culture in April 2004.
The Royal Library has also been active in the
Nordic Web Archive project.
Links to more specific information about Danish Web archiving
initiatives are listed under Web archiving resources - Denmark.
Updated May 2004
Finland
In Finland, the EVA archival system to
collect, register and store Internet publications and ensure their long-term
preservation, ran as a joint project of libraries, publishers and expert
organisations within the country, initiated in 1997. The program was led by the
Helsinki University Library (National Library of Finland) in association with
three Finnish universities, the Edita publishing house and CSC Scientific
Computing, with the objective of collecting publicly available static Web pages
and other inline material in the Finnish domain. Specific aims of the project
have included the formulation of criteria for electronic document selection,
the creation of a direct deposit system for publishers, and the development of
a reliable long-term archive.
Building on the work of two earlier stages, the third phase of the EVA
project was completed in 2001, and the resultant archival system is based on a
whole domain approach to collection. Following a successful first sweep of the
Finnish Web in June 2002, snapshots of the .fi Web space are now gathered
periodically. In addition, partial harvests are carried out in connection with
special events such as national elections. Cooperation has been obtained in
capturing the national domain from the Finnish portal Info Center Finland,
which forwards Finnish server locations in other domains than .fi for loading
into the project harvester.
Finland has also been an active partner in the
Nordic Web Archive and
NEDLIB projects, with programming
of the NEDLIB harvester prepared by the CSC to project specifications. Further
development of the NEDLIB system was jointly undertaken by the CSC and the
Helsinki University Library (HUL). Up to Spring 2004 the NEDLIB harvester was
used, but it will be replaced by the Heritrix harvester, developed jointly by a
consortium consisting of the Internet Archive and a number of national
libraries.
The national library's right to harvest Web content will be granted in
the new Copyright Act, to be approved by the Parliament in late 2004 - early
2004. The new Copyright Act and Legal Deposit Act have been developed in
parallel, so that these laws refer to one another. The new Legal Deposit Act,
which covers electronic resources (on-line and off-line) plus radio and TV
programs, is yet to be passed from the Ministry of Education to the
Parliament.
Links to more specific information about Finnish Web archiving
initiatives are listed under Web archiving resources - Finland.
Updated July 2004
France
An extension of the French legal deposit law to networked digital
material should be voted by the French Parliament in 2004. It will allow BnF
and INA (Institut National de l'Audiovisuel, in charge of TV and Radio
preservation) to harvest sites on-line as well as request a deposit from
publishers, when online harvesting is impossible and make the collection
publicly available on site. At present the experimental archive is not publicly
accessible.
Ahead of any legislative amendments, the
Bibliothèque nationale de
France (BnF) has been active since 2000 in developing a combined
methodology including:
- Automatic large scale domain crawls several times a year
- Continuous crawl of automatically selected sites (10% of the
total)
- Deposit of deep web sites that can't be harvested on-line
- Thematic event-based collection for very ephemeral sites (that
crawlers would take too long to find)
In collaboration with INRIA (the French National Institute for Research
in Computer Science and Automatic Control), the BnF has been experimenting in
Web collection techniques using automatic harvesting and evaluation methods. A
test of automatic selection parameters based in-linking ranking (like Google)
has been made.
The initial phase of the program, which ran until June 2001, focused on
content gathering, involving the collection of 16 audiovisual sites with small
robots to assess the limits of these crawlers. In 2002 two harvests were made:
the first a thematic collection of sites relating to the French elections (1900
sites), and the second a comprehensive crawl of the .fr Web.
In a second pilot on methods for archiving hidden Web sites, which ran
from 2002 to June 2003, 100 selected Web owners were approached to deposit
their Web sites in the BnF for permanent archiving. A portable extraction tool
(DeepArc) was developed to enable simple extraction of database to XML by the
producers. This tool, currently being tested by IIPC members will be released
in open-source soon. It is envisaged that harvester technology may be employed
as a discovery tool to identify sites in the Deep Web of interest for
collection, which would then be transferred to the BnF after a technical
negotiation with site owners.
Aware of the necessity of international collaboration in the domain of
Web Archiving, BnF has launched the
International Web Archiving
Workshop (IWAW) series, and the
Web archive discussion
list and actively participated in the creation of the International
Internet Preservation Consortium (IIPC) that BnF coordinates.
Links to more specific information about French Web archiving
initiatives are listed under Web archiving resources - France.
Updated July 2004
Germany
In 1997, whilst awaiting updating of its legal deposit legislation to
include online publications, the German
National Library (DDB) commenced a voluntary deposit trial for electronic
materials. The project began by collecting electronic dissertations and theses
and was extended in 2000 when the SpringerLink publishing house began
depositing electronic journals. The scope of the program was broadened in the
same year with the issue of guidelines for the deposit of networked digital
publications, and the development of an automatic collection process for online
materials in 2001.
A cooperative agreement between the DDB and the German publishers' union
was reached in March 2002 to formalise the voluntary deposit scheme. Under the
terms of the arrangement, publishers deposit their electronic resources in the
Deposit.DBB.DE archive server,
to the contents of which the DDB provides in-house access. Dynamic publications
and databases are able to be transferred to the system subject to certain
transferability criteria, including their ability to operate with standard
software but independently of other servers.
An experiment in collecting dynamic Web sites was also conducted using
the German government Web site (Die Bundesregierung) and HTTrack harvester
technology. Crawls were performed in December 2000 and February 2001, from
which it was concluded that the most efficient and effective strategy for
Germany would be a legally-enforced deposit model rather than a resource
intensive 'pull' approach. The studies have also highlighted the importance of
ensuring the usability of archived material at an early stage in the archival
process so that access problems are not obscured until it is too late for them
to be resolved.
Links to more specific information about German Web archiving
initiatives are listed under Web archiving resources - Germany.
Updated May 2004
Japan
In 2000 amendments were made to Japanese legal deposit law to encompass
CD-ROMs and other physical format materials. The National Diet Library
submitted a request to the Legal Deposit System Research Council in 2002
requesting that either legal deposit legislation be extended to online digital
information, or that a legal framework be developed for their collection by the
NDL.
Whilst awaiting the outcome of the Council's deliberations, the NDL has
initiated three-year trial projects and studies, one of which is
WARP (Web Archiving Project), an
experimental investigation into the gathering and archiving of Web resources,
and another is the research of the country's long-term digital preservation
needs.
Through the WARP project, the NDL is collecting snapshots of Japanese
Web sites and testing storage and preservation solutions, with the agreement of
selected participating organisations. WARP is also being used to collect and
archive electronic periodicals as the basis for an online periodicals
collection. In June 2004, 600 Japanese Web sites and more than 1100 electronic
journals had been collected.
Future research interest is focused on the compilation of comprehensive
selection guidelines for the preservation of Japanese digital heritage, the
exploration of comprehensive harvesting techniques, and the development of
rendering and searching techniques.
Links to more specific information about Japanese Web archiving
initiatives are listed under Web archiving resources - Japan.
Updated July 2004
Lithuania
The Martynas Mažvydas National Library of Lithuania is the legal
deposit institution for Lithuanian publications. Electronic resources were
included in a 1996 general deposit decree, though the exact extent of the
provision was not defined. To guide the NLL in administering its national
collecting responsibilities for online materials, selection criteria were
compiled to include the collection of resources from the .lt domain, other
documents of special quality and of acknowledged Lithuanian interest, and
commercial and government e-publications. In accordance with the guidelines,
physical format materials are to be deposited in a traditional manner, whilst
static electronic documents will be gathered using harvesting methods. Dynamic
resources and other public database content are to obtained by way of
depository agreements concluded with publishers.
To support the collection of online Lithuanian material the
Archive
of Electronic Resources was established in November 2002 as a subsystem of
the LIBIS (Lithuanian Integrated Library Information System) program. Once
fully operational, it is envisaged that the AER will operate as a deposit
system for online publications, based upon the NEDLIB model.
In addition, the NLL has been a participant in the
NEDLIB project and the NEDLIB
harvester was employed in the first sweep of the Lithuanian Web in October
2002. A second crawl in November 2002 involved the selective capture of
approximately 60 registered e-periodicals. Many of these comprised dynamically
generated Web pages, which would be overlooked by existing harvesting
techniques. To address this issue, agreements for the deposit of e-publications
were forwarded in March 2003 to the principal Lithuanian publishers. Regular
harvests of the Lithuanian Web are now conducted every six months and rights
clearances for each resource are being simultaneously obtained, together with
metadata elements in Dublin Core standard.
Links to more specific information about Lithuanian Web archiving
initiatives are listed under Web archiving resources - Lithuania.
Updated May 2004
The Netherlands
In 1995 the Depot van Nederlandse Electronische Publicaties (DNEP)
project was established in The Netherlands to construct a system for the
voluntary deposit of digital publications. The country is one of the few in the
world that does not have legal deposit legislation, and lodgement of Dutch
resources in the National Library of the Netherlands (Koninklijike Bibliotheek)
is effected through voluntary agreements with publishers.
After an extended period of experimentation, the KB entered into a
project with IBM in 2000 to develop a deposit archive for Dutch electronic
publications based on the OAIS framework. The resultant
e-Depot
was introduced at the KB in 2002 where it has become the official archive of
the major Dutch publishing houses Kluwer and Elsevier Science, and other
members of the Dutch Publishers Association. Deposit of electronic publications
in the e-Depot is not restricted to publications with Dutch imprint, but is
based on agreement with publishers. While Elsevier and Kluwer were Dutch
publishers in origin, new arrangements have been drawn up with international
publishers that do not have a specific connection with the Netherlands, like
the open access publisher BioMed Central and Blackwell Publishing. Now fully
operational, work is continuing on the development of a Long-Term Preservation
Module to provide access to stored data, manage metadata, and enable migration
and emulation strategies to be effected. The Dutch research initiatives have
been extensively documented, and a number of reports and a project overview
have been produced by the working group.
In another collaboration, the KB was the lead partner in the
NEDLIB project, which ran between
1998 and 2000. Through this program the KB trialled harvesting of the Dutch
domain using the NEDLIB harvester and participated in the development of
guidelines for online deposit systems.
Links to more specific information about Dutch Web archiving initiatives
are listed under Web archiving resources - The Netherlands.
Updated July 2004
New Zealand
The National Library of New Zealand has been formally researching Web
archiving since 2002 (although earlier tests were undertaken during the 1999
election).
Harvesting experiments have been undertaken using HTTrack technology and
have targeted the 2002 national election, the America's Cup regatta held in
Auckland and government web sites. Over 400 discrete objects and full web sites
have been captured and stored since December 2002. In important supporting
research, the Library has developed a preservation metadata model, the full
version of which was issued in July 2003.
The National Library of New Zealand Act 2003 expanded the scope of the
Library's legal deposit mandate to include online and offline electronic
resources including both publicly available and 'deep-web' Web resources. In
anticipation of the legislative amendments, the NLNZ began actively developing
its processes for the selection, acquisition and archiving of physical format
and online electronic documents into its existing workflows.
In 2003 a selection database was created for recording selection
decisions and in early 2004 the Library began researching available web
archiving software including the National Library of Australia's PANDAS
technology.
In May 2004, following a major review of the NLNZ's digital library
directions, substantial funding was announced for the establishment of a New
Zealand digital archive to safeguard the national output of online and offline
electronic resources which will include web archiving activities.
The National Library of New Zealand's web archiving approach will
comprise a combination of snapshot harvesting of the whole .nz domain at
defined intervals, selective harvesting of specific web sites or parts of web
sites, and event harvesting of resources associated with topics such as the
national elections and the America's Cup regatta.
Links to more specific information about New Zealand Web archiving
initiatives are listed under Web archiving resources - New Zealand.
Updated July 2004
Norway
The National Library of Norway started the
Paradigma Project in
August 2001 and it is scheduled to run until 31 December 2004. The main goal of
the initiative is to ensure the satisfactory legal deposit of Norwegian digital
documents, and this includes the development of the technology, methodology and
routines for the selection, collection, description and identification of all
types of digital documents - including dynamic Internet documents.
The Library is also working to give users access to its Internet archive
in compliance with the Legal Deposit Act, for purposes such as research and
documentation. Considered extremely modern when it came in to effect in 1989,
the current Legal Deposit Act, covers all generally available Norwegian
documents stored on any medium: including paper, microforms, photographs,
combined documents, sound fixations, films, video, digital documents and
broadcasting programs. Documents published abroad for Norwegian publishers and
those specially adapted for a Norwegian public are also covered.
Based on the Legal Deposit Act and recommendations from the Paradigma
Project, the National Library has started the general harvesting of all
generally available digital documents from the Norwegian Web space ('.no'). The
first harvesting round was carried out in December 2002-January 2003, and it
resulted in approximately 3.1 million URLs. The second harvesting round in
August 2003 resulted in approximately 4.1 million URLs. The third harvesting
round is currently underway, and in time, documents found on domains such as
'.com', '.org' and '.net', will also be harvested.
An extremely challenging issue is the deposit of resources from the Deep
Web, such as Internet newspapers, streaming media, documents from web cameras,
interactive media and e-materials of all types stored in databases. The
Paradigma Project has started the daily collection of approximately 65 Internet
newspapers, and it will be downloading several entire newspaper databases in
the near future, thus complementing the daily 'snapshots'. The National Library
of Norway is also discussing Deep Web problems within the framework of the
International Internet
Preservation Consortium.
The Paradigma Project has recommended selection criteria for digital
documents based on the Legal Deposit Act and the Library's collection strategy.
These criteria will influence the Library's manual routines as well as the
design of its Internet archive's system architecture. The Project plans to
implement a three-phase selection process, so that librarians receive technical
help to find the few documents that qualify to be catalogued at some level. The
first selection phase finds and collects the Norwegian and Sami documents from
the Internet. The second phase gives librarians the opportunity to
automatically produce ranked lists based on specific queries. These lists are
based on the use of vectors containing metadata that has been automatically
extracted from the collected documents. In the third phase, librarians choose
specific documents from the ranked lists for manual registration at some level,
using the selection criteria mentioned above.
The Paradigma Project is currently investigating metadata formats for
the description of digital documents. Although many digital documents may
deserve a bibliographic description, the Paradigma Project estimates that far
less than 1 per cent of the material collected from the Norwegian Internet
domain may ever be subject to cataloguing at some level. In contrast, 100 per
cent of the Internet documents will be fully indexed with FAST indexing
software after harvesting. This will allow the Library's staff and users to
search the Internet archive - both via free text as well as other indices. In
addition to cataloguing some documents, and indexing all documents, the Project
will be harvesting existing embedded metadata as well as the Internet documents
they describe, and the National Library is planning a future service that will
allow publishers to generate and deliver metadata with their documents at the
time of deposit.
Giving users access to the legal deposit Internet archive is a complex
matter, and the National Library must find satisfactory solutions in spite of
the many, and sometimes conflicting, regulations found in the Legal Deposit
Act, the Copyright Act and the Personal Data Act. The Paradigma Project is
currently trying to find answers to questions like: Which users can receive
access to different types of digital materials? Can they access the collections
from computers outside the National Library?
The Library plans to give users access to its Internet archive via the
Nordic Web Archive's (NWA) Access Tool. Today,
free text searching with Boolean operators, a search for a certain URL, and the
presentation of document history via a timeline are standard options. The
Paradigma Project will adapt this tool's interface to accommodate several
special user functions, and because it is desirable to present the archived
digital documents and metadata in an organised and structured way, IFLA's
Functional Requirements for
Bibliographic Records (FRBR) model will be used in the Internet archive's
design.
Links to more specific information about Norwegian Web archiving
initiatives are listed under Web archiving resources - Norway.
Updated June 2004
Sweden
From its inception in September 1996, the Swedish
Kulturarw3 (Cultural Heritage
Cubed) project has explored methods for the collection, preservation and
provision of access to Swedish online documents. A project of the Swedish Royal
Library, Kulturarw3 is engaged in the comprehensive collection of static and
dynamic Internet resources from the Swedish Web.
In a significant technological advance, Sweden was the first country to
investigate the use of harvester technology for archiving Web content. The
initial harvest in 1997 was limited to the .se domain but subsequent crawls
have been expanded to include other Swedish server addresses. Since 2002 the
Royal Library has been authorised to collect Swedish Web sites on the Internet,
and two to three harvests are carried out per year. The Royal Library is
responsible for storage and long-term preservation of the collected resources,
which may only be accessed on-site at the Library's premises. From 2003, access
to the archive content has been provided using surfing software akin to the
Internet Archive's Wayback Machine.
Research effort is now engaged in developing a tool to enable free text
searching.
The Royal Library has also been active in the
Nordic Web Archive project.
Links to more specific information about Swedish Web archiving
initiatives are listed under Web archiving resources - Sweden.
Updated June 2004
United Kingdom
Research and experimentation in digital archiving in the United Kingdom
can be characterised as beginning with pilot initiatives involving individual
institutions followed by collaboration nationally and internationally.
In 2001 the British Library carried out a six-month experiment to select
and capture (but not make publicly accessible) 100 UK Websites. The
Domain.uk project, as
it was called, was based on obtaining explicit agreement of Website publishers
and included in its scope a broad range of UK Websites of historical and
cultural significance. It also included some key current events such as the
General Election of 2001 and the foot and mouth disease epidemic of the same
year. The success of Domain.uk led to the setting up of the British Library's current Web Archiving Programme, which is focussed on the following interlinked
strands: the definition of a collection development policy for Web archiving;
participation, as a lead partner in the UK Web Archiving Consortium (UKWAC);
and participation in the International Internet Preservation Consortium (IIPC),
led by the Bibliotheque nationale de France.
In parallel with these developments, legal deposit legislation within
the UK has now become subject to the Legal Deposit Libraries Act 2003, which
reached the statute book in February 2004. The Act extends legal deposit to
non-print and currently comprises enabling legislation. A governmental Advisory
Panel is to be set up to oversee the various stages of secondary legislation
which are likely to lead to Regulations by format, e.g., offline, e-books,
e-journals, Websites. Pending secondary legislation, the British Library Web
archiving initiatives will be based entirely on a voluntary, rights-cleared
basis.
Links to more specific information about British Web archiving
initiatives are listed under Web archiving resources - United Kingdom.
Updated July 2004
United States
The Minerva (Mapping the
Internet the Electronic Resources Virtual Archive) project was established by
the Library of Congress in 2000 as an experiment to provide experience in the
archiving of 35 selected Web sites. Snapshots of three sites for closer study
were downloaded using HTTrack software and catalogued onto LC's library
management system. A Web site was created to test access, and legal issues were
examined in cooperation with the US Copyright Office. In a parallel study,
daily snapshots of approximately 200 sites relating to the 2000 presidential
election were gathered by the Internet
Archive. Based on the results of these pilots, recommendations were made
regarding future larger-scale archiving initiatives.
Today Minerva has developed into an ongoing thematic digital archiving
program, which focuses on the harvesting of topic based public Web content.
More than 35,000 sites have been archived since 2000, including the 9/11 and
9/11 Anniversary, Election 2000 and 2002, and the 2002 Winter Olympics sites,
in association with the Internet Archive which acts as the contractor for
crawling and capture. LC has also explored the legal aspects of collecting Web
content and it has been interpreted that the Library is entitled to collect and
make accessible networked materials, though this is not explicitly stated in
legal deposit legislation.
It is recognised that both comprehensive and selective harvesting will
need to be employed by LC in future if the volume and complexity of the United
States Web is to be adequately preserved. Present directions suggest that this
will best be achieved through partnerships with organisations such as the
Internet Archive. As part of this strategic path, the Library of Congress is
also leading a cross-sector cooperative online archiving venture known as the
National Digital Information
Infrastructure and Preservation Program (NDIIPP). The NDIIPP will
coordinate existing digital archiving research programs across the public and
private sectors in the United States to minimise replication, allocate
responsibilities and to share knowledge and experience.
Links to more specific information about United States Web archiving
initiatives are listed under Web archiving resources - U.S.A.
Updated May 2004
Collaborative
initiatives
International Internet
Preservation Consortium
(IIPC) In July 2003 the International Internet Preservation Consortium
(IIPC) was created as a collaboration between the national libraries of France,
Australia, Canada, Denmark, Finland, Iceland, Italy, Norway, Sweden, the
British Library, the Library of Congress and the Internet Archive. Led by the
Bibliothèque nationale de France, the Consortium has been formed to
support and foster development in the collection, archiving and provision of
access to international Internet content in the long-term. Encouraging the use
of common tools and methods to facilitate interoperability, and the raising of
awareness of Internet preservation issues the are other key objectives of the
Consortium.
Six working groups have been established to consider issues relating to
a common technical framework, researcher requirements, access tools, metrics
and the evaluation of Web crawlers, the Deep Web and content management. Until
the expiry of the initial three-year agreement, membership of the IIPC is
limited to the founding partners, but it is hoped that more national libraries
will join the Consortium after mid-2006.
Links to more specific information about the IIPC are listed under Web
archiving resources - IIPC.
Updated July 2004
Internet Archive
A San Francisco based non-profit commercial venture, the Internet
Archive is comprehensively harvesting and preserving publicly available
materials from the global Web, and has amassed the largest collection of public
Web pages in the world since it began its pioneering activity in 1996. The core
of the Internet Archive's collections are received from Alexa Internet, a
commercial crawler and research company which donates collected data to IA
immediately following each crawl period. Broad snapshot crawls are made every
two months, and the content of the archive was made publicly accessible in 2001
via the 'Wayback Machine'.
This whole domain approach is based on periodic harvesting, in which
snapshots of Web sites of all levels of quality and subject scope are collected
and stored for future reference. As many sites as possible are collected but
not all of these in their entirety, currently a total of 45 million sites. In
tandem some narrow crawls are made of selected sites to collect their entire
content.
More recently, the Internet Archive has also entered into partnerships
with other institutions to develop special collections according to the
institution's profile. The IA has also be active in trying to enter into
collaborations with national institutions in order to collect and build
interoperable national heritage collections, using IA technical expertise and
data collection based on each country's specifications.
Links to more specific information about the Internet Archive are listed
under Web archiving resources - Internet Archive.
Updated July 2004
NEDLIB
Funded by the European Commission, the NEDLIB (Networked European
Deposit Library) Project ran between 1997 and 2000 with the purpose of
developing harvesting software specifically for the collection of Web resources
for a European deposit library. Harvesting technology for indexing Web content
had been in place since the mid-1990s, but NEDLIB was one of the first crawlers
to be specifically developed for archiving purposes. The new software was
created by the Finnish IT Center for Science (CSC), to a specification
formulated by the multiple project partners (a consortium of eight European
national libraries, a national archive, two IT companies and three publishing
houses).
In January 2000, the first version of the NEDLIB harvester was released
and tested by project members and the national libraries of Estonia and
Iceland. An updated system was tested on the Icelandic Web in January 2001, and
with further modifications on the Finnish domain in 2002. The NEDLIB initiative
revealed that whilst it is a fairly simple undertaking to design a basic
harvester, it is a very complex task to develop software that is efficient,
robust, and that can be programmed to capture every nuance of selection.
Although the project is now defunct, the NEDLIB site is managed by the
Koninklijke Bibliotheek (the National Library of the Netherlands). The NEDLIB
harvester can still be found at http://web.archive.org/web/20051217014839/www.csc.fi/sovellus/nedlib/.
The latest version was published in September 2002. The application is no
longer supported or developed; in Finland, it will be replaced by Heritrix in
Summer 2004.
Links to more specific information about the NEDLIB project are listed
under Web archiving resources - NEDLIB.
Updated July 2004
Nordic Web Archive
The Nordic Web Archive is a cooperative project of the national
libraries of Denmark, Finland, Iceland, Norway and Sweden for the sharing of
experience in Web archiving. The major focus of the collaboration has been the
development of an interoperable toolset for providing long-term access to Web
archives. Funding was received in August 2000 for the first phase of the
program, which is administered from the National Library of Norway.
Digital archives exist in all the participating countries, the content
of which is transferred to the central NWA. A software package known as the NWA
toolset has been developed using expertise from each partner library to enable
navigation and searching of archived content in both time and space.
The second phase of the NWA project began in late 2002, and is focused
on the enhancement of software stability, the need for additional user
functionality, and the re-evaluation of the user interface.
Links to more specific information about the Nordic Web Archive are
listed under Web archiving resources - Nordic Web Archive.
Updated May 2004
UK Web Archiving
Consortium
Comprising six leading UK institutions, the UK Web Archiving Consortium
(UKWAC) will work, with the permission of rightsholders, on an experimental
system for archiving selected key UK Websites - ensuring that invaluable
scholarly, cultural and scientific resources remain available for future
generations.
The UKWAC - comprising The British Library, Joint Information Systems
Committee of the Higher and Further Education Councils (JISC), The National
Archives, The National Library of Wales, The National Library of Scotland and
the Wellcome Trust - will run for an initial period of two years, during which
approximately 6,000 websites will be collected and archived.
Consortium members will obtain the permission of website owners to
archive selected sites whilst working collaboratively to explore how to develop
compatible selection policies and to investigate the complex technical
challenges involved in collecting and archiving Web material.
Each consortium member will select and 'capture' content relevant to its
subject and/or domain. For example, The British Library will archive sites
reflecting national culture and events of historical importance. These could
include Web pages focusing on key events in national life, museum Web pages,
e-theses, selected blogs to support research material and Web-based literary
and creative projects by British subjects.
Wellcome will preserve a record of medicine on the Web whilst The
National Archives will focus on archiving selected material from six main
clusters of government departments. The Scottish and Welsh National Libraries
will collect material reflecting the culture and history of Scotland and Wales
and JISC will preserve Websites from leading-edge, innovative ICT projects in
UK Higher and Further Education.
Infrastructure costs, such as software, hardware, and ongoing technical
development and support will be shared equally amongst the Consortium members.
UKWAC will use HTTrack - the open source Web crawler - to acquire films for
storage. The software to carry out the archiving processes - PANDORA Digital
Archiving System (PANDAS) - has already been developed and tested by the
National Library of Australia and its partners for archiving Australian
Websites and making them accessible through PANDORA.
UKWAC members have selected Magus Research Limited to help extend the
PANDAS software for UK needs and provide the shared hardware and technical
support they require.
Updated July 2004
4. Resources
Web archiving resources -
Overviews
Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/Webarchive.html
Archiving the World Wide Web
http://www.clir.org/pubs/reports/pub106/Web.html
Management of Networked Electronic Publications : A Table of Status in
Various Countries
http://www.collectionscanada.ca/obj/r7/f2/r7-100-e.pdf
Best Practices for Digital Archiving : An Information Lifecycle Approach
http://www.dlib.org/dlib/january00/01hodge.html
Digital Preservation and Permanent Access to Scientific Information :
The State of the Practice
http://www.dtic.mil/cendi/publications/04-3dig_preserv.html
Web Archiving : Bibliography
http://www.ifs.tuwien.ac.at/~aola/links/WebArchiving.html
Digital Electronic Archiving : The State of the Art and the State of the
Practice http://web.archive.org/web/20040715012747/http://www.icsti.org/Dig_Archiving_Report_1999.pdf
ERPANET Workshop - The Long Term Preservation of Databases : ERPANET
Workshop Report
http://www.erpanet.org/events/2003/bern/Bern_Report_final.pdf
Collecting and Preserving the World Wide Web : a Feasibility Study
Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_feasibility.pdf
Legal Issues Relating to the Archiving of Internet Resources in the UK,
EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf
The Last Page of the Internet : the Importance of Preserving the
Dynamic Aspects of the Internet
http://www.deflink.dk/upload/doc_filer/doc_alle/1023_NBR.doc
Web archiving resources - Australia
PANDORA - Preserving and Accessing Networked Documentary Resources in
Australia http://pandora.nla.gov.au/index.html
PANDORA, Australia's Web Archive, And The Digital Archiving System That
Supports It
http://www.digicult.info/downloads/dc_info_issue6_december_20031.pdf
Digital Archiving
http://www.nla.gov.au/initiatives/digarch.html
Archiving the Web : the PANDORA Archive at the National Library of
Australia http://www.nla.gov.au/nla/staffpaper/2001/cathro3.html
Report On Senior Executive Fellowship to Research Digital Archiving in
National Libraries
http://www.nla.gov.au/nla/staffpaper/2002/elect.html
Digital Preservation : The Australian Experience
http://www.nla.gov.au/nla/staffpaper/dw001004.html
Ensuring Long-Term Access to Online Publications
http://www.press.umich.edu/jep/04-04/phillips.html
Digital Archiving - Developing Policy And Best Practice Guidelines At The
National Library Of Australia
http://web.archive.org/web/20010303103129/www.icsti.org/icsti/2000workshop/gatenby.html
Legal Issues Relating to the Archiving of Internet Resources in the UK, EU,
USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf
Collecting and Managing Web Resources for Long-Term Access : Web
Harvesting and Guidelines to Support Preservation (ICABS Actions 3.3 and 3.4)
http://www.ifla.org/IV/ifla70/papers/026e-Gatenby.pdf
Web archiving resources - Austria
AOLA : Austrian On-Line Archive
http://www.ifs.tuwien.ac.at/~aola/
Uncovering Information Hidden in Web Archives: a Glimpse at Web
Analysis Building on Data Warehouses
http://www.dlib.org/dlib/december02/rauber/12rauber.html Austrian
On-Line Archive : Current Status and Next Steps
http://bibnum.bnf.fr/ecdl/2001/austria/austria.pdf
Long-Term Preservation of Digital Material - Building an Archive to
Preserve Digital Cultural Heritage From the Internet
http://www.ifs.tuwien.ac.at/~aola/publications/thesis-ando.pdf Part
of Our Culture is Born Digital : On Efforts to Preserve it for Future
Generations http://www.inst.at/trans/10Nr/rauber10.htm
Archiving the Internet : Challenges, Projects, and the Austrian Perspective
http://www.ifs.tuwien.ac.at/~aola/publications/slides_reichenau_01.pdf
Web archiving resources - Canada
Networked Electronic Publications : Policy and Guidelines
http://epe.lac-bac.gc.ca/100/200/301/nlc-bnc/networked_epubs-ef/9/8/index-e.html
National Library of Canada Electronic Collection
http://collection.nlc-bnc.ca/e-coll-e/index-e.htm
Electronic Publications Pilot 1996/97 : Final Report
http://epe.lac-bac.gc.ca/100/200/301/statcan/electronic-e/0009711-534-RIE.pdf
Electronic Publications Pilot Project (EPPP) : Summary of the Final
Report http://epe.lac-bac.gc.ca/100/200/301/nlc-bnc/eppp_summary-e/ereport.htm
Networked Electronic Publications Policy
http://web.archive.org/web/20041123172033/http://www.collectionscanada.ca/bulletin/015017-9905-07-e.html
Web archiving resources - Czech Republic
WebArchiv : Archive of the Czech Web http://en.webarchiv.cz
Archiving the Czech Web : Issues and Challenges
http://bibnum.bnf.fr/ecdl/2003/index.html
Research Intent - Digital Libraries
http://wwwdata-old.muni.cz/research/cez_item.asp?ID=23
Digital Cultural Heritage and the Cooperation of National Memory Institutes http://en.webarchiv.cz/files/dokumenty/konference/Zabickabanskabystrica2006.doc
Web archiving resources - Denmark
Netarchive.dk : ECDL Workshop, October 19th 2002
http://bibnum.bnf.fr/ecdl/2002/Kb/Kb.html
netarchive.dk - the Pilot Project
http://www.netarchive.dk/pilot-index-en.htm
netarchive.dk - 2nd phase 2004
http://netarchive.dk/index-en.php
netarchive.dk - the Pilot Project : Project Reports
http://www.netarchive.dk/rap/index-en.htm
Legal Deposit From the Internet in Denmark - Experiences With the Law From
1997 and the Need for Adjustments
http://www.deflink.dk/upload/doc_filer/doc_alle/1023_BNH.doc
Legal Deposit In Denmark - The New Law and Electronic Products
http://web.archive.org/web/20070608091825rn_1/liber-maps.kb.nl/articles/dupont11.htm
Web archiving resources - Finland
EVA : The Acquisition and Archiving of Electronic Network Publications
http://www.lib.helsinki.fi/eva/english.html
EVA : Elektronisen Verkkoaineiston Hankinta ja Arkistointi : Raportteja
http://www.lib.helsinki.fi/eva/raportit.html
Harvesting the Finnish Web Space : Practical Experiences
http://bibnum.bnf.fr/ecdl/2001/finland/sld001.htm
EVA - The Acquisition and Archiving of Electronic Network Publications
In Finland http://www.lib.helsinki.fi/tietolinja/0199/evaart.html
Experiences and Conclusions From a Pilot Study : Web Archiving of the
District and County Elections 2001
http://netarchive.dk/publikationer/webark-final-rapport-2003.pdf
Legal Deposit Act (Draft)
http://www.minedu.fi/export/sites/default/OPM/Julkaisut/2003/liitteet/opm_132_Legal_Deposit_Act.pdf?lang=en
Web archiving resources - France
Towards Continuous Web Archiving : First Results and an Agenda for the
Future http://www.dlib.org/dlib/december02/masanes/12masanes.html
A First Experience in Archiving the French Web
ftp://ftp.inria.fr/INRIA/Projects/verso/gemo/GemoReport-229.pdf
The BnF's Project for Web Archiving
http://bibnum.bnf.fr/ecdl/2001/france/sld001.htm International
Web Archiving Workshops (IWAW) http://www.iwaw.net Web Archive (discussion
list) http://listes.cru.fr/wws/info/web-archive
Web archiving resources - Germany
Archive Server DEPOSIT.DBB.DE
http://deposit.ddb.de/index_e.htm
Collecting,Indexing and Archiving Net Publications
http://www.d-nb.de/eng/wir/ueber_dnb/netzpubl.htm Submission of online publications to Die Deutsche Bibliothek
http://deposit.ddb.de/netzpub/web_abgabe_np_gesamt_e.htm
Collection of German online resources by Die Deutsche Bibliothek
http://bibnum.bnf.fr/ecdl/2001/germany/sld001.htm
Long-term archiving of digital documents : what efforts are being made in
Germany?
http://caliban.ingentaselect.com/vl=9314169/cl=57/nw=1/fm=docpdf/rpsv/cw/ alpsp/09531513/v16n3/s9/p207
NESTOR : Network of Expertise in Long-term STOrage of Digital Resources - A
Digital Preservation Initiative for Germany
http://www.dlib.org/dlib/april04/dobratz/04dobratz.html
NESTOR : Kompetenznetzwerk Langzeitarchivierung : Network of Expertise in
Digital Preservation
http://www.langzeitarchivierung.de/index.php?newlang=eng
Long-term Preservation of Electronic Theses and Dissertations
http://edoc.hu-berlin.de/etd2003/liegmann-hans/HTML/index.html
Web archiving resources - Japan
WARP http://warp.ndl.go.jp/
Digitizing, Archiving and Preserving Japanese Cultural Heritage
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287396&reqid=118767#feature1
Web Resources as Cultural Heritage: International Symposium on Web
Archiving http://www.ndl.go.jp/en/publication/proceedings/index.html Legal
Deposit System Council
http://www.ndl.go.jp/en/aboutus/deposit_council_book.html
Electronic Publications and the Legal Deposit System in Japan
http://www.ndl.go.jp/en/publication/ndl_newsletter/107/076.html
Electronic Publications and National Bibliography in Japan
http://www.ndl.go.jp/en/publication/ndl_newsletter/112/123.html
Web archiving resources - Lithuania
Harvesting and Archiving of Electronic Resources in Lithuania : Towards
Virtual Library
http://www.inforum.cz/inforum2003/prispevky/Jodelis_Remigijus.pdf
Web archiving resources - The Netherlands
e-depot and Digital Preservation
http://www.kb.nl/dnp/e-depot/e-depot-en.html KB/IBM
Long Term Preservation Study
http://www.kb.nl/hrd/dd/dd_onderzoek/reports/6-webpublications.pdf
Digital Preservation in Practice: The e-Depot at the Koninklijke
Bibliotheek
http://puck.emeraldinsight.com/vl=2423084/cl=15/fm=html/nw=1/rpsv/cw/ mcb/03055728/v34n1/s3/p21
Legal Deposit of Digital Materials
http://www.kb.nl/kb/hrd/dd/dd_links_en_publicaties/publicaties/LIBER-Oltmans-input.doc
Digital Archiving : A Necessary Evil or New Opportunity?
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6W63-4BV52PK-1&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=3ab0c901795f97b1a891e20529da4e85
Digital Preservation and Permanent Access: The UVC for Images
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/uvc-ist.pdf The
Road to E-Deposit at the Koninklijke Bibliotheek
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/eskb-roadtoe-deposit3_20aug02.pdf Permanent Archiving of Electronic Publications : Research &
Practice
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/ summerschoolticer2003.pdf
Preservation Functionality in a Digital Archive
http://portal.acm.org/citation.cfm?id=996350.996416&coll=portal&dl=ACM&type=series&idx =996350=Proceedings&WantType=Proceedings&title=International %20Conference%20on%20Digital%20Libraries Preserving
electronic publications
http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/isu350.pdf
Web archiving resources - New Zealand
National Library to Capture New Zealand's Digital Heritage
http://www.natlib.govt.nz/bin/media/pr?item=1085885702
National Library of New Zealand (Te Puna Matauranga o Aotearoa) Act
2003 http://www.natlib.govt.nz/files/Act03-19.pdf
National Library to lead electronic harvesting
http://www.natlib.govt.nz/bin/media/pr?item=1064531843
National Library of New Zealand - Digital Library Development Review
http://www.natlib.govt.nz/files/ross_report.pdf
National Library of New Zealand Preservation Metadata Schema
http://www.natlib.govt.nz/en/whatsnew/4initiatives.html#meta
Metadata Standards Framework - Preservation Metadata (revised)
http://www.natlib.govt.nz/files/4initiatives_metaschema_revised.pdf
National Library of New Zealand Preservation Metadata Data Model
http://www.natlib.govt.nz/files/nlnz_data_model.pdf
Web archiving resources - Norway
Paradigma : Preservation of Digital Material
http://www.nb.no/paradigma/eng_index.html
The Paradigma Project
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287396&reqid=118767#feature2
The Paradigma Web Harvesting Environment
http://bibnum.bnf.fr/ecdl/2003/index.html
Identification of Network Accessible Documents : Problem Areas and
Suggested Solutions http://bibnum.bnf.fr/ecdl/2003/index.html
The Paradigma Project and its Quest for Metadata Solutions and User
Services http://www.ifla.org/IV/ifla70/papers/009e-Nuys.pdf
Web archiving resources - Sweden
Kulturarw3 http://www.kb.se/english/find/internet/websites/
New Decree for Kulturarw3
http://web.archive.org/web/20020803234652/http://www.kb.se/Info/Pressmed/Arkiv/2002/020605_eng.htm
The Collection of Swedish web pages at the Royal Library - The Web Heritage
of Sweden http://www.ifla.org/IV/ifla68/papers/111-163e.pdf
Kulturarw3, the Swedish WWW-Archive : Or, to Preserve the Swedish World
Wide Web http://bibnum.bnf.fr/ecdl/2001/sweden/sld001.htm
The Kulturarw3 Project - The Royal Swedish Web Archiw3e - An Example of
'Complete' Collection of Web Pages
http://www.ifla.org/IV/ifla66/papers/154-157e.htm
The Kulturarw3 Project - The Royal Swedish Web Archiw3e
http://web.archive.org/web/20060623011336/http://www.kb.se/kw3/articles/article000605.pdf
Web archiving resources - United Kingdom
Domain UK : Britain on the Web
http://bibnum.bnf.fr/ecdl/2002/uk/uk.html
Digital Preservation Activities in the UK : Building the Infrastructure
http://www.ifla.org/IV/ifla69/papers/129e-Jones.pdf
Web Archiving : a Feasibility Study for JISC and the Wellcome Trust
http://www.jisc.ac.uk/index.cfm?name=project_webarchiving
Legal Issues Relating to the Archiving of Internet Resources in the UK,
EU, USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf
Historic Change in Legal Deposit Law Saves Electronic Publications for
Future Generations - Bill to Extend Legal Deposit to UK Non-Print Materials
Receives Royal Assent http://www.bl.uk/cgi-bin/press.cgi?story=1382
UK Government Web Archive
http://www.nationalarchives.gov.uk/preservation/archivedwebsites.htm
Web archiving resources - United States
MINERVA : Mapping the Internet Electronic Resources Virtual Archive
http://www.loc.gov/minerva/
Collections Policy Statement : Web Site Capture & Archiving
http://lcweb.loc.gov/acq/devpol/webarchive.html
Web Preservation Project Final Report : A Report to the Library of
Congress http://web.archive.org/web/20041106155137/http://www.loc.gov/minerva/webpresf.pdf
Preserving Our Digital Heritage : Plan for the National Digital
Information Infrastructure and Preservation Program
http://www.digitalpreservation.gov/library/pdf/ndiipp_plan.pdf
National Digital Information Infrastructure and Preservation Program
http://www.digitalpreservation.gov/
Editor's Interview : National Digital Information Infrastructure and
Preservation Program
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287092&reqid=118767#feature1 National
Digital Preservation Initiatives: An Overview of Developments in Australia,
France, the Netherlands, and the United Kingdom and of Related International
Activity http://www.clir.org/pubs/abstract/pub116abst.html
Web Preservation Projects at Library of Congress
http://www.archimuse.com/mw2003/papers/grotke/grotke.html
Collecting and Preserving the Web : the Minerva Prototype
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287468&reqid=118767#feature1
Building Thematic Web Collections: Challenges and Experiences from the
September 11 Web Archive and the Election 2002 Web Archive
http://bibnum.bnf.fr/ecdl/2003/index.html
Legal Issues Relating to the Archiving of Internet Resources in the UK, EU,
USA and Australia : A Study Undertaken for the JISC and Wellcome Trust
http://www.jisc.ac.uk/uploaded_documents/archiving_legal.pdf
Collaborative initiatives
Web archiving resources - IIPC
Netpreserve.org : International Internet Preservation Consortium
http://netpreserve.org/about/index.php
Web archiving resources - Internet Archive
Internet Archive http://www.archive.org/ TechNotes -- The
Internet Archive: An End to the Digital Dark Age
http://www.cswe.org/publications/jswe/03-2technotes.htm
Saving the Web http://bibnum.bnf.fr/ecdl/2002/ia/ia.html
Archiving the Internet http://web.archive.org/web/20030605113917re_/www.archive.org/ sciam_article.html
The Internet Archive, An Interview with Brewster Kahle
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287741&reqid=118767#interview
Web archiving resources - NEDLIB
NEDLIB : Networked European Deposit Library
http://nedlib.kb.nl/
Collecting and Preserving the Web: Developing and Testing the NEDLIB Harvester
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp?fileid=0000070519:000006287468&reqid=118767#feature2
NEDLIB Local Situation s
http://nedlib.kb.nl/results/local_situations_v2.htm
Long-Term Preservation of Electronic Publications: the NEDLIB Project
http://www.dlib.org/dlib/september99/vanderwerf/09vanderwerf.html
Setting up a Deposit System for Electronic Publications, the Nedlib
Guidelines http://nedlib.kb.nl/results/NEDLIBguidelines.pdf
Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html
Web archiving resources - Nordic Web Archive
NWA : Nordic Web Archive http://nwa.nb.no
Nordic Web Archive
http://www.lib.helsinki.fi/tietolinja/0100/nwa.pdf
Nordic Web Archive http://bibnum.bnf.fr/ecdl/2003/index.html
Archiving the Web : European Experiences
http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html
Articles
| Books
| Events
| Organisations and Websites
| Online tutorial
| Policies, Strategies & Guidelines
| Projects and Case Studies
| Bibliographies
| Glossaries
| Journals & Newsletters
| Discussion Lists
| Surveys
| Tools
| Evaluations
- 2007 Australian Federal Election on the Internet
Crook, Edgar
(Date Created: Dec 2007)
(Australia)
- This paper describes the challenge of archiving web sites for the Australian 2007 Federal Election in the PANDORA web archive.
- http://www.nla.gov.au/nla/staffpaper/2007/documents/Election2007.pdf
-
2nd ECDL Workshop on Web Archiving
Day, Michael
In: Cultivate Interactive
(Date Created: Nov 2002)
- A report from the 2nd ECDL Workshop on Web Archiving held in Rome on the 19th September 2002 organised by the Bibliothèque nationale de France (BnF) and the Vienna University of Technology.
- http://www.cultivate-int.org/issue8/ecdlws2/
- 3rd ECDL Workshop on Web Archiving
Day, Michael
In: Ariadne
(Date Created: 30 Oct 2003)
- A report on the 3rd ECDL Workshop on Web Archives held in Trondheim in August 2003, describing work presented on identifiers, web archiving tools and experiences in web archiving from various countries.
- http://www.ariadne.ac.uk/issue37/ecdl-web-archiving-rpt/
- 4th International Web Archiving Workshop
Day, Michael
- The 4th International Web Archiving Workshop was held at the University of Bath in September as part of ECDL 2004. This report features a paragraph is on each paper.
- http://www.ariadne.ac.uk/issue41/ecdl-web-archiving-rpt/
-
Access to Web Archives: the Nordic Web Archive Access Project
Brygfjeld, Svein Arne
(Date Created: 22 Aug 2002)
(Scandinavia)
- Presented at the 68th IFLA General Conference and Council held in Glasgow, 18-24th August, 2002, this paper describes a prototype access system for access to large-scale web archives, as developed by the Nordic Web Archive Access Project. The project is an initiative of the National Libraries of Denmark, Finland, Iceland, Norway and Sweden.
- http://www.ifla.org/IV/ifla68/papers/090-163e.pdf
- Also available in French at http://www.ifla.org/IV/ifla68/papers/090-163f.pdf
- Architecture of DAMP : a System for Harvesting and Archiving Web Publications
Milinovic, Miroslav ; Topolscak, Nebojsa
(Date Created: Jul 2005)
(Croatia)
- This article discusses the DAMP project (Digital Archive for Web Publications). DAMP is used to archive Croatian websites. It was developed by The University Computing Centre based at the University of Zagreb (SRCE) and the National and University Library (NUL) in Zagreb. The article discusses the first version of DAMP, including its architecture, the gathering process and future developments such as the instigation of an authentication and authorisation system.
- http://widwisawn.cdlr.strath.ac.uk/Issues/Vol3/issue3_3_1.html
- WIDWISAWN, Vol. 3, no. 3 (July 2005)
- Archive Server DEPOSIT.DBB.DE
Barth, Kristina; Michel, Michaela
(Regularly Updated)
(Germany)
- The German National Library's (Die Deutsche Bibliothek) storage system for the national collection of born digital and born again digital publications. At present German legal deposit law only requires the deposit of static electronic documents, but networked materials are also accepted into the system via voluntary arrangements with publishers. Subject to copyright limitations, the content of the DEPOSIT.DBB.DE archive can be searched and accessed via the Library's gateway and online catalogues.
- http://deposit.ddb.de/index_e.htm
- Archiving and Accessing Web Pages : the Goddard Library Web Capture Project
Allen, Robert B. ; Anderson, Nikkia; Hodge, Gail; Senserini, Alessandro ; Smith, Daniel
In: D-Lib magazine
(Date Created: Nov 2004)
(United States of America)
- This is a report on capturing intranet web pages by the NASA Goddard Space Flight Center. The aim of the Web Capture project was to ensure the long term preservation of this information, and to extract and index metadata in a format that is searchable. The article discusses how this was achieved including descriptions of system flow, web site selection, and the capture process. Technical limitations of the process are also discussed.
- 10.1045/november2004-hodge
- http://www.dlib.org/dlib/november04/hodge/11hodge.html
- Archiving the Canadian Web : Experiences at Library and Archives Canada
Lilleniit, Roselyn
(Date Created: Aug 2007)
(Canada)
- This article describes current practices for selection, acquisition, management and accessibility of Canadian web sites at Library and Archives Canada and outlines future plans.
- 10.1300/J123v53n01_11
- http://www.haworthpress.com/store/ArticleAbstract.asp?ID=101558
- Available in The Serials Librarian, Vol 53, No 1/2, 20 Aug 2007, pp. 139-149
- Archiving the Greek Web
Eirinaki, Magdalini; Jevtuchova, Darija; Lampos, Charalampos; Vazirgiannis, Michalis
(Date Created: 2004)
(Greece)
- This paper describes a project to archive Greek orientated web materials using the domain name (.gr), the Greek language and Greek content as selection criteria. It describes the methodology for archiving, including details on the web crawler and semantic analysis, and presents the results of the experiments.
- http://www.iwaw.net/04/Lampos.pdf
-

Archiving the Internet
Kahle, Brewster
In: Report on the WWW 8 Conference
(Date Created: Apr 1996)
- Brewster Kahle discusses his rationale for an Internet Archive of public documents. He gives insights into some of the logistical issues including those of data gathering, cost, access and collecting mandate.
It is intended that the Archive will provide resources for research and data analysis.
- http://web.archive.org/web/20030605113917re_/www.archive.org/sciam_art
icle.html
- Archiving the Slovenian Web : Recent Experiences
Grobelnik, Marko; Kavcic-Colic, Alenka
(Date Created: 2004)
(Slovenia)
- This paper reports on a two year project ending in September 2004 which aimed to develop a national repository for the long term preservation of the Slovenian web. The two main goals of the project were to develop a methodology to archive the Slovenian web and to develop a web crawler to facilitate this.
- http://www.iwaw.net/04/Kavcic.pdf
- This workshop was held in conjunction with the 8th European Conference on Research and Advanced Technologies for Digital Libraries, September 2004.
-
Archiving the Web - Some Legal Aspects
Kavcic-Colic, Alenka
(Date Created: 21 Aug 2002)
- A paper presented at the 68th IFLA General Conference and Council held in Glasgow, 18-24 August, 2002, discussing some legal aspects of archiving publications from the web. Copyright and legal deposit legislation are discussed with regard to rights implications for harvesting, public access and long-term preservation.
- http://www.ifla.org/IV/ifla68/papers/116-163e.pdf
- Also available in French at http://www.ifla.org/IV/ifla68/papers/116-163f.pdf
- Archiving the Web : European Experiences
Hakala, Juha
In: Tietolinja News
(Date Created: Oct 2003)
- This paper was presented at CONSAL XII, 20-23 October 2003, Brunei, and is an updated version of Collecting and Preserving the Web: Developing and Testing the NEDLIB Harvester, published in 2001. It outlines the outcomes of the NEDLIB Harvester Project for the archiving of Web resources. Some of the key issues in using this form of technology for preserving materials on the Web are reviewed. New developments are discussed including the Nordic Web Archive II project and the establishment of the International Internet Preservation Consortium (IIPC) to develop standards and tools for web archiving.
- http://www.lib.helsinki.fi/tietolinja/0203/webarchive.html
-
Archiving the Web : Political Party Web Sites in the Netherlands
Druiven, Henk; Keyzer, André; Voerman, Gerrit; den Hollander, Frank
(Date Created: Jun 2003)
(Netherlands)
- This article describes the Dutch project Archipol, which archives Dutch political Web sites, and examines why this information may be important to researchers in the future. Also discussed are various international initiatives to archive similar Web sites.
- http://search.epnet.com/direct.asp?an=9972863&db=aph
- Available in Information Services and Use, vol. 23, issue 1 (2003), ISSN 0167-5265; and by subscription online at www.ebscohost.com
-

Archiving the Web: The PANDORA Archive at the National Library of Australia
Cathro, Warwick; Webb, Colin; Whiting, Julie
(Date Created: Jun 2001)
(Australia)
- This paper describes the National Library of Australia's PANDORA web archive. It includes description of the Library's framework for digital collection management, the reasons for a selective approach to web archiving, the archiving processes, persistent identifiers, access control, preservation plans and opportunities for global cooperation and information sharing.
- http://www.nla.gov.au/nla/staffpaper/2001/cathro3.html
-

Archiving the World Wide Web
Lyman, Peter
In: Building a National Strategy for Digital Preservation: Issues in Digital Media Archiving
(Date Created: Apr 2002)
(United States of America)
- A paper in the series of six environmental scans commissioned as part of the US National Digital Information Infrastructure and Preservation Program (NDIIPP), this article examines archiving the web from several perspectives. As well as technical issues, such as those involved in defining the digital object, authenticity and provenance and the technologies needed for preservation, the article considers the roles of librarians and computer scientists in managing digital material and organisational issues such as copyright and the responsibility to preserve.
- http://www.clir.org/pubs/reports/pub106/web.html
- The collected papers are also available in PDF format via: http://www.clir.org/pubs/abstract/pub106abst.html
- Archiving Web Site Resources : a Records Management View
Kelly, Brian; Pennock, Marueen
(Date Created: 2006)
- This brief paper forms part of the poster exhibition at the 15th International World Wide Web Conference in 2006. The paper describes a records management approach to the archiving and maintenance of web sites so that enough metadata is collected on the resource to ensure legal and archival requirements. The authors propose several ways to facilitate this including collaboration between record creators and managers, education and training.
- http://www.dcc.ac.uk/docs/WWW2006_Archiving_Web_Site_Resources.pdf
- Poster presented at the WWW 2006 conference, Edinburgh Scotland
- Archiving Web Sites for Preservation and Access
Guenther, Rebecca
(Date Created: Apr 2007)
(United States of America)
- This article examines the technical challenges in harvesting and managing web archives using as an example the Library of Congress' Minerva project.
- http://web.ebscohost.com/ehost/detail?vid=12&hid=17&sid=a15a30f8-9aa2-
45e7-86ee-d138a9fb323a%40sessionmgr108
- Journal of Archival Organization, April 2007, Vol 4 No1-2, p 141-166 and by subscription through EBSCOhost
-

Archiving Websites
Boudrez, Filip; Van den Eynde, Sofie
(Date Created: Jul 2002)
(Belgium)
- A report from the Flemish DAVID Project examining the need for archiving of web sites, approaches for collection and management, and a range of legal issues including copyright. The report is available in PDF format.
- http://www.expertisecentrumdavid.be/davidproject/teksten/Rapporten/Rep
ort5.pdf
- Archiving, Indexing and Accessing Web Materials: Solutions for Large Amounts of Data
Cowart, Charles; Minor, David; Moore, Reagan; Zhu, Bing
(Date Created: 23 Jun 2007)
(United States of America)
- This article describes two different technologies applied by the San Diego Super Computer Centre for archiving, indexing and accessing web archived materials for two projects: Library of Congress/San Diego Supercomputer Center and the National Science Digital Library.
- http://iwaw.net/07/IWAW2007_minor.pdf
- Paper presented at IWAW 2007.
- Australian Web Domain Harvests: A Preliminary Quantitative Analysis of the Archive Data
Koerbin, Paul
(Date Created: 15 Apr 2008)
(Australia)
- This report provides a preliminary quantitative analysis of content from three Australian web domain harvests conducted between 2005 and 2007. It reports on the size of the Australian web harvests; number and percentage of hosts and URLs by 2nd level domain; MIME types and changes between the 2006 and 2007 harvests.
- http://pandora.nla.gov.au/documents/auscrawls.pdf
-

Austrian Online Archive Processing: Analyzing Archives of the World Wide Web
Aschenbrenner, Andreas; Rauber, Andreas; Witvoet, Oliver
(Date Created: Sep 2002)
(Austria)
- A paper describing the use of OLAP technologies (On-Line Analytical Processing) to build a data warehouse of a web archive. Data warehousing allows exploration and analysis of technical aspects of the archive, such as the use of operating systems, scripts and file types, from which inferences of change over time may be drawn.
- http://www.ifs.tuwien.ac.at/ifs/research/pub_pdf/rau_ecdl02.pdf
-

Blog Today, Gone Tomorrow? Preservation of Weblogs
Entlich, Richard
In: RLG DigiNews
(Date Created: 15 Aug 2004)
- Blogs (weblogs) are a recent development on the Internet and they are becoming more popular as a communication tool. The article outlines the nature and ephemeral life of weblogs and how content can be lost, and also looks at the nascent efforts to preserve them.
- http://digitalarchive.oclc.org/da/ViewObject.jsp?objid=0000070519&reqi
d=4139
- Characteristics of .au Websites: An Analysis of Large-Scale Web Crawl Data from 2005
Ackland, Robert; Bailey, Peter; Spink, Amanda
(Date Created: 02 Jul 2007)
(Australia)
- This paper provides a preliminary analysis of .au domain websites collected by the Commonwealth Scientific and Industrial Research Organisation (CSIRO) in 2005. It is a first step in the planned ".au Census" project.
- http://ausweb.scu.edu.au/aw07/papers/refereed/spink/paper.html
- Paper presented at AusWeb07, the thirteenth Australasian World Wide Web Conference, Coffs Harbour NSW 30 June - 4 July 2007
-

Collecting and Managing Web Resources for Long-Term Access : Web Harvesting and Guidelines to Support Preservation (ICABS Actions 3.3 and 3.4)
Gatenby, Pam
(Date Created: 2004)
(Australia)
- This paper outlining the National Library of Australia’s recent contributions to the advancement of understanding of international Web archiving issues, was presented at the 70th IFLA Conference in August 2004. The NLA’s work on web harvesting and the preservation of digital materials as part of its ICABS responsibilities are detailed, together with its involvement in the International Internet Preservation Consortium’s (IIPC) Deep Web Working Group.
- http://www.ifla.org/IV/ifla70/papers/026e-Gatenby.pdf
-

|