Staff Papers

"… and then there’s the rest of the stuff …"
The role of the PANDORA Archive

Presentation by Paul Koerbin, Supervisor, Digital Archiving Section, to the Australasian Sound Recording Association 2004 Conference “Sounding Out a Cultural Heritage”, Session title “Delivering the Digits”, 13 May 2004

Introduction

In this talk I want to give a brief overview of the PANDORA Archive: what it is and how it works; talk briefly about the archiving role of PANDORA; consider some of the constraints and issues we face in trying to fulfil this role; and suggest the importance of cooperation and collaboration in these efforts. I should make it clear at the outset that the PANDORA is not specifically a sound archive; it is an archive of online, web resources. While we do confront the archiving of multimedia resources, this is in the context of all “the rest of the stuff”, by which I mean all those resources accessible over the Internet: the web-page content that helps to provide cultural context and material that will be of use to future researchers – or so we hope.

The PANDORA Archive in brief

It is now more than eight years since the National Library began developing and implementing the policy, procedures and systems to select, capture and archive Australian online resources. The outcome of this endeavour is what we call PANDORA: Australia’s Web Archive.

For a number of years now, archiving web sites for PANDORA has been a routine part of the selection, acquisition and collection management processes of the National Library. Moreover PANDORA is increasingly becoming a cooperative effort, led by the National Library. Currently, there are nine PANDORA partners in addition to the National Library, including all mainland state libraries, the Australian War Memorial, the Australian Institute of Aboriginal and Torres Strait Islander Studies, and ScreenSound Australia. ScreenSound was in fact the second partner to join PANDORA.

The development of a suitable and efficient management system to facilitate the selection, gathering of and access to archived online resources, has been a significant task for the Library in recent years. A web-based system, able to be used by all PANDORA partners wherever they are, was developed by the Library and first implemented in June 2001. This system is known as PANDAS (the PANDORA Digital Archiving System). A second, significantly enhanced version was implemented in August 2002 and we are continuing to develop and enhance the functionality of the system. PANDAS has attracted the interest of many institutions concerned with archiving digital resources in Australia, the United Kingdom, the United States and elsewhere.

In order to support long-term preservation and access to the archived resources, PANDORA forms part of the overall framework for digital collection management at the Library. So, for example, PANDORA resources are an important focus of the Library’s digital preservation activities; preservation copies of the archived resources are maintained on the Library’s Digital Object Storage System; and access is enhanced through the application of the Library’s persistent identification scheme to the archived resources.

The Library is also involved in international research activities and is part of the International Internet Preservation Consortium, established in 2003. As a lead participant in the Consortium’s Deep Web Archiving Working Group, we are currently investigating the complex issues of archiving databases with the aim of developing a practical means to preserve and provide access to such data sets.

The role of PANDORA in archiving and preservation

The National Library’s function under the National Library Act 1960 is to maintain and develop a national collection of library material including a comprehensive collection of library material relating to Australia and the Australian people; and indeed to make this material accessible. With the advent of the Internet as a major medium for the publication of such material, the Library established the PANDORA Archive as a means to collect and manage online resources. While institutional specific repositories are becoming more common in academic institutions, particularly with the availability of open source systems like DSpace to manage them, there remains “all the rest of the stuff”, those online resources that are not able or likely to be preserved and made accessible, in the long-term, by their creators or publishers, whether individuals or institutions.

What particularly characterises the role of PANDORA is the scope of what it potentially has to deal with. In broad terms this is anything “published” online that is Australian or about Australia or of significance to Australia. And, it must be said, we work with a broad definition of “published” in regard to online resources. In the PANDORA selection guidelines this is summarised as:

“… information, regardless of its format or method of delivery, that is made available to the general public, or to an identified public, either free of charge or for a fee.”

The PANDORA Selection Guidelines identify priority areas for collection and also identify some types of publications that, given currently available resources, will not be archived. Beyond that, anything of relevance or significance to Australia’s cultural heritage is a potential candidate for archiving. The technical constraints of the archiving technology, for example, do not constitute a criterion to reject at title at the selection stage.

Among the selection intentions identified in the guidelines, content, authority and research value – the latter defined somewhat narrowly in the Guidelines as a “substantial compilation of information” – are identified as significant principles in determining selection. A principle not explicitly stated but which is unavoidable in the selection process is the matter of context. We have to define what the entity is that we will archive. In doing this, consideration is given both to the need to be able to provide useful access and discovery paths to the archived resource; and to the available resources to undertake the archiving process. It is not simply a matter of determining that we will always archive at the whole web site level nor that we will archive at the smallest definable part. This is a matter of decision and is a consequence of the selective approach to archiving. Clearly, such decisions do have implications for researchers

Limitations and problems

The job of archiving web resources involves, to state the obvious, confronting many constraints, limitations and problems. I want to consider just a few of these, including problems associated with selectivity, legal issues and technical constraints. This is really the important part of my talk since these issues have implications for how researchers may expect to use an archive such as PANDORA. Selectivity

Selectivity

The fact that the PANDORA Archive is a selective archive is inherently problematic when considered as a resource for future researchers. The scope of the Archive in terms of subject coverage is not limited and therefore must take a wide purview. Selection decisions are made by the archiving team based on the PANDORA Selection Guidelines which identify priority areas of collection. Broadly speaking priorities are focused on government, academic and publications of substantial content. But staff obviously do not have expertise across all possible subject areas. Nevertheless judgements have to be made as to what is selected and what is not.

Some other notable international web archives operate on a different model and aim to harvest as much of the Internet as possible; or a specified domain. This approach has attractions in as much as it aims to retain the larger context of the Internet. As Catherine Lupovici, Head of the Digital Library Department at the Bibliotheque nationale de France argued at this year’s VALA conference, the context of the site within the wider net-scape, and the hyperlinks to and from a resource, are a fundamental part of that resource since this is the intrinsic nature of the Web. To this way of thinking therefore the only really satisfactory option is to archive the whole. Another attraction of this approach is that the archivers do not need to second guess what researchers in the future will want from the archives.

While the attractions of a whole of domain harvesting approach are perhaps self-evident, there are problems with this approach too. The scale of whole domain archiving means that you have to be reliant upon your harvesters. They have to cover a lot of territory and you cannot realistically manually check what has been harvested for completeness and functionality. A little time spent perusing the Internet Archive [1] (a massive attempt to archive the whole Internet – or a large part of it) will show you just how piecemeal, incomplete, dysfunctional and frustrating such archiving can be when trying to access these resources for research purposes. By highlighting the problems of the Internet Archive approach, I don’t mean to disparage what is a truly great resource. But whole domain crawling is not in itself the answer, yet anyway. Whole domain archiving is important and work is being done to create the necessary tools to refine such archiving and provide useful access to the results of such harvesting. Perhaps the ideal would be a combination of both whole domain and selective archiving. Indeed, the Internet Archive has been involved in such an approach in collaboration with the United States Library of Congress in doing specialised quality assessed collections for the 2002 US elections and the September 11 collection.

The selective approach, being scalable, when viewed beside the problems encountered with whole domain archiving does provide some positive outcomes for future researchers because time can be spent ensuring the archived instances are as complete as possible. Thus there is more likelihood of the resource being functional and useful in the future; and perhaps too we may be more usefully able to apply preservation processes to such resources since more information can be gathered about the resource. Greater confidence about the completeness of the resource also means that it is more obviously justifiable to invest in preservation strategies to maintain it and maintain access to it. Of course these are functions that in time we would hope to automate as much as possible to allow for a greater scale of archiving.

To illustrate the sort of problems I have been alluding to, when we archive sites with Real Media files the harvesters only capture the .ram metafile. This metafile includes a reference to the actual media file usually delivered from a secure Real Media server. Therefore, in such cases, the only way to capture the actual media file and include it in the archive is to obtain the file by means other than automated harvesting – in this case having the publisher supply the original media files and PANDORA staff re-referencing the archived metafiles to point to the archived media file. Because the selective approach to archiving is essentially only scaleable to the resources available to do the job, it does allow for a reasonably high degree of manual quality assessment that will identify missing resources and take action to obtain them to make the archived resources as complete and functional as possible.

Legal

Legal concerns are another serious constraint on archiving. Since the National Library does not own copyright in the “the rest of the stuff” – the web sites – we are archiving, we must obtain a copyright licence from the owner in order to copy the files and maintain them in the Archive. It is clearly little help to researchers if we archive something but cannot legally provide the researcher with access to it.

The selective approach to archiving allows us to pursue these copyright licences and negotiate any access restrictions or conditions with the publisher. Obviously this is not possible with whole domain archiving unless the Archive is covered by legal deposit provisions to copy, maintain and provide access to the resources. Currently Australia does not have such legislation at the Commonwealth level.

Whole domain archiving, then, is left with either a situation where the archived resources cannot be made widely available (although it may be possible to make the resource available through one in-house terminal for example); or, like the Internet Archive, they require their harvesting robots to comply with robot.txt harvest refusal rules. The Internet Archive also comply with post-archiving requests from copyright owners to “takedown” the files from the Archive.

When archiving for the PANDORA Archive we do not require our harvesters to comply with robot.txt rules, since we seek permission to archive prior to harvesting. Our licence permission should also protect us from future requests to remove resources from the Archive. Provided of course we are scrupulous in our archiving parameters. We have indeed in recent times had a growing number of takedown requests – which attests to the risk this poses to an archive of material not owned by the archiving agency – and, due to weaknesses in the form of the license permissions we have received in the past, we have had no option but to comply with a number of these requests. We have since tightened up our licence agreements. The point to note is that it is fundamentally antithetical to the archive process to remove files from the archive – indeed our archive is technically configured so to maintain the integrity of the archive and make such removal a difficult process. Moreover, since persistent identifiers are given to archived resources to assist researchers with reliable access and citation, we must take what steps we can to eliminate the risk of such tampering with the integrity of the archive.

The nature of publishing on the Internet is that illegal, or perhaps the non-negotiated or vaguely negotiated, use of copyrighted material is common. In obtaining our license permissions we are dependent upon the publisher of the resource we are archiving to ensure all rights of use are in order. This remains a risk and a troublesome issue, but at least our experience with the selection and negotiation process can go some way to identifying resources that are likely to be risky or dubious publications to archive. Of concern in this environment is that such publications may be of no less interest to researchers than those that are scrupulously and legitimately published and managed. The introduction of legal deposit provisions to cover electronic resources would at least greatly assist the archiving process in our jurisdiction, while not eliminating the risk of Australian material unlawfully using overseas copyrighted material which may yet compromise the archived resource.

Technical constraints

PANDORA currently operates primarily on a “pull’ model. That is, we harvest or download the files from the Internet. However, we can accommodate a push model as well and do indeed obtain some resources in this way, either by email, FTP or on a physical medium.

It is important to note is that we are archiving are resources as delivered over the Internet – that is the resources as presented through the Internet browser. Unlike digitised or some born-digital resources acquired by the Library we do not necessarily have the creator’s original files to work with. Certainly the audiovisual files that we obtain are the format versions as presented for download over the Internet from the original web site such as Real Media, mpg or QuickTime files. Of course these are often of a low quality, but functional for delivery over the Web.

Harvesting files from the Internet is the most efficient means of obtaining the resources for archiving. However there are technical constraints with this method since the harvesting software (crawlers, spiders, indexers) currently available to us can only go so far and do so much. We can only use the software to download those files it is able to locate and follow. It is commonplace now for web sites to be delivered using active server technology where the content is delivered from database applications. This in itself is not an impediment to the gatherers provided all the resources have links that can be understood by the crawler. However, complex JavaScript, cascading menus and deep web resources where content can only be accessed by intelligent interrogation on behalf of the user, are all barriers to the crawler software.

Sites that deliver multimedia, sound and video can also, but not necessarily, be an impediment. I have already mentioned the problem with Real Media’s use of metafiles. But capturing live webcasts is not possible for us as yet. Thus for these sort of sites we have to attempt to obtain files from the publisher [2].

When we are dealing with “the rest of the stuff” we are faced with potentially any and every conceivable type of web site, file format and content delivery system. Even if we are able to obtain the files from the publisher we may still encounter problems in delivering that content from our server; or at least it may require considerable resources on our part to do so.

Broadly speaking we take the view that we should retain the look and feel of the resource as well as the content. This raises one of the difficult issues faced by archivers: namely, what the significant properties of an archived resource are for preservation purposes. An issue that is particularly difficult and dense when the archived resource is a complex object like a web site. We would aim to retain the resource as an entity recognisable as that which was originally archived, but we don’t know if this is possible into the future and some compromise may have to be made. This is an issue that researchers need to be aware of given that the Internet is a pre-eminent medium for the presentation of information.

Cooperation and collaboration

When faced with archiving the Internet (or domains of it) it is obvious there are major issues in respect of providing a useful resource for researchers. The scale of the task focuses the need for technical solutions and tools for efficient and reliable harvesting, quality analysis, functional access and resource discovery. Moreover, the fact that much content accessible via the Internet is deep-web and not able to be readily harvested by crawlers is a major impediment to those of us committed to archiving quality content.

Such issues are deeply felt by those institutions and organisations who have been concerned with archiving online resources for some time. In July 2003 a number of these bodies came together to launch the International Internet Preservation Consortium [3] to address these issues in a practical and collaborative way. This Consortium includes major players in the archiving world such as the major Nordic libraries, the Bibliotheque national de France, the Library of Congress, the British Library, the National Library of Australia and the Internet Archive (among others). It is the objective of this Consortium to develop tools, standards, methods, policy and requirements that will advance and coordinate our ability to undertake the significant task of archiving web resources.

Meanwhile we really have to do what we can while recognising that we cannot attempt to archive and preserve all the resources that all prospective researches may need. It is surely better for research outcomes that we are (and have been for some time) actually doing something in terms of archiving – not just thinking about it (while recognising that thinking and re-evaluation is important); and not just modelling (although that too is important). Because, with all its limitations at least what we do have are quality resources we can aim to maintain and provide access to now and in the future. The ephemeral nature of online resources means you do not get second chances when a resource can disappear in an instant. Already about 6% of the titles archived in PANDORA are known to have disappeared from the web. Other sites may move so that links become broken; whereas the archive provides persistent and stable access.

The selective approach is scaleable to available resources and it follows that to achieve the best results for researchers it is important that there is cooperation and collaboration. ScreenSound became a PANDORA partner quite early on – they began archiving in 2000. Thus specific expertise in film and sound resources can be brought to the selection process. In other areas we engage indexing and abstracting agencies to provide us with proposals for archiving. Again this provides for expert input into the selection process and, significantly, the indexing/abstracting agency is able to include a persistent citation in their records – surely something that researchers require.

Conclusion

The selective approach with its scalability and concentration on quality of archiving was a decision and approach the Library continues to support. It continues to see a role for this approach into the future. This approach has the advantage of allowing archiving to have begun fairly early on, has helped us to identify and address archiving issues, and produce an active and accessible web archive. However, we also recognise the limitations of such an approach and the need to work on a larger scale in order to really attempt to meet the need of future researchers. As such we are committed to our involvement with the International Internet Preservation Consortium – and our leading role in the Consortium’s deep-web archiving component – and to the need to develop technical solutions to automate the archiving of online resources as much as possible.

Endnotes

  1. http://www.archive.org/

  2. An example is the Official 1998 Mardi Gras Netcast, http://nla.gov.au/nla.arc-10023

  3. http://www.netpreserve.org

Top

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us