Staff Papers
PANDORA - Towards a national collection of Australian electronic publications
A paper presented by Fiona Ash Relf, Australian Electronic Unit, at the 10th national Library Technicians' conference, Fremantle, 8-10 September 1999.
Abstract
The National Library of Australia strongly believes that the preservation of Australian online publications is an essential component of the Librarys broader responsibility to preserve and make available Australias documentary heritage. In June 1996 the National Library established the PANDORA (Preserving and Accessing Networked Documentary Resources of Australia) Project in order to develop policies and procedures for the selection, capture, archiving and provision of long-term access to Australian electronic publications. By the end of 1997 a functional proof-of-concept archive existed and policy documents outlining the conceptual framework for a permanent electronic archive had been developed. The Projects next challenge is the development of a working model for a national collection of Australian electronic publications. Another priority is to foster working partnerships with other national libraries and overseas agencies who are also undertaking research and development in this area.
This paper looks briefly at the PANDORA Project and its key business principles. Collaborative work with the State libraries to develop a viable distributed model for preserving Australian online publications is also discussed. Some issues that need addressing in order to ensure the ultimate success of the PANDORA Project, for example permanent naming, authentication and rights management, are also explored.
Introduction
The purpose of this paper is to outline the National Library of Australias response to the challenge of preserving and providing long term access to Australian electronic publications. Preservation and enabling future access to publications have always been the core functions of libraries and will remain so in our electronic age. This paper seeks to emphasise the importance of preserving Australias electronic publications as an integral part of the national imprint
The National Librarys role includes preserving the documentary resources of Australia and it once attempted to do so exhaustively. In more recent years, the Distributed National Collection concept has led to cooperative collecting and preservation arrangements amongst Australian libraries. This approach was developed long before electronic publishing emerged and still works well for traditional library materials. However, the recent explosion in electronic publishing has now seriously undermined the ability of Australian libraries to maintain this level of control over the national imprint. Extending the Distributed National Collection concept to include electronic publications is a practical solution for dealing with this new publishing phenomenon.
The PANDORA Project is the first step along the road towards this aim.
With collaboration between libraries, academic institutions, and the creators and publishers of electronic publications, Australian libraries can still manage their preservation and access functions in the electronic age. The Project seeks to create a workable national model to meet this objective. A national approach has facilitated the successful management of Australias national imprint up until now, and is even more essential if libraries are to cope with the vast amount of electronic material being published.
This paper will cover some of the background and policy decisions surrounding the Project. It will also briefly outline the processes and problems involved in the archiving of electronic publications and discuss some of the issues affecting preservation and access. The current move towards a national approach to this issue is described and its importance underlined.
1. The PANDORA Project and its key business principles
The PANDORA Projects overall objective is to develop policy and procedures for preserving and making available Australian electronic material while simultaneously creating an archive of selected Australian Internet publications. PANDORA is a catchy acronym and stands for Preserving and Accessing Networked Documentary Resources of Australia. But the name also brings to mind the story of Pandoras box from Greek mythology. One dictionary definition of the term Pandoras box describes it as any source of extensive troubles, especially one expected at first to yield blessings (Penguin Macquarie Dictionary 1986). However PANDORAs staff prefer to take heart in the Roman version of the myth which states that hope remained in the box after all the evils had departed! Definitions and mythology aside, the National Librarys PANDORA Project has achieved a great deal both nationally and internationally and is a source of much pride for the National Library.
1.1 The Project and its purpose
The Project was set up three years ago to develop and implement procedures for the selection, capture, archiving and provision of long-term access to online Australian publications. Right from the outset, the Projects ultimate goal has been to utilise the knowledge gained to develop a proposal for a national strategy for preserving electronic publications. The approach taken has involved the creation of a proof-of-concept archive, which in turn has provided a practical framework for the development and testing of policies and procedures. Strategies adopted to achieve these outcomes have included:
- Assessing the technical infrastructure requirements of a management and preservation system for the archive
- Evaluating various technical methods, standards and products for creating and providing access to the archive
- Estimating the financial, human and equipment resources needed for ongoing management of the archive
To achieve these aims the Project has involved staff from various areas of the Library contributing their expertise as necessary. The ongoing work of developing and managing the PANDORA Archive is currently undertaken by the Librarys Electronic Unit, (comprising six staff at present), in conjunction with significant IT assistance.
1.2 The proof-of-concept archive
The PANDORA Archive is the Projects showpiece (visit the Archive at: http://pandora.nla.gov.au/ )1. The Archive is impressive not only because it represents the information future, but more poignantly, it already represents the past as well. We are admittedly talking recent past now, but for researchers in the decades and centuries to come its value will be immeasurable. Some of the publications archived have already disappeared from the Internet and now only exist in the Archive. The archived version has thus become irreplaceable. Others have changed location or presentation and those changes have been tracked and recorded. But as the Internet continues to grow and electronic publications proliferate, PANDORA is less capable of keeping up, largely because of resource constraints. A broader national approach must be taken sooner rather than later to address this and extending the concept of the Distributed National Collection is one way to do this.
It must be stressed that the Archive is NOT an archive of digitised traditional library materials like manuscripts or pictorial works, but rather an archive of digitally-born publications. It is an ongoing initiative that also serves as an experimental platform for trying out new procedures, policies and technical solutions. PANDORA is a fully functional live archive that is growing steadily and is accessible to the public via Kinetica and the Librarys OPAC. At the end of June this year there were 229 titles in the Archive and many of these titles contain multiple versions or issues. At least that many titles again have been selected for inclusion but have not yet been archived largely for technical or resource reasons.
1.3 Key business principles
In order to commence the business of archiving, the PANDORA Project came up with a set of principles to guide their activities. These principles still govern the way the Archive is managed on a daily basis. They include:
- Rigorous selection of titles
- Gratis access
- Retention of the look and feel of titles wherever possible
- Exclusion of titles which have a print or microform equivalent and therefore can be preserved in another form
- Archiving only links internal to the selected site and disallowing links to other sites
These principles were largely developed for practical reasons governed by resource and technical constraints. They are subject to revision as the Project develops and more Australian libraries become involved. The selection guidelines in particular may be relaxed and broadened as an improved technical infrastructure reduces the requirement for the labour-intensive procedures currently undertaken.
2. Creating the Archive and problems encountered
The process of creating the Archive has thrown up innumerable challenges. As no precedent existed, each procedure was developed from scratch and then strenuously tested. In effect a parallel workflow was designed to operate alongside traditional library materials. Given the current complexity of procedures and the technological problems encountered, it seems unlikely that the work will be integrated with traditional technical services work in the short to medium term. The work as it stands today requires staff possess highly-developed technical services skills, competence with serials, strong problem-solving abilities and IT proficiency. These can be onerous staff demands to make and lend weight to the argument for increased national cooperation.
The actual work processes undertaken are briefly outlined below:
2.1 Selection
In building the Archive, the National Library has chosen to take a selective rather than inclusive approach for a number of important reasons. Primarily, it remains a truism that many Internet publications have little current or long term research value. Selecting high quality publications, providing them with full descriptions in the National Bibliographic Database and then ensuring they operate and are accessible seems the best strategy at present (Cameron & Phillips, in press). To fulfil this goal the Librarys current approach is to archive selected publications in a dedicated repository, keep it updated and then as need arises, migrate the data in response to technological change (Smith 1997).
The selection of electronic publications for the archive is governed by PANDORAs selection guidelines (see http://www.nla.gov.au/scoap/guidelines.html ) They act as a guide and are not strictly prescriptive. Generally, works should be about Australia, or be on a topic of significance and relevance to Australia and by an Australian author, or be written by an Australian of recognised authority and be of international significance. Scholarly publications, publications of national interest and those of current and long term research value are selected. Other publications are included as cultural snapshots or if they represent innovative Internet usage. Some categories of titles excluded are: those with a print or microform equivalent, online databases, discussion lists and bulletin boards, and titles of local rather than national significance.
The Electronic Unit carries out regular Internet searching to locate suitable works but many titles are also referred to the Unit through various channels. All titles considered for selection are brought to regular group selection sessions involving all Electronic Unit staff. These sessions are essential for maintaining the Archives integrity and ensuring only the most appropriate titles are included. Due to the complex nature of many electronic publications it is often very difficult to single-handedly decide if a title should be included or not. Titles are never rejected solely on the basis that they will be technically difficult to archive.
2.2 Work processes
The actual process of getting a title into the archive involves:
- Evaluating the publication to determine its extent, file structure, peculiarities etc.
- Gaining permission from the publisher to archive. This is essential until such time as Commonwealth legal deposit legislation covers electronic publications.
- Cataloguing the title onto the National Bibliographic Database to ensure the publication is searchable, accessible, and notified to other libraries. A hotlink to the publication is also provided.
- Submitting a request to archive the publication to the archive management system. This instructs a robot called Harvest to gather the files from the Internet. Sometimes other software called WebZip is used or the publisher is asked to send the files.
- Comparing the gathered (collected) version with the publication as it exists on the Internet. This involves checking every page and its links and can be extremely time consuming.
- Submitting an error report if there are problems with the gathered version and waiting for our IT people to fix it.
- Constructing an entry page for the title in the Archive and allocating a PURL, (Persistent Uniform Resource Locator), to it. This is a more stable form of URL through which changes in Internet addresses can be tracked.
- Repeating the gathering and comparison processes periodically for titles such as serials that need gathering more than once.
2.3 Problems encountered
The problems encountered with the above processes are numerous and can dramatically extend the time and effort it takes to archive a publication. Problems frequently encountered include:
- Difficulties in obtaining permission from publishers to go ahead (e.g. no reply received, confusion over our request, other communication problems)
- Problems with gathering or displaying the files (e.g. Java-based files and databases cannot be gathered, random images will regularly not be gathered by Harvest, some titles will not display properly in Netscape)
- Sites disappearing before we are able to archive them or changing location in between periodic gathers
- Cataloguing difficulties (e.g. the descriptive cataloguing rules for online titles are more complex, the publications often very difficult to describe in a traditional sense, more revisions are necessary if the title moves or disappears)
The length of time taken to complete all the work processes for even relatively straightforward titles requires high levels of commitment and patience from the staff involved.
3. Issues in archiving and provision of access
The creation of the PANDORA Archive is a tremendous achievement but its existence automatically means that a whole range of other issues must be considered and worked through. Libraries around the world are exploring these issues and the National Library is cooperating internationally in various ways. These issues can be roughly divided into three categories:
3.1 Technology issues
This is the area where the biggest challenges for the future of preserving and managing archives of electronic publications lie.
- If libraries are to make serious efforts at archiving they need systems which support and automate many of the processes. Only sophisticated, robust, large-scale systems will be able to automate the collection, storage, management, preservation and provision of long term access to online publications. At present such systems have not been employed for this purpose. The Librarys Digital Services Project, (see http://www.nla.gov.au/dsp/ ), is testing the availability of such systems to provide the necessary technical infrastructure, not solely for the PANDORA Archive, but also for other digital collections in the Library as well.
- Electronic publications need permanent naming which allows unique and persistent identification of electronic publications for national bibliographic and resource discovery purposes. The National Library currently uses PURLS but these are still an address and not a name. Future options may include URNS (Uniform Resource Names) or DOIs (Digital Object Identifiers). Whichever method is chosen in the future will need to be supported by an international technical infrastructure.
- Preservation techniques include refreshing, migration, emulation, and hardware and software museums. A mix of these will probably be applied in practice to ensure future access to publications. The National Library has decided one copy of each version of a title will be kept in original format, then copies will be made and migrated as necessary when changes in software, file format or technology platform occur.
3.2 Access issues
- A review of the Copyright Act 1968 is currently taking place. This will hopefully extend legal deposit provisions at the national level to cover electronic publications. This should streamline our contacts with publishers of electronic publications. Copyright is often complicated with electronic publications as there may be many creators contributing to one publication. In many electronic journals, creators of individual articles often retain copyright.
- A rights management system is needed to monitor access to digital publications, manage their use and access as required by the creator and deal with payment.
- The integrity of an electronic resource must be protected by methods such as encryption or watermarking. This ensures the resource cannot be tampered with. A related area is validation of resources, commonly done by applying copyright and/or other metadata. This confirms which institution or person is providing the resource and who owns the intellectual property.
- Metadata, or structured data, allows resource discovery and access but is also essential for preservation and administrative purposes. There has been much international action concerning metadata standards in the last few years. To allow data exchange and access to take place between national archives, metadata must be widely adopted and standardised. The PANDORA Projects Logical Data Model, (see http://www.nla.gov.au/pandora/ldmv2.html ), defines the metadata that ideally needs to be recorded for the Archive. In the future there may be a national metadata repository to aid resource discovery below the title level.
- Authentication is necessary to ensure the person accessing an electronic resource is who they say they are. Most commonly a log-in and password or a credit card number verifies this. A related issue is authorisation which allows an authenticated user to access certain resources or services but not others. This is mostly managed by levels of access being attached to particular passwords or credit card numbers.
3.3 Library issues
- Cataloguing is vital for national and international integrated resource discovery. The National Library has adopted the view that electronic publications comprise a vital component of the national imprint. As such, they must be given the same treatment as traditional library materials and be included in the National Bibliographic Database. Only in this way will the full potential of electronic archives be realised.
4. Moving towards a National Collection
The development of a National Collection of Australian Electronic Publications is considered essential by the Library, if Australia is to successfully preserve and make available electronic publications. The huge volume of material and high costs involved in archiving and preservation means networking will benefit all partners. To this end the Library is seeking cooperation from other major libraries in Australia to share the responsibility. In a national model, it is envisaged that State libraries will ultimately store and manage their own distributed archives on their own servers, with all titles accessible nationally through the National Bibliographic Database. In the shorter term the National Library is prepared to host State libraries collections if requested. Discussions have also taken place with university libraries although they have not yet been considered as formal partners. At this stage of development, what matters is not how each library manages the material they are responsible for but rather that the end result is a co-operatively managed distributed national archive which is based firmly on an agreed set of principles (Cameron 1999). These principles are outlined in 4.2 below.
4.1.Current situation
The State Library of Victoria is currently the only formal partner in PANDORA. It is selecting, archiving and managing Victorian electronic publications and at present is using PANDORAs server to house its archived material. The National Library of Australia provides help with technical matters and training. The State Library of New South Wales and the State Library of Tasmania are carrying out their own archiving projects. Ongoing discussions are currently taking place with other State libraries and there has been much interest expressed in PANDORAs aims and objectives.
4.2 Principles for a national model
The National Library considers it vital that for a national model to succeed the following principles are adopted:
- A set of collecting agreements that reduces duplication in collecting, will ensure there is coordinated movement in the same direction and costs are minimised.
- Agreement that electronic publications comprise part of the national bibliography and should therefore be catalogued onto the National Bibliographic Database and be given a unified point of access. Cataloguing at title level will integrate electronic publications with other formats.
- A commitment to permanence in this area means not only archiving, but also cooperating in the area of long term preservation.
- Pursuing the goal of ultimate free availability over open networks with publishers of electronic titles, for maximum user benefit. The National Library favours immediate onsite access with full national access being provided after an agreed period of time.
4.3 Adoption of international standards
The National Library is participating in discussions with other national libraries to share knowledge gained, and work towards the development of international protocols governing access between digital archives. There is much work being done at the international level in the area of electronic archiving. Significantly, Australia joins only a handful of other countries that have actually created archives of electronic publications and our experience is consequently highly valued internationally.
Conclusion
Hopefully the points made above have helped demystify the concept and practice of electronic archiving. And hopefully this paper has stimulated interest in Australian electronic publishing and its place in the national imprint. It deserves reiterating once again that the only way Australian libraries can effectively manage this emerging information medium is to collaborate.
To conclude this paper, the following quote from Chilvers and Feather (1998) highlights the most definitive aspect of electronic publishing:
When asked if electronic records were short-term in nature an archivist expressed the view that I suppose yes unless you ensure otherwise. That they are short-term in terms of life expectancy of the media and access compared to if they were in print, it is perhaps their most distinctive feature (p. 368)
This statement emphasises the inherently transient and fragile nature of Internet publishing. The Internet has entrenched the idea of instant access to information but that does not make this information less valuable. Librarians must treat electronic publications as equal partners to traditional library materials or risk losing this information altogether.
Australians have wholeheartedly embraced the Internet and our libraries must recognise that the Internet is rapidly becoming the first choice for information exchange. With Australias vast geographic distances and sparse population, the nations libraries must cooperate to manage this phenomenon. In the end it does not matter how we do it, but it does matter a great deal if we do not do it.
Bibliography
Cameron, J. (1999) National Collection of Australian Electronic
Publications
(URL: http://www.nla.gov.au/nla/staffpaper/cameron1.html
)
Cameron, J. & Phillips, M. (in press) Building national collections of Internet publications. In World libraries on the Information Superhighway
Chilvers, A. & Feather, J. (1998) The management of digital data: A metadata approach. The Electronic Library 16 (6), pp. 365-372
The Penguin Macquarie dictionary: The international dictionary for all Australians (1986) Ringwood, Vic.: Penguin Books.
Smith, W. (1997) Future access: Long term preservation of Australian
electronic legal publications
(URL: http://www.nla.gov.au/nla/staffpaper/wsmith2.html)
1 30 May 2001 - Link to Pandora Archive changed from http://www.nla.gov.au/pandora/archive.html to correct link of http://pandora.nla.gov.au/
