Staff Papers
The Preservation of Internet Publications
Paper delivered at WWW7 Conference (Brisbane : April 1998) by Margaret Phillips (Electronic Unit)
Electronic formats are the latest in a long line of media that human kind has used to record our knowledge and experiences. Certainly, an HTML document on a computer screen doesnt seem to us now as valuable or romantic as an illuminated manuscript on aged parchment. But to researchers in the future, 100, 200, 500 years hence, the content, at least, may be very important. They may never see the original HTML format, the technology to support it having long since disappeared, but they are likely to be as interested in this incunabular period for electronic publications, as we are in early printed books.
Along with a number of other national libraries around the world, the National Library of Australia considers that its legislated mandate to collect print publications must also be extended to include electronic. For well over two years now, the PANDORA Project at the National Library of Australia has been addressing issues relating to the preservation of Internet publications. We have established selection guidelines and a small proof of concept archive has been set up. We have examined in depth the issues and requirements of translating this small experimental archive into a large, stable, scaleable one under the control of an archive management facility. Currently we are preparing a Business Specification document, with a view to putting the required technical infrastructure in place by the end of this year.
We are also developing a national model. Working in the environment of a federal system of government, where both the National and State libraries have preservation responsibilities, we consider it essential to collaborate to ensure the broadest possible coverage and cost effective use of taxpayers resources.
The issues relating to the long-term preservation of Internet publications are many and complex. Many of them are closely related, for instance, technical issues and costs. My observations come mainly from the practical experience of working on the PANDORA Project. They and are confirmed and extended by reports of the work of others, including National Library of the Netherlands, the British Library, and The Task Force on Archiving of Digital Information.
I will not go into details about the PANDORA Project here. Further information is available from the National Librarys server at http://www.nla.gov.au/pandora/
To set the context, in Australia to date, there are very few commercial publishers publishing on the Internet. Traditional print publishers are showing no signs of moving onto the Net, although some are starting to use the Internet to update CD-ROM publications. We do not have the equivalent of the large firms such as Elsevier and Kluwer bringing out electronic versions of their print publications. Most online publications in Australia are still freely available on the Internet.
This paper elaborates on what I see as the principle issues facing preservation libraries in their efforts to ensure long term access to online publications.
- Longevity of Internet publications
- Volume
- Identification of those that should be preserved
- Responsibility for preservation
- Costs
- Technical issues
- Legal issues
- Permanent naming
- Authentication
- Metadata
Longevity
Documents published on the Internet are volatile in nature, being subject to rapid change and unpredictable removal. This means that they are highly susceptible to loss from the permanent record. Documents on paper, parchment, or clay can remain legible for hundreds, even thousands, of years without any special equipment or preservation treatment. This is not the case with electronic documents. They cannot be read without the hardware and software and knowledge of the technical environment in which they were created.
Having acquired a print item and accessioned it, a library can place it on a shelf and, given reasonable collection management practices, have a good chance that it will still be there in 100 years time. Should a library fail to acquire a publication while it is still in print, there is always the reasonable chance of acquiring a copy through the second hand market. Theres no such second chance with online publications.
In contrast, the long term preservation of Internet publications requires timely and active intervention at all stages. If we have managed to capture it before it has disappeared from the Net, decision making and specific preservation measures will be required at time of acquisition and on an ongoing basis, every few years, as technology changes.
In order to ensure long term access to Internet publications we need to identify items to be archived in a timely manner, so that they can be captured before they disappear.
We also need preservation techniques including refreshing, migration, emulation and possibly (although this is not a favoured option) hardware and software museums. None of these techniques are fully developed.
Volume
One of the most daunting aspects of online publications is the huge quantity of them and the rapidity with which many of them grow and change. At the National Library, the decision to be selective was based on two factors: the large amount of material of low research value; and the labour intensive nature of managing online publications with the limited staff resources available. (This second factor will be mentioned again under a later issue, Costs.)
Others, the National Library of Sweden, for instance, are attempting to be comprehensive.
Regardless of the approach taken, there are challenges. In deciding to be selective, the challenge is to establish selection guidelines that will lead to the development of a collection that contains all of the materials that researchers of the future are likely to want to see. As well, we need to be representative of the types of publications that appeared on the Net at this stage of our history.
To manage the large volume of Australian Internet publications that we would want future generations to be able to access, the National Library considers that it is necessary to
collaborate with other collecting institutions, the State Libraries, for example, to share the workload and avoid duplication. Co-operative collection development policies should be established. The National Library has already begun to collaborate with other national libraries involved in this type of work to share information, avoid duplication and to cut development costs.
Identification
How do we know what is out there to be preserved? Librarians engaged in selecting online publications largely lack the selection tools that we have been accustomed to with print. Although there are some useful sites that report the latest publications, in general, search engines are inadequate, publishers are inexperienced in the roles they might play in assisting with preservation, legal deposit is lacking. For the PANDORA Project, identification means hours and hours of searching the Internet each month, a very labour intensive process.
Once again, collaboration to share effort is important for broad coverage of the national output. Widespread use of standard metadata coupled with improved search engines that would search on this metadata would assist the process of identification of titles. Greater awareness by publishers of the need to archive, to use metadata and to report titles to preservation agencies is also needed. The extension of legal deposit provisions to online publications is expected to aid with this aspect. Permanent naming which will be mentioned in greater detail later on is crucial to identification.
Responsibility
There has been some debate about who has responsibility for preserving online publications. Is it the creator/publisher, an associated institution such as a university, traditional collecting institutions such as archives or libraries, or another body entirely? All of the players in the chain of information creation, distribution, storage and provision have a role. The way in which the creator/publisher sets up the title, the software used, whether standards are adhered to, will affect the ease with which it can be preserved (or whether it can be done at all). Associated institutions have a role in setting standards and for providing at least short to medium term support for their own publications. At the National Library we argue that ultimate long-term responsibility must be taken by those collecting institutions which have the preservation of documentary heritage as their primary role.
Co-operation with all stakeholders is essential. Formal agreements with other collecting institutions willing to take a role may be negotiated.
Costs
In our experience, the acquisition and management of an online publication is much more expensive than for a print item. With the technology currently available to us, it takes one person at least one full working day to do all of the tasks associated with acquiring an online publication: identification, selection, negotiation with publishers, cataloguing, loading, post-load checking, etc. We estimate that it is about five times more expensive to acquire an electronic item than a print item.
That does not include preservation. As already mentioned under longevity, electronic publications will need ongoing, frequent preservation intervention in the form of migration to new technology platforms. There is little experience of this so far. Until we are in the situation of having to migrate from the current technology to the next, we shall not know the full ramifications or the full extent of the costs.
For us, as for all libraries involved in this work, this is new business. The number of print publications we must add to our collections is showing no sign of declining. The management of electronic publications is additional to what we already do. We need extra staff with special skills, equipment, hardware and software. This all means additional costs for libraries.
Apart from the cost of the development work we are now engaged in, there is also the long-term cost of the ongoing storage and management of a large, growing archive.
We require recognition that the preservation and management of electronic publications is new business for libraries and sufficient ongoing funding for development and maintenance of secure stable archives. Collaboration among collecting agencies to share costs of development and avoid duplication will help to make effective use of available resources.
Technical issues
As already mentioned, many of the technical solutions that libraries require to achieve long term access to online publications are still under development. We are still in experimental mode.
Gathering software is primitive and intolerant. There are some publications that the PANDORA Project has not been able to archive using this pull technology, for example, publications structured as databases and constructed for the user on the fly. Alternative means are being explored. We have begun a trial with a few publishers whereby they will push their publications to us, by ftp, disk or CD.
Publishers continue to use new formats and software. Gathering software must be continually reprogrammed to keep up.
We need effective, tolerant, intelligent, gathering software, a sophisticated archive management facility, interfaces between the archive management facility and the catalogue to reduce the amount of duplicated metadata input, rights management software, and authentication software
Collaboration with other libraries doing similar work and partnerships with research institutions and suppliers would facilitate technical developments.
Legal issues
The legal issues relating to Internet publications are intricate and sensitive. Technically, the capacity for libraries to make information available to their users regardless of location is now almost unlimited. Ironically, however, our right to do so may now be even more limited than has been the case in print. This is frustrating but it would be counterproductive to undermine the commercial interest of publishers who are an essential component of the information industry.
Copyright is complex issue in the online environment, particularly where multi media sites are concerned. There are often many more creators involved with an online publication than is the case with print: authors of text, photographers, graphic artists, software developers, and many more. Even with fairly standard e journals, it is not uncommon for copyright to be retained by the creators of individual articles, rather than being held by the publisher. This means that there are potentially many more copyright holders that the Library may need to deal with than is the case with print.
Most countries have found that their legislative provisions relating to legal deposit and copyright are out of date in the electronic environment. In Australia, the Copyright Act, which also governs legal deposit, is in the process of review. Libraries in Australia have been lobbying for the right to retain the concepts of fair dealing in the use of electronic information and for legal deposit to be extended to electronic publications.
In the absence of legal deposit, the National Library has been employing alternative means to build the National collection. Voluntary deposit arrangements are being entered into with publishers and a Deed is being developed. This is an agreement that can be signed by the publisher and the Library to govern the use to which the Library may put the publication.
Permanent naming
As any Internet user appreciates, the current system of naming Internet documents, for location purposes, is inadequate. Development of permanent naming techniques is therefore of great interest. PURLs (Permanent URLs) are the only viable means at this stage, although not a perfect solution. DOIs (Digital Object Identifiers) URNs and Handles are other proposed solutions but are not yet fully developed. They require further development and international infrastructure for their operation as well as browsers that recognise them.
As an interim measure, the National Library has implemented a PURL Resolver Service based on OCLC software.
For progress on this issue, the development of and cross-sectoral international support and infrastructure for a permanent naming system are required. Development of browsers that can exploit permanent naming systems are also necessary.
Authentication
Since the aim is to provide a record of a nations documentary output, it goes without saying that it must be an accurate record. The documents that are archived must remain unchanged and true to the original.
Encryption, hashing, time stamping, watermarking and digital signatures are potential solutions.
A lot more work is required before the need of preservation libraries for widely accepted and supported techniques for authenticating Internet documents is met.
Metadata: Description, Access and taking the archive into the future
It is not sufficient just to create an archive of publications. Users must be able to discover what is in it and to gain access to titles. Archive managers must be able to maintain the titles in the archive, to add new issues or versions of a title, to store and access software needed to operate them, to compress them, decompress them, to manage access rights and conditions, to migrate them from proprietary or superseded formats and from obsolescent technologies. We must be able to send them forward to future generations with a complete record of their origin, status and treatment.
While a lot of work has been done on metadata, the emphasis has been on resource discovery rather than preservation and administrative metadata. Widely adopted metadata that conforms to set standards for purposes including both resource discovery and for recording preservation information is needed. The different kinds of metadata must be able to inter-relate with each other and to be interoperable across systems
To summarise, while some progress has been made, there is a tremendous amount of work still to be done before we can be confident that we can ensure long-term access to Internet publications. The issues to be resolved are well beyond the control of a single institution. The work requires cooperation between libraries and other collecting institutions, as well as with other stakeholders such as publishers, creators, users, researchers, software and product developers and suppliers, not only in Australia, but overseas too. It involves working in new ways with new partners.
