PANDORA - Boxing for survival:
Archiving, preservation and access issues related to Australian Internet based publications
Introduction
In Greek mythology, when all the evils were released from Pandora’s box, hope was said to remain. The National Library of Australia has implemented its PANDORA (Preserving and Accessing Networked Documentary Resources of Australia) project to preserve and provide long term access to Australian publications being produced in online format on the Internet. Pandora’s box - in the form of a dedicated computer based archive - will contain those publications. It will maintain them and migrate them to new technologies as needed. It will provide access to them now and in the future through the Library’s own online services to users in Australia and throughout the world.
The Internet and the World Wide Web
The Internet was thought up in the 1960s during the Cold War to ensure long distance communication in the event of a holocaust. Around thirty years later, it was enhanced significantly by the introduction of World Wide Web technology. This was developed by particle physics researchers to communicate esoteric and, to most other people, limited information. The Internet has become the premier method of communication for many people around the world. It breaks down geographic and time barriers in a way that no former method of communication could do.
While the Internet was based in access mechanisms such as ‘file transfer process’, its use was relatively restricted to research and academic communities. In 1992, there were fewer than 50 servers providing access to the Internet worldwide.
In 1994, when the number of servers was still only 2,500, access to the Internet by World Wide Web browser technology outstripped the less user friendly gopher access. The advent of browser mechanisms such as Netscape and Internet Explorer has extended access to the Internet to a much wider client group, who may have less developed computer skills. Web browsers use methods which are relatively easy and convenient. This is partly reflected in the explosion of servers. By 1997, there were over 650,000 servers worldwide. Although they are concentrated in major industrialised countries, some access to the Internet is possible in most areas of the world, even if the telecommunication infrastructure has not always kept pace with the Internet.
Publishing on the Internet
The Internet has proved to be a very attractive medium for publishing information. It has certain advantages over print media, which may in part have accounted for the exceedingly rapid growth of Internet publishing. Some advantages include:
- the ability of hypertext links to allow speedy jumps between different parts of the same publication or even into entirely different publications
- the inclusion of multimedia - sound, images, video - as well as plain text in the one publication
- the immediacy of Internet publishing and its ready availability as a publishing medium to anyone with access to an Internet service provider
- the ease of accessibility - Internet publications are not housed in a fixed location or subject to fixed opening hours - they can be accessed wherever and whenever there is an online computer connection.
At the Ausweb97 conference in July, Dan Shafer, a commercial publisher of CNET in the USA defined Web publishing as ‘remote and periodic distribution of information by those who possess it to those who desire it’.
One of the major problems identified for ‘those who desire it’ is ensuring that there will always be a copy of information available, especially when those who initially possess it relinquish their control over it. This is an essential feature of the PANDORA project. It establishes an archive that will capture and ‘possess’ identified material which currently exists only on the Internet, then maintain access to it into the future.
Some Australian Publishing Initiatives
Australians have been enthusiastic adopters of Internet technology, and have been involved with the Internet from its very early days. In 1990, the Australian Vice Chancellors Committee (AVCC) established the Australian Academic and Research Network (AARNET) which put Internet access into most universities and other research establishments throughout Australia. Some Australian universities have had, and continue to maintain, very active Internet publishing sites.
The National Library of Australia in 1995 established a Web based information server which provided a host service for Commonwealth Government departments, as well as mounting Library information online. In early 1996, a listing of online Australian electronic journals was commenced. This list now has over 1000 entries which provide hypertext links to Australian serial publications with a significant Web presence. They can be accessed by title or by subject. A subsidiary list of publications, chiefly promotional sites, is also maintained.
The AVCC continues to play a major role in maintaining Australia’s Internet presence. Through the Electronic Publishing Working Group (EPWG) of the Australian Vice Chancellors Committee Standing Committee on Information Resources (AVCC SCIR) funds are made available for innovative and investigative Internet projects.
Their aim has been to develop models of best practice which might provide guidance for the Australian academic community on the possibilities - and pitfalls - of advanced information technologies. With academic funding and support, it is not too surprising that some of the earliest materials published on the Web in Australia were scholarly journals and other research based materials.
Projects funded by the AVCC have included:
- the commissioning of new and experimental serial titles, such as Psyche - an interdisciplinary journal of research on consciousness and Australian Humanities Review;
- experiments in the conversion of print based serials to electronic format, such as Australian Journal of Management;
- investigations into the use of electronic pre-publication and open peer review of scholarly research articles, as in the electronic Medical Journal of Australia (eMJA)project;
- general access related issues - as, for example in Key issues in Australian electronic publishing, collected reports 1995-1996 (http://www.adfa.oz.au/EPub/key/Opening.html), which covers library and archival issues, legal issues, technical standards, and a description of some electronic projects.
The PANDORA project is yet another recipient of AVCC funds. A sum of $100 000 has been made available for the project, with matching support coming from the National Library of Australia.
An Australian Electronic Archive
PANDORA will be more than just an archive of scholarly journals. As the initial grant application stated:
the overall goal of this project is to develop and implement procedures for the capture, archiving and provision of long term access to online electronic Australian publications selected for national preservation. It will cover the full range of materials published online in Australia - including serials, newspapers and books, scholarly papers and theses, as well as unique online formats like ‘homepages’.
The PANDORA project hopes to provide access to future generations to an archive that represents the state of Australian online publishing from its earliest day - the incunabula period of online information - up to its most current manifestation.
At this stage, there is a possible backlog of around 5 years of online publishing. Due to the vagaries of electronic publishing, and the sometime unforeseen difficulties in mounting and maintaining a web site, some of that material is no longer available. Some of it will have changed its ‘look and feel’ several times, as online publishers keep their products at ‘the cutting edge’ of Web design. In particular, most of the original ASCII based text used in gopher systems has been converted to more modern formats. The plain black and white text, with minimal layout features of early Internet publishing, has almost disappeared.
There have been some efforts worldwide to consider the issue of archiving the Internet for future generations - some of the most notable and ambitious being that of Brewster Kahle in the USA to create an Internet archive and the Dejanews project to maintain a database of all messages posted to Usenet News since 1995. The National Library of Australia’s efforts are more modest than those projects.
The Library believes that, as a national library, it has an obligation to preserve Australia’s published output of information. It does so for print media through the mechanism of legal deposit, where a copy of any item published in Australia should be lodged with it for preservation in perpetuity. At this stage, legal deposit does not apply in Australia to electronic materials. For both tangible electronic objects which exist in a real, physical form like CDROM, and online electronic information which has no physical form, alternative methods of ensuring preservation and hence long term access have had to be put in place.
The Library has in place a voluntary deposit agreement for tangible electronic objects, and has been accepting materials under this arrangement since 1994. There are now around 600 tangible items in the Library’s collections, and a process has recently been put in place to ensure that long term access to these materials will be ensured. For these materials, there are two major problems. First, the physical format itself is expected to have a relatively short lifetime. Second, and more critical is the fact that tangible electronic objects are heavily reliant on software for their interpretation. This software can be unique, and/or machine dependent, and may be embedded within the object itself and invisible to the user. Issues associated with tangible objects are being addressed by a subgroup of the PANDORA team.
Some Difficulties for PANDORA
One of the major difficulties during the early development of the PANDORA project has been that Web publications are not the same as printed publications. Some of the characteristics of Internet publishing that creates problems for librarians wanting to capture and archive them include:
- there are no standardised ways of finding information about new publications.
- at the current time in Australia they are not covered by legal deposit legislation
- bibliographic standards for description are still evolving, and the lack of physical format means that it can be quite time-consuming to analyse their content
- online publications are ‘virtual’ - they can disappear completely if their owner ceases to maintain his site, or for some other reason they are deleted. Alternatively, if the publisher’s address changes, hypertext links to the previous site cease to be active - ‘link rot’ means that parts of the publication can just get lost
- the concept of the integrity of a single print ‘edition’ or serial ‘issue’ disappears in online publishing, so that the information in the current online version may be different to what it was a week or even a day ago.
The hardest thing, at the moment, is ensuring that the version of any publication captured into the archive faithfully represents the online edition. This has involved a careful analysis of each publication selected for archiving in order to understand both its publishing schedule and the way the information is arranged on the site. All publications currently being archived have been assessed and a ‘gathering schedule’ determined. This determines how frequently the online publication will be captured and a copy transferred to the archive. The titles, URL’s and gathering schedules for publications in the archive in July 1997 is given in the Appendix.
Some Limitations on the ArchiveRCHIVE
The main effort for PANDORA at the moment is to establish an archive of online electronic Australian publications. In order to keep the project within manageable bounds, some restrictions on that broad phrase ‘online electronic Australian publications’ has been necessary. The overall guidelines for selection for the archive are available online. Some of the more important points are:
- ‘publications’: the Library is taking the view that material which is mounted online and available either free or by subscription is published.
- ‘Australian’ - although there are well developed rules for selection of Australian materials in the print media, the rules have had to be amended for online materials. It is the nature of the Web that national barriers break down. It is just as easy, and sometimes more efficient to mount or ‘publish’ an ‘Australian’ publication on a US server, than to maintain it on an Australian server. This is particularly so if the publication has a large overseas readership. The publication Yoni initially published out of northern NSW has moved to a US server, and is now only mirrored on an Australian site. Speed of access was one reason for the move. Yoni will continue to be considered ‘Australian’ however. The alternative situation also can apply - there have been one or two titles rejected for the PANDORA, because, although they were mounted on an Australian based server, it appeared to be more for convenience rather than any real ‘Australianness’ in their content.
- ‘online’ - not all Australian online publications will be included in the PANDORA archive. Although discussion lists and newsgroups are available online, at this stage they have not been considered for archiving. It is agreed that there are some significant lists which could be selected for long term access, but resources do not permit their inclusion in the project at this stage. Many of the more scholarly and serious lists are currently maintaining their own archive. Another major category of material currently not in contention for NLA archiving is what we are calling ‘mega sites’ - sites like the Australian Broadcasting Corporation, the Coombs site at ANU, the AustLII site. For some sites, such as Australian National Botanic Gardens site, individual publications might be selected for archiving - such as In flower this week, while other information which is basically internal to the organisation will not be selected. The third major category of online information excluded is that which is also available in print format, if the online version does not have any significant additional features. This decision was made because it is, and is likely to remain so for some years to come, easier to preserve a paperbased object that to preserve an online one.
What is in PANDORA’s Box?
The archive has moved beyond PANDORA Mark 0, which was mostly experimental, into its first production phase, PANDORA Mark 1. As the overall logical structure of the archive is clarified and the amount of material archived grows, further refinements will be inevitable.
At the time of writing of this paper, there are nearly 30 publications in the archive. They are accessible through the Library’s online public access catalogue via its Z39.50 link to the server. There are another 15 or so publications awaiting capture.
The split of ‘serials’ to ‘monographs’ is about 2:1, when traditional library cataloguing terminology is applied. In reality, it is not so easy to apply the terms ‘serial’ and monograph’ to online publications as their publishing schedule and ease of making changes mean that content can change quite frequently and in a very nonprint-like way. The terms ‘cumulating’ or evolving’ can be used to describe these processes.
Electronic ‘Serials’ in the Archive
Regular serials
Out of eighteen ‘serials’ listed in the Appendix, only six can be considered to fully emulate the print format - with individual issues of fixed content, issued sequentially, with a regular (or more or less regular) publishing schedule. They are Case manager, Compute-Ed, eJIST, E-Law, Grilled pterodactyl and Ibn Qirtaiba. These are the easiest to archive, and a regular and reliable gathering schedule is possible for them.
E-Law, for example, which dates from 1993, is published in a relatively simple serial format. Initially it was available by a variety of methods, which now seem quite primitive, like email and ftp. Since 1996 it has been available via the Web. Its earliest issues, initially in ASCII and available by email or gopher have also been converted to HTML format and mounted on the Web.
Most of the publications selected for archiving still maintain their own online archive, so readers still have access to all issues from the publisher’s site. As the number of issues online increases, it is likely that publishers will delete earlier issues. For instance, for the publication Grilled pterodactyl, there are now only issues online from June 1997, although the title itself commenced in 1995. The only way to access those early issues now is from the PANDORA archive.
A further five have the regular publishing schedule of printed serials, but have additional characteristics which make their archiving more complex. These are publications like Australian Humanities Review which is taking advantage of the Web characteristics to create a publication in a format different than would be possible in print. Although each issue is numbered and published to a schedule, the content of each issue grows over the period of its life and includes ongoing reader feedback etc.
The Australian Electronic Journal of Nursing Education also adds to the latest online issue until it is complete. The Atheist Society of Australia basically operates a regular serial, but adds in other non numbered articles as well. Spin doctor is also basically a regular serial, but it includes an ongoing message board facility. These serials are also relatively easy to archive, once their publishing sequences are understood. In Music and Media is another regular serial, but it rearranges its archived material on a regular basis. In order to retain the layout of the archive, it needs to be captured very frequently.
‘Cumulating’ serials
A different variety of ‘serial’ publication is one which has no obvious individual issues, but appears to accumulate content on an ongoing basis. For convenience, these have been called ‘cumulating’ serials. Sometimes the oldest information is pushed off the site by newer information, as the publisher rationalizes online storage. This complicates the archiving process considerably. There are five publications which fall into this category - Electronic journal of Australian and new Zealand History, Internet Journal of Health Promotion, Mindgate, Write Stuff and Yoni. They need analysing in rather more depth to ensure that no information is lost or overwritten during the archiving process.
Other ‘serials’
There are two other publications which have been classified as serials in the archive, Coastal bibliography series and Electronic Frontiers Australia, They do not closely follow serial cataloguing rules. The first of these is a series of six bibliographies on coastal themes. Each bibliography is likely to change with time and additional bibliographies could be added. It will be gathered relatively infrequently.
The second publication is more of a composite administrative and publishing site. It includes a regular newsletter of Electronic Frontiers Australia, and also minutes of meetings and policy documents. In particular, it is a publication which has no simple equivalent in the print media.
Electronic ‘Monographs’ in the Archive
There are eight other publications in the archive at the time of writing this paper, one of which is conference proceedings. None of the other seven fit the mould of print monographs. They are all ‘evolving’ or changing with time, and will be captured, generally every 6 months, to reflect the state of the publication as a series of periodic snapshots. Flight of Ducks, which is undergoing the most change, is being captured on a monthly basis. In the twelve months or so that it has been monitored, it has increased from around 100 electronic ‘pages’ to nearly 800.
Electronic Ephemera
There is some material already identified for archiving which has been classified as ‘ephemera’. This is material which often captures an event or activity, but which does not have any ongoing substance as a publication. Ephemera will be treated in the archive in a similar manner to print ephemera and be available under subject access. As yet, the ephemera component of the archive has not been set up.
Lost and Missing Publications
Monitoring and selection of publications for the archive commenced early in 1996. In the past year, three publications of the initial twenty selected for archiving have disappeared from the Web. An emergency rescue operation was undertaken for one of these, when it was noticed that the online site gave notice it would close in a few days. The other two have vanished without trace, and without informing the Library, even though the publisher had agreed to the archiving of the site. This represents a loss of around 15%. This percent may be higher than usual due to instability in early Australian Web publishing, and it is expected that overall the loss will be less. However, no matter what level the loss rate reaches, a fully operational PANDORA archive will redress this loss by ensuring timely capture of all publications selected for archiving.
Besides total disappearance from the Web, some publications move location and change their URL. Four out of the twenty-seven or 15% of the publications listed in the Appendix have changed location in the past twelve months. PANDORA is hoping to take part in a project that introduces a system of permanent naming for Australian online publications. This will reduce the number of locations changes that need to be tracked.
Conclusion - Future Directions
The amount being published online continues to grow. At this stage, the rate appears to have slowed down slightly, although the quality is being maintained or even improving. As the archive stabilises and becomes a routine activity within the National Library of Australia, the number of titles added to it should increase substantially. already, there are another one hundred or so publications which are being processed prior to archiving.
In total, however, only a small number of Australian publications have been selected for national preservation by the Library - at around 10% of the total number of publications surveyed. If a full representation of Australian online publications is to be established, then cooperation from other sections of the library community is necessary.
The Library is currently considering a national model for the preservation of online information. The volume of material produced is such that it is unlikely that any one institution will ever be able to preserve everything. This may see the major players such as universities, state libraries and national research organisations taking responsibility for their own sites. Some Australian universities are already looking at creating a distributed archive of research theses held on line, modeled on the approach by Virginia Tech and UMI in the USA. The Library is also talking to state libraries, concerning publications of state based information - state government publications, local council publications and the like.
One thing is sure, that if no active steps are taken by Australian libraries to capture and continue to provide access to on-line publications, then Australia’s full cultural heritage of recorded information will be severely compromised in the future.
Author
Wendy Smith has been Manager of the National Library of Australia’s PANDORA project since 1996 when she returned to the Library after 3 years secondment lecturing in paper conservation and library and archive preservation at the University of Canberra. Prior to that, she was Director of the Library’s Preservation Services Branch where she was responsible for preservation policy and management of the Library's preservation program.
Appendix
Publications in the PANDORA archive, July 1997
See the updated site for the most recent listing.
‘Serials’ (listed on the National Library of Australia’s electronic journal list)
TITLE |
URL |
GATHERING SCHEDULE |
Atheist Society of Australia |
http://www.ozemail.com.au/~ksolway/asa.html |
annual |
Australian Electronic Journal of Nursing Education |
http://www.csu.edu.au/faculty/health/nurshealth/ |
6 monthly |
Australian Humanities Review |
http://www.lib.latrobe.edu.au/AHR/ |
3 monthly |
Case Manager |
http://www.esra.gov.au/tcm.htm |
3 monthly |
Coastal Bibliography Series |
http://www.erin.gov.au/portfolio/ahc/hera/ |
6 monthly |
Compute-Ed |
http://www.edfac.uts.edu.au/projects/comped/ |
annual |
e JIST |
http://www.usq.edu.au/electpub/e-jist/homepage.htm |
6 monthly |
E Law |
http://www.murdoch.edu.au/elaw |
6 monthly |
Electronic Frontiers Australia |
http://www.efa.org.au/ |
3 monthly |
Electronic Journal of Australian and New Zealand History |
http://www.jcu.edu.au:80/aff/history/ |
6 monthly |
Grilled Pterodactyl |
http://www.ozemail.aust.com/~drgrigg/ptero.html |
annual |
Ibn Qirtaiba |
http://sf.sig.au.mensa.org/ + mirror in US |
monthly |
IN MUSIC & MEDIA |
http://www.immedia.com.au/im_m/index.html |
weekly |
Internet Journal of Health Promotion |
http://www.monash.edu.au/health/IJHP/index.htm |
6 monthly |
MindGate |
http:www.melbourne.net/mindgate/ |
6 monthly |
SPIN Doctor |
http://sunsite.anu.edu.au/region/spin/SPINDOC/ |
6 monthly |
Write Stuff |
http://www.utas.edu.au/docs/ahugo/tws/ |
6 monthly |
Yoni |
http: //www.ion.com.au/yoni/ main site now http://www.yOni.com/ in US |
6 monthly |
‘Monographs’ - Titles not on the electronic journal list
TITLE |
URL |
GATHERING SCHEDULE |
Ausflag |
http://www.ausflag.com.au/ |
3 monthly |
Bougainville Freedom Movement |
http://www.magna.com.au/~sashab/BFM.htm |
6 monthly |
Circit |
http://teloz.latrobe.edu.au/circit/conf0995.html |
single gather |
Council for Aboriginal Reconciliation |
http://www.austlii.edu.au/car/ |
6 monthly |
Cyberpoetry (was DiMoCoPo) |
http://student.uq.edu.au/~s271502/ |
6 monthly |
Firebreak |
http://msowww.anu.edu.au/~barling/firebreak.html |
3 monthly |
Flight of Ducks |
http://www.cinemedia.net/FOD/FOD0001.html |
monthly |
Jazclass with Michael Furstner |
http://www.ozemail.com.au/~jazclass |
6 monthly |
Uluru |
http://www.orbitex.ch/uluru |
6 monthly |