National Library Home > News & Events
Gateways

The Work of Pandora

Photograph courtesy Ting ZhaoPast and present staff celebrating the 10th aniversary of PANDORA. Margaret Phillips (Centre) the former Director of Digital Archiving and leader of the team that established PANDORA in 1996.

PANDORA: Australia’s Web Archive has now been in existence for 10 years. In that relatively short time it has become one of the most talked about library innovations in the world, and certainly one of the most important Australian initiatives in Australian library history.

In 1995, it became clear that a significant amount of Australian information was being made available online only via the Internet. The National Library, as part of its mission to ensure that Australians have access to a comprehensive record of Australian history and creative endeavour, soon recognised this and sought other ways to make this material accessible to its users.

Initially many libraries thought that cataloguing items on the Internet might be enough to guarantee access for Library users. However, the National Library soon realised that many online publishers could not guarantee the long-term availability of their own publications, particularly given the unreliability of many Internet Service Providers (ISPs) at that time. Therefore, it was decided that for assured long-term access it was important for the Library to actually hold copies of these publications, as it did other more traditional formats.

Setting up the Electronic Unit on 1 April 1996, under the management of Margaret Phillips for an initial period of six months, was the Library’s first step. The project group comprised Leanne McKinnon, Anne Daniels and Sue Wepner, all traditional librarians from the Library’s Acquisitions Unit.

Being librarians, the first thing the Unit did was work out how to catalogue websites. It then set about working on selection guidelines, setting out what was wanted from the Internet within this as-yet-unborn digital archive.

At the end of the six-month trial, given the exponential growth of the Internet, the Electronic Unit was retained and an archiving approach accepted. In November 1996, the digital archive was named PANDORA, an acronym standing for Preserving and Accessing Networked Documentary Resources of Australia. The long process of finding technical solutions to fill and manage the archive thus began with the Australian spirit of learning by doing.

Initially the Library’s IT department gathered files for the Electronic Unit. However, this proved to be an unsatisfactory arrangement, and so solutions to allow Electronic Unit staff to archive and manage gathered files were sought. Progress was slow. Although many hundreds of titles were selected as worthy of archiving, only 30 titles were actually archived by the end of 1997, and only 90 by the end of 1998.

At first many websites could not be gathered, due to the early inabilities of web gathering software. Websites in basic HTML were able to be gathered, but sites using Frames and JavaScript initially posed many problems. Once those problems were surmounted the Unit had to contend with applets, sound files, cascading style sheets, Shockwave Flash and a myriad of different file and style formats. All these Internet developments needed to be studied and experimented with before sites were able to be archived. New skills needed to be learnt quickly, new workarounds invented, and new web crawlers or gatherers found that could gather and parse all the different file types.

The PANDORA staff and management sought partnerships with other collecting bodies to assist with the enormous task of archiving the Web. The first partner, the State Library of Victoria, joined in 1998, and was soon followed by the National Film and Sound Archive. By 2003 all Australian state libraries (except for Tasmania which has its own archive), the Northern Territory Library, the Australian War Memorial and the Australian Institute of Aboriginal and Torres Strait Islander Studies have joined PANDORA and are now successfully adding to PANDORA’s collection—selecting, cataloguing and archiving publications in their areas of interest.

Initially the early partners were hindered from participating fully in this enterprise through a lack of independence that was caused by an inability to directly access the archive, as staff at the Library were needed to physically enter the partners’ gathers into the collection. To promote greater uptake of the opportunities that PANDORA gives to its partners, and also for efficient gathering in the National Library, a fully distributed archive system was needed.

As no such system was available on the market, the Library employed two Australian National University students, Steven McPhillips and Mark Fritz, to build a comprehensive archive management system. That system was named PANDAS (PANdora Digital Archiving System), and went live in June 2001. This development led to great improvements in productivity and access for the Library and its partners. The system has been so successful that versions of PANDAS are now being used by other libraries including the British Library.

The Electronic Unit was renamed the Digital Archiving Section in 2003 to better reflect its role. Both units have never exceeded six permanent staff who have been drawn from the Library’s general staff.

By June 2006, the publicly accessible PANDORA archive contained nearly 12 000 titles. Some of these titles are monographs which have been gathered once, others are entire websites that have been archived regularly to reflect their changes over time, and a number are serial publications that have been gathered repeatedly (some weekly). These titles together consist of over 31 million separate files comprising over a terabyte of data.

Some of the highlights of the archive are the comprehensive 2000 Sydney Olympics collection, and the collections created for all elections since 1996. The archive has a wealth of government online publications in addition to websites created by individuals and organisations which are of cultural or educational value. There are also interactive novels, blogs, podcasts and, as improved web gatherers and new software have become available, we have also recently been able to archive databases.

PANDORA is a selective archive. This means that every publication in the archive has been selected for its long-term research value; had archiving permission sought from the publisher; been quality checked and, if necessary, fixed to make it viewable in the archive; and been catalogued as a separate entity. Alternative types of archiving that other libraries and organisations have adopted are whole of domain or whole of Internet archiving, whereby very large gather snapshots of the Internet are taken. This method gathers large amounts of data but is not always easily searchable or viewable. The National Library has achieved great things by committing itself to selective archiving but understands that whole of domain archiving is now also a realistic and sensible complimentary activity. Therefore, in 2005, a gather of the whole of the .au domain was attempted in conjunction with the Internet Archive, an organisation based in the United States.

The Library continues to find or develop new ways of preserving access to digital content. It has come a long way but no doubt the next 10 years will present even more changes and challenges for PANDORA and its staff to overcome.

Those who have worked on PANDORA at the National Library include Margaret Phillips, Leanne McKinnon, Anne Daniels, Sue Weppner, Paul Koerbin, Michael Porter, Antje Kavanagh, Maxine Fisher, Diana King, Amy Bartlett, Fiona Ash Relf, Julie Whiting, Edgar Crook, Jelena Vukovic, Rhonda Poultney, Weidong Chen, Jennifer Howlett, Rowan Henderson, Julie Watson and Martina van Veen.

Edgar Crook