The Work of Pandora
Past and present
staff celebrating the 10th aniversary of PANDORA. Margaret Phillips
(Centre) the former Director of Digital Archiving and leader of the
team that established PANDORA in 1996.
PANDORA:
In 1995, it became clear that a significant amount of Australian information
was being made available online only via the Internet. The National Library,
as part of its mission to ensure that Australians have access to a comprehensive
record of Australian history and creative endeavour, soon recognised this
and sought other ways to make this material accessible to its users.
Initially many libraries thought that cataloguing items on the Internet
might be enough to guarantee access for Library users. However, the National
Library soon realised that many online publishers could not guarantee
the long-term availability of their own publications, particularly given
the unreliability of many Internet Service Providers (ISPs) at that time.
Therefore, it was decided that for assured long-term access it was important
for the Library to actually hold copies of these publications, as it did
other more traditional formats.
Setting up the Electronic Unit on
Being librarians, the first thing the Unit did was work out how to catalogue
websites. It then set about working on selection guidelines, setting out
what was wanted from the Internet within this as-yet-unborn digital archive.
At the end of the six-month trial, given the exponential growth of the
Internet, the Electronic Unit was retained and an archiving approach accepted.
In November 1996, the digital archive was named PANDORA, an acronym standing
for Preserving and Accessing Networked Documentary Resources of Australia.
The long process of finding technical solutions to fill and manage the
archive thus began with the Australian spirit of learning by doing.
Initially the Library’s IT department gathered files for the Electronic
Unit. However, this proved to be an unsatisfactory arrangement, and so
solutions to allow Electronic Unit staff to archive and manage gathered
files were sought. Progress was slow. Although many hundreds of titles
were selected as worthy of archiving, only 30 titles were actually archived
by the end of 1997, and only 90 by the end of 1998.
At first many websites could not be gathered, due to the early inabilities
of web gathering software. Websites in basic HTML were able to be gathered,
but sites using Frames and JavaScript initially posed many problems. Once
those problems were surmounted the Unit had to contend with applets, sound
files, cascading style sheets, Shockwave Flash and a myriad of different
file and style formats. All these Internet developments needed to be studied
and experimented with before sites were able to be archived. New skills
needed to be learnt quickly, new workarounds invented, and new web crawlers
or gatherers found that could gather and parse all the different file
types.
The PANDORA staff and management sought partnerships with other collecting
bodies to assist with the enormous task of archiving the Web. The first
partner, the State Library of Victoria, joined in 1998, and was soon followed
by the National Film and Sound Archive. By 2003 all Australian state libraries
(except for Tasmania which has its own archive), the Northern Territory
Library, the Australian War Memorial and the Australian Institute of Aboriginal
and Torres Strait Islander Studies have joined PANDORA and are now successfully
adding to PANDORA’s collection—selecting, cataloguing and archiving publications
in their areas of interest.
Initially the early partners were hindered from participating fully in
this enterprise through a lack of independence that was caused by an inability
to directly access the archive, as staff at the Library were needed to
physically enter the partners’ gathers into the collection. To promote
greater uptake of the opportunities that PANDORA gives to its partners,
and also for efficient gathering in the National Library, a fully distributed
archive system was needed.
As no such system was available on the market, the Library employed two
Australian National University students, Steven McPhillips and Mark Fritz,
to build a comprehensive archive management system. That system was named
PANDAS (PANdora Digital Archiving System), and went live in June 2001.
This development led to great improvements in productivity and access
for the Library and its partners. The system has been so successful that
versions of PANDAS are now being used by other libraries including the
British Library.
The Electronic Unit was renamed the Digital Archiving Section in 2003
to better reflect its role. Both units have never exceeded six permanent
staff who have been drawn from the Library’s general staff.
By June 2006, the publicly accessible PANDORA archive contained nearly
12 000 titles. Some of these titles are monographs which have been gathered
once, others are entire websites that have been archived regularly to
reflect their changes over time, and a number are serial publications
that have been gathered repeatedly (some weekly). These titles together
consist of over 31 million separate files comprising over a terabyte of
data.
Some of the highlights of the archive are the comprehensive 2000 Sydney
Olympics collection, and the collections created for all elections since
1996. The archive has a wealth of government online publications in addition
to websites created by individuals and organisations which are of cultural
or educational value. There are also interactive novels, blogs, podcasts
and, as improved web gatherers and new software have become available,
we have also recently been able to archive databases.
PANDORA is a selective archive. This means that every publication in the
archive has been selected for its long-term research value; had archiving
permission sought from the publisher; been quality checked and, if necessary,
fixed to make it viewable in the archive; and been catalogued as a separate
entity. Alternative types of archiving that other libraries and organisations
have adopted are whole of domain or whole of Internet archiving, whereby
very large gather snapshots of the Internet are taken. This method gathers
large amounts of data but is not always easily searchable or viewable.
The National Library has achieved great things by committing itself to
selective archiving but understands that whole of domain archiving is
now also a realistic and sensible complimentary activity. Therefore, in
2005, a gather of the whole of the .au domain was attempted in conjunction
with the Internet Archive, an organisation based in the United States.
The Library continues to find or develop new ways of preserving access
to digital content. It has come a long way but no doubt the next 10 years
will present even more changes and challenges for PANDORA and its staff
to overcome.
Those who have worked on PANDORA at the National Library include Margaret
Phillips, Leanne McKinnon, Anne Daniels, Sue Weppner, Paul Koerbin, Michael
Porter, Antje Kavanagh, Maxine Fisher, Diana King, Amy Bartlett, Fiona
Ash Relf, Julie Whiting, Edgar Crook, Jelena Vukovic, Rhonda Poultney,
Weidong Chen, Jennifer Howlett, Rowan Henderson, Julie Watson and Martina
van Veen.
Edgar Crook