National Library Home > News & Events
Gateways

A tale of two PANDORAs

Have you listened to Pandora Internet Radio yet? It’s located at: http://www.pandora.com/

Pandora Internet Radio is a US-based online music discovery service which allows individual users to indicate the kinds of music they want to hear, and helps them shape what they hear by providing feedback. (As yet, PIR is not fully rolled out to listeners in Australia, but you can visit the site to see how it works, and listen to sample tracks.)

What makes Pandora Internet Radio work is the combination of a huge database, not just of songs but of classified qualities of songs, and the input of users—all enabled by good technology and some adventurous thinking. PIR is a great example of a Web 2.0 application—using the knowledge, interest (and yes, sometimes prejudice) of a community of users to build information resources of great quality and interest, and to tailor services. (Other popular examples of Web 2.0 applications include Wikipedia, Flickr, SecondLife, and LibraryThing.)

Readers of Gateways know there is another PANDORA, which is closer to the heart of the National Library. This is PANDORA, Australia’s Web Archive, located at http://pandora.nla.gov.au/.

This PANDORA was set up as a freely available service by the National Library of Australia in the mid-1990s, to partially address its collecting and preservation mandate in an increasingly digital environment. (A story in Gateways 82, August 2006, commemorated the 10th anniversary of this PANDORA, covering much of the early history and current approaches of our pioneering web-archiving initiative.)

More than a decade of librarian-managed selective collecting by the National Library and its PANDORA partners has produced a very important and growing collection of Australian web resources, selected for their long-term research value, all carefully crafted to ensure functionality before being ingested into the archive, and cleared for ongoing public access long beyond their natural life on the web.

In the past two years, the National Library has also explored a radically different approach. In 2005 and 2006, the US-based Internet Archive was contracted to carry out very large-scale continuous crawls of the Australian web domain to produce two Whole Domain Harvests—much larger snapshots across the range of web activity in Australia. Each of these snapshots hauls in about 100 times what we can collect with our selective approach each year—but without any guarantee of functionality or, at this stage, mechanisms to provide public access to such a large body of material. Even these ‘whole domain’ harvests fall short of ‘wholeness’, because there is much that they cannot capture for technical or legal reasons. Because they are such large exercises, carried out over many weeks once a year, they also miss important information which the Library may want to capture more frequently.

Change

Eleven years ago, PANDORA (our PANDORA) was a response to change, and the National Library has begun a process that may lead it to new approaches to web archiving. What are the factors driving this change process?

One imperative is to evaluate the experience of our selective collecting and our larger domain harvests, to maximise the benefits we get from both approaches.

A second imperative is to look at other possible approaches. Since 2003, the National Library has been an active member of the International Internet Preservation Consortium (IIPC), a group made up from the Internet Archive and national libraries involved in web archiving, which has already developed a number of tools to make web archiving easier and more effective. IIPC members recently agreed to extend their program of shared development for another three years, but with a much wider membership of libraries, archives and other organisations who can contribute to and learn from further advances in web archiving. (The National Library of Australia is currently testing a new version of its PANDORA Archiving System (PANDAS), used to manage web-archiving workflows. One hope is that IIPC activities will eventually obviate the need for individual members such as the National Library to develop or maintain their own in-house software tools.)

A third imperative for change is to plan long-term preservation action for the tangled mass of file formats in our web archives. The National Library has agreed to lead an IIPC working group on this subject.

Fourthly, and perhaps the greatest driver for making change in the approach to internet preservation, is the changing nature of web information itself. While the National Library won’t be bringing the two Pandoras (Australia’s web archive, and the American online user-customised radio service) together—because their Pandora is beyond the geographical and genre scope of our PANDORA!—there are many, many other Web 2.0 applications with significant Australian content that are poorly preserved in either our PANDORA archive or our domain harvests. The nature of this material—ever-changing, building and going in new directions—presents serious conceptual and technical challenges to methods built around ‘freezing’ occasional snapshots of static states of information.

Changing needs and changing opportunities are often reflected in organisational changes. Within the National Library, a new branch has been created to bring our Web Archiving and Digital Preservation functions together. This new arrangement should help us to take a more integrated approach to managing the whole life-cycle of the born-digital materials which we need to collect and preserve.

The new branch is embarking on a wide-ranging review of web-archiving activities aimed at addressing these imperatives. The review, to be carried out in consultation with other PANDORA participants, will look at the need for change in many key areas, including:

  • what web material is collected
  • how it is collected (possibly including models for legal deposit of digital materials)
  • how it is described
  • how it is preserved
  • how it is delivered for access
  • workflows and systems
  • roles and responsibilities
  • dependencies and the implications of change.

If this sounds like a lot of issues to be considered, it is. The National Library decided in the early 1990s not to be daunted or debilitated by such issues. We are revisiting them from a position of strength, and with many more tools, and a community of support, that were not available when we started.

Watch this space (or, as the other Pandora might say: keep listening!)

Colin Webb
Director, Web Archiving and Digital Preservation Branch
National Library of Australia