PANDORA - Review of Progress to June 1997

Abstract

The National Library of Australia (NLA) is developing an electronic archive called PANDORA to provide long term access to significant Australian online publications. A dedicated online archive will be established in the first instance. Publications contained in it will be updated on an ongoing basis and converted to new formats as technology demands.

The processes of selection, preservation and access have undergone close scrutiny. This report discusses the results of that scrutiny and indicates future directions for the project

1. Introduction

1.1 PANDORA objectives

The PANDORA project has, as its main objective, to:

  • capture, archive and provide long term access to significant Australian online publications selected for national preservation.

It will do this by:

  • identifying and selecting online publications based on the SCOAP Guidelines for the Selection of Online Australian Publications Intended for Preservation by the National Library (online at http://www.nla.gov.au/1/scoap/scoapgui.html)
  • cataloguing selected publications onto the Australian Bibliographic Network (ABN)
  • arranging with the owners of identified publications to capture a copy of their publication for safe keeping in the National Library's electronic archive
  • making the information from the archive available to users in line with fair dealing provisions, while taking into account the publisher's commercial interests
  • preserving the "look-and-feel" of online electronic publications wherever appropriate
  • updating the information in the archive on an ongoing and systematic basis, while maintaining date stamped previous issues or versions
  • migrating information in the archive to new formats as current technology ages.

Other objectives, with wider implications beyond the PANDORA project itself, are to:

  • contribute to discussions on extending copyright and legal deposit to online publications
  • implement a system of permanent naming for Australian Internet publications to overcome the location problems associated with change of service provider
  • implement and publicise a system of describing documents based on the Dublin Core attributes, to make online searching for information more efficient
  • cooperate with other Australian libraries to ensure that there is no duplication of archived materials, and to establish a distributed national archive of electronic publications.

PANDORA does not sit in isolation from the rest of the Library's business. There are many other 'virtual library' developments within the Library, including collections digitisation projects, that all have a direct impact on the PANDORA project.

1.2 Online electronic publications

Significant resources are needed to establish and maintain an electronic archive.

Although all Australian online publications could potentially be included in the PANDORA archive, the amount of material and the special needs of electronic publications mean that the Library is only selectively choosing publications for the archive. A list of titles for preservation is now available and will be regularly updated.

1.2.1 Materials out of scope

Some major categories of material currently out of scope include:

  • material which also exists in print format, unless the online version has significant additional information or value
  • publications at the state level , where the National Library wishes to develop cooperative agreements whereby State libraries would agree to ensure long term access to online publications of the relevant State government, local governments within the State, State-based organisations and publications containing information that is specific to the State or a region of it
  • organisational and personal sites ('home pages'), except on a very limited basis
  • promotional sites and advertising materials, although occasional samples may be taken for their value as ephemera
  • sites which chiefly serve the purpose of organising Internet information eg Guide to Australia
  • the records of businesses or organisations, which more properly belong in repositories of archival records.

Guidelines for selection for archiving will be reviewed regularly to ensure that selection procedures remain appropriate.

1.2.2 PANDORA categories of online publications

Although some electronic publications emulate traditional printed forms of publishing, there are some publications, such as 'home pages' that do not have any print equivalent. Even publications which seem superficially to emulate printed materials can behave very differently online. The overall ease of making changes online means that the content of any publication is rarely fixed, and can change - sometimes quite substantially - with time. It is essential to make repeated captures of online publications to reflect these changes, even monograph-like materials. For the purpose of the PANDORA archive, online publications have been categorised as follows:

  • Monographs
    • fixed content as in a traditional print publication
    • cumulative or evolving, whose contents change over time
  • Serials
    • regular serials - issues appear sequentially in traditional print publication patterns, and fit the definition of 'serial' for cataloguing purposes and for assignment of International Standard Serial Numbers (ISSN)
    • evolving serials, whose contents change over time
  • Home pages
  • Ephemera

These categories have been chosen to reflect the way the publications are collected and stored in the archive. The majority of publications selected for the archive fall into the monograph and serial category. At this early stage of online publishing some classes of publication are being sampled over time in order to build up a record of changes in online publishing. Newspapers will be treated like this.

Ephemera is considered a separate category to other online electronic publications, because of its different characteristics - see the Attachment for a discussion on the treatment of ephemera in PANDORA.

While most Australian online publications are currently available free of charge, some electronic publications lodged in the Library's collection are expected to have an associated fee, for example, a pay-per-view requirement, a subscription, or a "purchase" price per issue. This needs to be taken into consideration when providing access to the archived copy.

Under the Copyright Act of 1968, electronic publications are NOT subject to legal deposit in the National Library of Australia. Pending any change in the Copyright Act, permission is sought from individual publishers before a publication is archived. If changes to the Copyright Act lead to legal deposit for online electronic materials, this will have some impact on the operation of the archive, particularly with respect to negotiation with publishers and provision of access to fee based materials.

1.3 Progress to date

Work on PANDORA has proceeded at two levels:

  • to establish the specifications and operation of the final production archive
  • to implement a prototype, operational archive.

1.3.1 Operational specifications

A small team consisting of Wendy Smith (PANDORA project manager), Margaret Phillips (Manager, Australian Electronic Unit), and Debbie Campbell (PANDORA IT Research Officer) held a series of workshops in the first half of 1997 to establish the specifications of the archive. Leanne McKinnon (Senior Librarian, Electronic Unit), and Deb Woodyard (Electronic Preservation Officer, Preservation Services Branch) provided ongoing input in their areas of specialisation. Judith Pearce, Margaret Kennedy, Richard Stone and Jasmine Cameron were also called on for specific workshops.

Four major processes were identified for the archive:

  • selection
  • user access
  • archive management
  • report generation

These components are discussed in more detail in the sections that follow. A Logical Data Model is being produced which establishes the overall metadata for the final archive.

1.3.2 Prototype archive

As at June, 1997, over 1800 publications have been considered by the Electronic Unit for archiving in PANDORA, and around 180 have been selected. Work practices for selection, evaluation and contact with publishers have become standardised and are undertaken as part of routine work flows. A small online archive has been established. Access to publications held in it is available through the Library's online public access catalogue (OPAC).

2. The selection component

2.1 Identification, selection, documentation

The Electronic Unit (within Technical Services Branch) regularly scans the Internet to identify Australian online publications in scope for preservation by the Library. Suggestions also come from other members of staff and some external triggers, such as the request for an allocation of an International Standard Book Number (ISBN), an International Standard Serial Number (ISSN) or a Cataloguing in Publication (CIP) record from a publisher, can also bring publications to the attention of the Electronic Unit.

When a publication is identified, some preliminary details are checked to find if:

  • the publication has already been rejected
  • a print or other version is available in the Library's collection
  • the publisher produces or intends to produce an equivalent print-based publication

A publication passing these checks is then considered in more detail against the SCOAP guidelines, where it will be:

  • accepted for preservation OR
  • rejected OR
  • placed on hold, for later reconsideration

All decisions are recorded in a Select/Reject list, currently maintained online. The reasons for rejection are briefly noted. For publications placed on hold, the publisher is contacted for more information and/or the site is monitored to establish how it develops. Manual documentation is currently maintained for all selected or 'on hold' publications, in a SNIP (Serial/Non-serial Internet Publications) form.

For publications accepted for preservation, the following occurs:

  • the publisher is contacted for permission to archive the publication and to negotiate conditions of access from the archive. As the Library's involvement with the PURL resolver and metadata services is established, publishers will be told about these services and encouraged to use them. At this point, if the publisher does not respond, details about the publication are filed for future follow-up action. To date, no outright refusals have been encountered, but if this occurs, the information will be entered on the Select/Reject list.
  • descriptive metadata is entered on the SNIP form. At this stage it should contain details about: title; publisher/producer; (P)URLs; identifier, such as ISBN or ISSN; frequency of publication; life date; other notes, for example if a subscription is payable or if any restricted access conditions apply
  • once permission to archive is received from the publisher, Electronic Unit staff create a MARC record on ABN for the publication and include the descriptive metadata. The (P)URL is placed in the 856 tag and in the TED field. An ISSN is allocated to regular serials if this has not already occurred, and the publisher notified. In the future, PANDORA will create a unique identifier for every publication in the archive. This is likely to be based on a PURL resolver service to be established by the Library. This identifier needs to be assigned before or during cataloguing to reduce the need to amend catalogue records at a later date.

2.2 Technical evaluation

Concurrently or following receipt from the publisher of permission to archive the publication, an in-depth technical evaluation of it is carried out. Where appropriate, the following information is added to the existing documentation for the publication:

  • current issue, and method of compilation, for example whether it is an evolving serial or a cumulating monograph
  • date of evaluation
  • availability online of back issues, including details of archived material or previous issues, and any changes noted, for example whether back issues 'disappear' from the publisher's site
  • any copyright provisions quoted in the publication
  • the format of the contents, for example the recommended browser and other 'plug-in ' software needed for viewing the publication
  • the overall size of the publication or individual file sizes
  • method of access (generally via the World Wide Web)
  • hypertext links - internal and external - and the overall structure of the publication. Currently a brief diagram of the structure is recorded
  • any upward links to relevant information that should be captured with the publication.
  • email - (online) addresses
  • other contact details including postal address, telephone number, contact names
  • proposed gathering schedule
  • other metadata as appropriate.

If the publication is a serial, some additional access information may also need to be captured:

  • whether the publication could be sent regularly to an NLA email address
  • any password for the publisher's site which is sent by the publisher to the NLA permitting the publication to be downloaded from the external site
  • if the publisher has agreed to notify the NLA when each issue is posted or when other significant change occurs

3. The user access component

3.1 Users

Users include NLA staff; internal and external users of the Library; staff in other libraries; both Australian and international; and publishers and authors.

3.2 User access screens

At the moment, the only entry point for user access to the PANDORA archive is via the WebPAC (the OPAC's Web-based interface). The user is presented with standardised screens, which have, where possible, features in common with the Library's WebPAC or other standard NLA screens.

Additional screens, available to archive managers, will be needed for the overall administration of the archive by Library staff.

3.2.1 PANDORA archive title entry screen

For each publication, the following information is displayed:

  • PANDORA heading
  • title of publication, publisher, copyright statements, etc

An information button will be provided for the user to get a brief introduction to the purposes and scope of the archive, as well as general navigational hints. A search button will activate a search engine, for use within PANDORA. The search engine is likely to be restricted to author or title keywords.

Users will be encouraged to access the publisher's site for the latest issue or version of any publication and for any issues not yet captured by PANDORA. The publisher's (P)URL will be displayed above PANDORA's in the catalogue record and on the title entry page in the archive. The publisher's (P)URL will be an active link to the publisher's own site as long as it is maintained.

3.2.2 PANDORA archive holdings screen

Holdings details are provided on the title entry screen or on additional screens, depending on the amount of information available for the publication. A single title entry screen is usually sufficient for monographs, but serial publications may need additional screens. Holdings for journals will include issue number, date of capture and any access restrictions. Article level access has not been built into the model at this point. Holdings for monographs will include version numbers (assigned by PANDORA) and date of capture. Holdings information will have a hypertext link to the individual archived issue or version.

3.3 Restrictions on access

Although the Library wants to keep all restrictions on access to a minimum, there will be occasions when it is necessary to restrict access to some materials in the archive, either to all users or to some specific groups of users. Categories of access are likely to be:

Unrestricted access Most publications freely available on the Internet. Some fee-based titles may also fall into this category.
Partial commercial restriction Could occur for publications which involve payment of a fee to the publisher. Although the Library encourages publishers to deposit titles and to permit free access to both internal and external users, it is anticipated the majority of commercial publishers will permit free access to current materials to internal users only. Where a fee based online title is likely to be in high demand, the Library may also subscribe to it or purchase the right to access the publisher's site.
Full commercial restriction Could occur when, for a specified period of time, a publisher requires that no access is given to current issues/versions in the archive, in order to protect vested commercial interests. If access to a title is considered important for the provision of information services, the Library may subscribe to or purchase the right to access the publisher's site.
Partially restricted access Could occur for legal, moral or other reasons - for instance, if a publication contained pornographic material that the Library could not make available to users under 18.
Fully restricted access Could occur for legal, moral or other - for instance, if a publication is the subject of a court case or is banned.

Restricted access materials will still be displayed in the OPAC's browse lists. Where material is restricted - either in whole or in part - it will be necessary to flag this to users. Restrictions on access will be noted in the MARC record and/or in the title entry page details of the PANDORA archive.

4. The archive management component

4.1 Archive maintenance

The following table summarises the major activities that will occur as part of the maintenance of the archive:

Adding new publications to the archive ( part of the selection process to create a new holding in the archive)
  • creating a catalogue record on ABN
  • checking the PURL database to ensure new publications have been registered
  • collecting the publisher's metadata
  • collecting other PANDORA archive metadata
  • determining the file structure for the publication and its versions
  • activating the harvesting software. Currently, the URL indicating the location of the title (or home page) of the publication is provided to the harvesting software
  • creating the title entry screen
  • informing the publisher of the new link's availability
Enabling access to the archive (so users can find and access publications in the PANDORA archive)
  • adding a new publication to the search engine's indexes
  • capturing the software which supports the publication, for example, special browsers, and placing it into a repository
  • testing the captured software with the publication
  • converting documents from non standard formats to PANDORA's internal standard formats if appropriate
Updating the archive (an ongoing workload for management of the archive)
  • adding new volumes and issues to a captured electronic journal
  • tracking missing issues of electronic journals
  • adding new versions of a captured monograph
  • adding new versions of an evolving serial
  • updating the metadata (for both management and reader use) to reflect the addition of new content or other information about the publication
  • reviewing the subsequent capture of publications against previous versions or issues held in the archive.
Modifying a publication in the archive ( to assure the quality and completeness of the content and framework of each captured publication)
  • de-activating payment functions in the archived copy
  • removing duplicated content ( content which may have been harvested twice)
  • de-activating pull-down prompts for software browsers
  • deleting unwanted or extraneous parts of publications
  • de-activating any part of the publication with fully restricted access.
Deleting (removing whole publications from the archive)
  • harvesting errors
  • changes in collecting priorities over time
  • as a result of legal judgments

4.2 The capturing process

4.2.1 The gathering schedule

The gathering schedule is the time interval between each successive capture of regular serial issues or of monograph and cumulating serial versions. It is set by the Electronic Unit, partly as a result of observing patterns of change in any publication, and partly as a result of discussions with individual publishers. These discussions can clarify publishers' methods and procedures of publication such as how often content is added or deleted or how often material is archived. Publishers are also encouraged to provide information to the Library about any significant changes they are planning to make to their publications. It may be necessary to override the gathering schedule occasionally if a publication is about to cease, or if it is about to undergo significant changes in its presentation.

The gathering schedule does not necessarily match the date of issue of any electronic publication, since management issues also need considering. For example, a journal issued monthly may only be captured quarterly or annually to meet PANDORA workflows and priorities.

PANDORA also has the capability to capture a publication as a one-off event such as election results, festival programs, or theatre ephemera.

4.2.2 Harvesting

Currently, the PANDORA archive relies on HARVEST software to capture publications on a regular basis. HARVEST detects changes at specified URLs, and captures only URLs that have changed. It will be necessary to vary this in the future, in order to re-capture the whole publication when required.

The Electronic Unit performs a check on the captured results, to confirm that the harvested publication is present and correct. If it is, a link to this new archived version or issue is inserted into the archive, as an addition to the title entry page or holdings page.

5. Report generation component

A range of reports will be needed for the different users of the archive:

PANDORA managers, including Electronic Unit staff and Information Technology Branch
  • daily activities monitored and reported
  • reports to reflect the status of the archive, for example the number of new archived publications in a given time-frame, the number of publications which have had information changed, the overall number of publications
  • metadata reports
  • PURL resolver lists
  • technical reports - time taken by the capture process, number of URLs/files captured and space utilised, number and list of valid and invalid URLs for each publication
  • total space usage by archive
  • traditional database software reports
  • transaction log activity, including counts of the number of times each publication has been accessed
other NLA managers, corporate staff and Preservation Services Branch
  • various reports for preservation which must contain the metadata to allow decision-making to occur regarding the preservation action to be taken.
  • analysis by formats, age, software used etc by publications
other interested parties, State Libraries and publishers
  • lists of the publications in the archive or rejected for the archive
  • a list of publications belonging to a particular publisher

6. Storage and display formats for online publications

There is a range of different formats used to store and display online information - for text, image, text as image, sound, video. There is also a range of Web browsers needed to provide access to the information. Publishers are likely to select formats which meet their strategic needs and budgets. If the 'look and feel' of all publications is to be maintained, then the PANDORA archive will need to:

  • have access to all available formats and versions of software
  • archive publications in all the formats they are produced in

There are two major contenders of formats for general text based information - the PDF and SGML-like families. There are other contenders for scientific or other specialist publications. Several text formats can occur within the one publication, for example a home page in HTML and the rest of the text in PDF. Similarly a range of image formats or of audio formats can occur in any one publication.

Long term maintenance of 'look and feel' of archived publications is an issue that has not yet been fully resolved. Resource implications suggest that it will be impossible to maintain it forever for all materials. A critical decision will be whether preserving the information itself is more important then preserving the publication's look and feel. 'look and feel' may only be essential in a small minority of cases where the value of a publication as an 'artefact' outweighs the value of the information it contains. However, given the newness of Web publications, and the developments taking place in developing innovative Web styles of publishing, the 'look and feel' - at least for a selected number of publications - will be essential for the PANDORA archive.

At this experimental stage of the archive, it is appropriate to attempt to preserve the 'look and feel' of all publications in order to quantify resources needs for the long term maintenance of the archive. In order to maintain the full functionality of the archived publications in their original formats, a database of all possible browsing and interpretive software will need to be established and updated as needed. If the browser has been modified by the publisher in some way, it should also be captured and validated. Ideally, the archived publication will automatically invoke the repository version of software when it is accessed. Even if a decision is ultimately made to retain only a narrow range of formats, a database of software will still be required, but on a smaller scale.

It will also be necessary to experiment with conversion to alternative formats. In the short term, for experimental purposes, it is thought appropriate to ensure that all publications wherever possible, are held in at least the two main text formats.

Where a publisher creates a publication in a format which is deemed to be non-standard, options for the Library are to:

  • negotiate with the publisher to provide a copy of the publication in a standard format
  • ask for a copy of the software which created the publication
  • convert the publication to one of the archive's standard formats.

The issue of whether to convert every publication in the archive to one or more standard formats has been considered. The following table summarises current thoughts.

1. Conversion of all publications to one standard set of formats

Advantages Disadvantages
  • the informational content of any publication is the only significant reason for its capture
  • publications may become boring - there is a possible loss of creativity and functionality

  • it will be cheaper to maintain and manage all publications in the same format

  • it is a form of future-proofing provided the right single format is chosen
  • migration costs in the future will be reduced

2. Maintaining all existing formats

Advantages Disadvantages
  • the 'look and feel' of all publications will be retained, to convey a true overview of the state of electronic publishing throughout history
  • a conversion path which is valid over time will need to be identified, created and modified, in order to protect publications from technological obsolescence
  • maintenance and management of the conversion process will be costly

3. Storing all publications in both PDF and SGML-like text formats

Advantages Disadvantages
  • provides true future-proofing against the loss of any one format ( where an publication is currently published in more than one format, they are all captured)
  • the cost of storage, although currently inexpensive, may become costly depending on what platforms the archive is forced to migrate to over time
  • readers are not required to use one browser only - a range of browsers will already be available in the archive
  • managers of the archive will need to monitor the future of more than one standard

7. Issues of archive management

Some general considerations for the management of the archive include:

PURL resolver service and link maintenance
  • Publishers will be encouraged to use a PURL resolver service, to be established by the Library. Many PANDORA functions will use the service transparently to the user and the manager. This is described in more detail in the documentation for the service, for example on the OCLC Internet site (online at http://purl.oclc.org)
  • When a publisher's site disappears, the PURL resolver service can redirect the (P)URL to PANDORA's archived copy of the publication. The ABN/OPAC catalogue record does not have to be updated.
  • Disabled links, caused by disappearing URLs and other events, will be brought to the attention of Electronic Unit staff by periodically running a software "spider" against both the PANDORA archive and the PURL resolver database.
Resource discovery metadata
  • Information describing the electronic publications (which will allow easy access to them and management of them in the PANDORA archive) needs to be captured as smoothly as possible. One means of assuring this is by using a standard set of metadata. MARC will be sued for the catalogue record
  • Some information in the PANDORA archive may be managed under the Dublin Core standard for metadata. Using this standard gives the potential for automated capture of the information by individual publishers.
Administrative metadata
  • Various administrative and preservation metadata will need to be captured at certain times during the selection, acquisition and maintenance of a publication.
File structure for the publications within the PANDORA archive
  • The file structures for electronic publications usually have a first entry point at the title level of the publication, which provides administrative data - philosophy, pointers to archived issues etc.
  • Monographs can evolve, for example, a publisher issues a "book" and then adds to it a review or a comments 'chapter'.
  • For evolving serials, some publishers restructure their publications by adding or deleting features, columns, reviews etc. Issues can be archived or re-arranged, for example, by sorting and grouping articles in order by author. These activities may change the file structure of the original publication significantly.
  • The structure of publications varies from publisher to publisher. Each publisher creates a unique set of file naming conventions to bring together the disparate parts of a publication. The PANDORA archive has to:
    • provide correct access to publications at all times
    • deal with inconsistencies between file structures
    • manage the reuse of URLs by the publisher

8. Summary of progress and future directions

As at June, 1997, a small experimental archive has been established and made available to key stakeholders through the Library's OPAC. Over 1800 publications have been considered by the Electronic Unit for archiving in PANDORA, and around 180 have been selected. Work practices for selection, evaluation and contact with publishers have become standardised and are undertaken as part of routine work flows.

An overall business process model for the archive has been derived and evaluated by Library stakeholders in a review session. Further workshops have been held on important issues identified at that first session. These issues were:

  • allocation of unique identifiers
  • cataloguing procedures
  • national access to information
  • other issues, including tangible electronic publications and document supply.

The PANDORA business process model will be modified based on the feedback given by the stakeholders at these workshops. A preliminary logical data model has been established, which will similarly be modified as necessary.

The next steps will be to:

  • complete the description of the technical infrastructure required to support the PANDORA archive management facility and review it
  • seek management approval for acquisition of recommended systems

Work will also be undertaken on implementation of prototypes to support the PURL resolver service and the metadata server

For further information contact:

Paul Koerbin

Attachment - Treatment of ephemera in the PANDORA archive

Ephemera is considered as a separate category to other online electronic publications, because of its different characteristics. Library collections of ephemera are usually chosen on the basis of topic or subject and not at the publication level.

The major categories of materials to be collected as ephemera in PANDORA are those that:

  • are on topical issues which receive special attention for a finite period of time, eg gun control, euthanasia.
  • cover similar topics to those collected in print ephemera, such as the performing arts, tourism
  • demonstrate innovative use of the online publishing medium

SCOAP guidelines indicate that an electronic publication will be treated as ephemera when it is not sufficiently substantial to be preserved and catalogued in its own right. A one off snapshot only will be taken and it will not be regularly updated. A lower level of publisher negotiation is undertaken for ephemeral items. Publishers are advised that the item is to be archived unless they notify the Library otherwise.

Because ephemera is not under bibliographic control at the publication level, the lack of an entry in the catalogue means that the material is not easily accessible to users. A trial to catalogue at subject item level was undertaken for a sample of ephemeral items. Cataloguing at this level mirrors the treatment currently given to print ephemera. It proved the most successful approach for small single online documents.

As a result of this trial, ephemera is treated in the PANDORA archive in one of two ways:

  • collective catalogue records are created on ABN at the subject level for titles that do not stand well on their own. These items are gathered together into a single archive location under the most appropriate thesaurus term. Ephemera thesaurus terms are maintained with and are searchable on ABN. They are indexed and displayed on the Library's OPAC. Each subject is further organised for regularly published items on a given subject by a corporate body (eg. The Right to Life Association on euthanasia). Access is provided from the collective entry in the OPAC to a title entry screen that lists individual items and gives a hot link to them.
  • a full catalogue record is created on ABN for each item that stands well on its own and can be catalogued under a meaningful title or author entry. Access to the item is provided in the archive from the OPAC catalogue record via a title entry screen - the same as the main archive. In addition subject thesaurus terms are given to help retrieval.

Back to the top