PANDORA - Review of Progress to June 1997
Abstract
The National Library of Australia (NLA) is developing an electronic archive called PANDORA to provide long term access to significant Australian online publications. A dedicated online archive will be established in the first instance. Publications contained in it will be updated on an ongoing basis and converted to new formats as technology demands.
The processes of selection, preservation and access have undergone
close scrutiny. This report discusses the results of that scrutiny and
indicates future directions for the project
1. Introduction
1.1 PANDORA objectives
The PANDORA project has, as its main objective, to:
- capture, archive and provide long term access to significant Australian online publications selected for national preservation.
It will do this by:
- identifying and selecting online publications based on the SCOAP Guidelines for the Selection of Online Australian Publications Intended for Preservation by the National Library (online at http://www.nla.gov.au/1/scoap/scoapgui.html)
- cataloguing selected publications onto the Australian Bibliographic Network (ABN)
- arranging with the owners of identified publications to capture a copy of their publication for safe keeping in the National Library's electronic archive
- making the information from the archive available to users in line with fair dealing provisions, while taking into account the publisher's commercial interests
- preserving the "look-and-feel" of online electronic publications wherever appropriate
- updating the information in the archive on an ongoing and systematic basis, while maintaining date stamped previous issues or versions
- migrating information in the archive to new formats as current technology ages.
Other objectives, with wider implications beyond the PANDORA project itself, are to:
- contribute to discussions on extending copyright and legal deposit to online publications
- implement a system of permanent naming for Australian Internet publications to overcome the location problems associated with change of service provider
- implement and publicise a system of describing documents based on the Dublin Core attributes, to make online searching for information more efficient
- cooperate with other Australian libraries to ensure that there is no duplication of archived materials, and to establish a distributed national archive of electronic publications.
PANDORA does not sit in isolation from the rest of the Library's business. There are many other 'virtual library' developments within the Library, including collections digitisation projects, that all have a direct impact on the PANDORA project.
1.2 Online electronic publications
Significant resources are needed to establish and maintain an electronic archive.
Although all Australian online publications could potentially be included in the PANDORA archive, the amount of material and the special needs of electronic publications mean that the Library is only selectively choosing publications for the archive. A list of titles for preservation is now available and will be regularly updated.
1.2.1 Materials out of scope
Some major categories of material currently out of scope include:
- material which also exists in print format, unless the online version has significant additional information or value
- publications at the state level , where the National Library wishes to develop cooperative agreements whereby State libraries would agree to ensure long term access to online publications of the relevant State government, local governments within the State, State-based organisations and publications containing information that is specific to the State or a region of it
- organisational and personal sites ('home pages'), except on a very limited basis
- promotional sites and advertising materials, although occasional samples may be taken for their value as ephemera
- sites which chiefly serve the purpose of organising Internet information eg Guide to Australia
- the records of businesses or organisations, which more properly belong in repositories of archival records.
Guidelines for selection for archiving will be reviewed regularly to ensure that selection procedures remain appropriate.
1.2.2 PANDORA categories of online publications
Although some electronic publications emulate traditional printed
forms of publishing, there are some publications, such as 'home pages' that do
not have any print equivalent. Even publications which seem superficially to
emulate printed materials can behave very differently online. The overall ease
of making changes online means that the content of any publication is rarely
fixed, and can change - sometimes quite substantially - with time. It is
essential to make repeated captures of online publications to reflect these
changes, even monograph-like materials. For the purpose of the PANDORA archive,
online publications have been categorised as follows:
- Monographs
- fixed content as in a traditional print publication
- cumulative or evolving, whose contents change over time
- Serials
- regular serials - issues appear sequentially in traditional print publication patterns, and fit the definition of 'serial' for cataloguing purposes and for assignment of International Standard Serial Numbers (ISSN)
- evolving serials, whose contents change over time
- Home pages
- Ephemera
These categories have been chosen to reflect the way the publications are collected and stored in the archive. The majority of publications selected for the archive fall into the monograph and serial category. At this early stage of online publishing some classes of publication are being sampled over time in order to build up a record of changes in online publishing. Newspapers will be treated like this.
Ephemera is considered a separate category to other online electronic publications, because of its different characteristics - see the Attachment for a discussion on the treatment of ephemera in PANDORA.
While most Australian online publications are currently available free of charge, some electronic publications lodged in the Library's collection are expected to have an associated fee, for example, a pay-per-view requirement, a subscription, or a "purchase" price per issue. This needs to be taken into consideration when providing access to the archived copy.
Under the Copyright Act of 1968, electronic publications are NOT subject to legal deposit in the National Library of Australia. Pending any change in the Copyright Act, permission is sought from individual publishers before a publication is archived. If changes to the Copyright Act lead to legal deposit for online electronic materials, this will have some impact on the operation of the archive, particularly with respect to negotiation with publishers and provision of access to fee based materials.
1.3 Progress to date
Work on PANDORA has proceeded at two levels:
- to establish the specifications and operation of the final production archive
- to implement a prototype, operational archive.
1.3.1 Operational specifications
A small team consisting of Wendy Smith (PANDORA project manager),
Margaret Phillips (Manager, Australian Electronic Unit), and Debbie Campbell
(PANDORA IT Research Officer) held a series of workshops in the first half of
1997 to establish the specifications of the archive. Leanne McKinnon (Senior
Librarian, Electronic Unit), and Deb Woodyard (Electronic Preservation Officer,
Preservation Services Branch) provided ongoing input in their areas of
specialisation. Judith Pearce, Margaret Kennedy, Richard Stone and Jasmine
Cameron were also called on for specific workshops.
Four major processes were identified for the archive:
- selection
- user access
- archive management
- report generation
These components are discussed in more detail in the sections that follow. A Logical Data Model is being produced which establishes the overall metadata for the final archive.
1.3.2 Prototype archive
As at June, 1997, over 1800 publications have been considered by the Electronic Unit for archiving in PANDORA, and around 180 have been selected. Work practices for selection, evaluation and contact with publishers have become standardised and are undertaken as part of routine work flows. A small online archive has been established. Access to publications held in it is available through the Library's online public access catalogue (OPAC).
2. The selection component
2.1 Identification, selection, documentation
The Electronic Unit (within Technical Services Branch) regularly scans
the Internet to identify Australian online publications in scope for
preservation by the Library. Suggestions also come from other members of staff
and some external triggers, such as the request for an allocation of an
International Standard Book Number (ISBN), an International Standard Serial
Number (ISSN) or a Cataloguing in Publication (CIP) record from a publisher,
can also bring publications to the attention of the Electronic Unit.
When a publication is identified, some preliminary details are checked to find if:
- the publication has already been rejected
- a print or other version is available in the Library's collection
- the publisher produces or intends to produce an equivalent print-based publication
A publication passing these checks is then considered in more detail against the SCOAP guidelines, where it will be:
- accepted for preservation OR
- rejected OR
- placed on hold, for later reconsideration
All decisions are recorded in a Select/Reject list, currently
maintained online. The reasons for rejection are briefly noted. For
publications placed on hold, the publisher is contacted for more information
and/or the site is monitored to establish how it develops. Manual documentation
is currently maintained for all selected or 'on hold' publications, in a SNIP
(Serial/Non-serial Internet Publications) form.
For publications accepted for preservation, the following occurs:
- the publisher is contacted for permission to archive the publication and to negotiate conditions of access from the archive. As the Library's involvement with the PURL resolver and metadata services is established, publishers will be told about these services and encouraged to use them. At this point, if the publisher does not respond, details about the publication are filed for future follow-up action. To date, no outright refusals have been encountered, but if this occurs, the information will be entered on the Select/Reject list.
- descriptive metadata is entered on the SNIP form. At this stage it should contain details about: title; publisher/producer; (P)URLs; identifier, such as ISBN or ISSN; frequency of publication; life date; other notes, for example if a subscription is payable or if any restricted access conditions apply
- once permission to archive is received from the publisher, Electronic Unit staff create a MARC record on ABN for the publication and include the descriptive metadata. The (P)URL is placed in the 856 tag and in the TED field. An ISSN is allocated to regular serials if this has not already occurred, and the publisher notified. In the future, PANDORA will create a unique identifier for every publication in the archive. This is likely to be based on a PURL resolver service to be established by the Library. This identifier needs to be assigned before or during cataloguing to reduce the need to amend catalogue records at a later date.
2.2 Technical evaluation
Concurrently or following receipt from the publisher of permission to archive the publication, an in-depth technical evaluation of it is carried out. Where appropriate, the following information is added to the existing documentation for the publication:
- current issue, and method of compilation, for example whether it is an evolving serial or a cumulating monograph
- date of evaluation
- availability online of back issues, including details of archived material or previous issues, and any changes noted, for example whether back issues 'disappear' from the publisher's site
- any copyright provisions quoted in the publication
- the format of the contents, for example the recommended browser and other 'plug-in ' software needed for viewing the publication
- the overall size of the publication or individual file sizes
- method of access (generally via the World Wide Web)
- hypertext links - internal and external - and the overall structure of the publication. Currently a brief diagram of the structure is recorded
- any upward links to relevant information that should be captured with the publication.
- email - (online) addresses
- other contact details including postal address, telephone number, contact names
- proposed gathering schedule
- other metadata as appropriate.
If the publication is a serial, some additional access information may also need to be captured:
- whether the publication could be sent regularly to an NLA email address
- any password for the publisher's site which is sent by the publisher to the NLA permitting the publication to be downloaded from the external site
- if the publisher has agreed to notify the NLA when each issue is posted or when other significant change occurs
3. The user access component
3.1 Users
Users include NLA staff; internal and external users of the Library; staff in other libraries; both Australian and international; and publishers and authors.
3.2 User access screens
At the moment, the only entry point for user access to the PANDORA archive is via the WebPAC (the OPAC's Web-based interface). The user is presented with standardised screens, which have, where possible, features in common with the Library's WebPAC or other standard NLA screens.
Additional screens, available to archive managers, will be needed for the overall administration of the archive by Library staff.
3.2.1 PANDORA archive title entry screen
For each publication, the following information is displayed:
- PANDORA heading
- title of publication, publisher, copyright statements, etc
An information button will be provided for the user to get a brief
introduction to the purposes and scope of the archive, as well as general
navigational hints. A search button will activate a search engine, for use
within PANDORA. The search engine is likely to be restricted to author or title
keywords.
Users will be encouraged to access the publisher's site for the latest issue or version of any publication and for any issues not yet captured by PANDORA. The publisher's (P)URL will be displayed above PANDORA's in the catalogue record and on the title entry page in the archive. The publisher's (P)URL will be an active link to the publisher's own site as long as it is maintained.
3.2.2 PANDORA archive holdings screen
Holdings details are provided on the title entry screen or on additional screens, depending on the amount of information available for the publication. A single title entry screen is usually sufficient for monographs, but serial publications may need additional screens. Holdings for journals will include issue number, date of capture and any access restrictions. Article level access has not been built into the model at this point. Holdings for monographs will include version numbers (assigned by PANDORA) and date of capture. Holdings information will have a hypertext link to the individual archived issue or version.
3.3 Restrictions on access
Although the Library wants to keep all restrictions on access to a
minimum, there will be occasions when it is necessary to restrict access to
some materials in the archive, either to all users or to some specific groups
of users. Categories of access are likely to be:
| Unrestricted access | Most publications freely available on the Internet. Some fee-based titles may also fall into this category. |
| Partial commercial restriction | Could occur for publications which involve payment of a fee to the publisher. Although the Library encourages publishers to deposit titles and to permit free access to both internal and external users, it is anticipated the majority of commercial publishers will permit free access to current materials to internal users only. Where a fee based online title is likely to be in high demand, the Library may also subscribe to it or purchase the right to access the publisher's site. |
| Full commercial restriction | Could occur when, for a specified period of time, a publisher requires that no access is given to current issues/versions in the archive, in order to protect vested commercial interests. If access to a title is considered important for the provision of information services, the Library may subscribe to or purchase the right to access the publisher's site. |
| Partially restricted access | Could occur for legal, moral or other reasons - for instance, if a publication contained pornographic material that the Library could not make available to users under 18. |
| Fully restricted access | Could occur for legal, moral or other - for instance, if a publication is the subject of a court case or is banned. |
Restricted access materials will still be displayed in the OPAC's
browse lists. Where material is restricted - either in whole or in part - it
will be necessary to flag this to users. Restrictions on access will be noted
in the MARC record and/or in the title entry page details of the PANDORA
archive.
4. The archive management component
4.1 Archive maintenance
The following table summarises the major activities that will occur as
part of the maintenance of the archive:
| Adding new publications to the archive ( part of the selection process to create a new holding in the archive) |
|
| Enabling access to the archive (so users can find and access publications in the PANDORA archive) |
|
| Updating the archive (an ongoing workload for management of the archive) |
|
| Modifying a publication in the archive ( to assure the quality and completeness of the content and framework of each captured publication) |
|
| Deleting (removing whole publications from the archive) |
|
4.2 The capturing process
4.2.1 The gathering schedule
The gathering schedule is the time interval between each successive
capture of regular serial issues or of monograph and cumulating serial
versions. It is set by the Electronic Unit, partly as a result of observing
patterns of change in any publication, and partly as a result of discussions
with individual publishers. These discussions can clarify publishers' methods
and procedures of publication such as how often content is added or deleted or
how often material is archived. Publishers are also encouraged to provide
information to the Library about any significant changes they are planning to
make to their publications. It may be necessary to override the gathering
schedule occasionally if a publication is about to cease, or if it is about to
undergo significant changes in its presentation.
The gathering schedule does not necessarily match the date of issue of any electronic publication, since management issues also need considering. For example, a journal issued monthly may only be captured quarterly or annually to meet PANDORA workflows and priorities.
PANDORA also has the capability to capture a publication as a one-off event such as election results, festival programs, or theatre ephemera.
4.2.2 Harvesting
Currently, the PANDORA archive relies on HARVEST software to capture
publications on a regular basis. HARVEST detects changes at specified URLs, and
captures only URLs that have changed. It will be necessary to vary this in the
future, in order to re-capture the whole publication when required.
The Electronic Unit performs a check on the captured results, to confirm that the harvested publication is present and correct. If it is, a link to this new archived version or issue is inserted into the archive, as an addition to the title entry page or holdings page.
5. Report generation component
A range of reports will be needed for the different users of the
archive:
| PANDORA managers, including Electronic Unit staff and Information Technology Branch |
|
| other NLA managers, corporate staff and Preservation Services Branch |
|
| other interested parties, State Libraries and publishers |
|
6. Storage and display formats for online publications
There is a range of different formats used to store and display online information - for text, image, text as image, sound, video. There is also a range of Web browsers needed to provide access to the information. Publishers are likely to select formats which meet their strategic needs and budgets. If the 'look and feel' of all publications is to be maintained, then the PANDORA archive will need to:
- have access to all available formats and versions of software
- archive publications in all the formats they are produced in
There are two major contenders of formats for general text based
information - the PDF and SGML-like families. There are other contenders for
scientific or other specialist publications. Several text formats can occur
within the one publication, for example a home page in HTML and the rest of the
text in PDF. Similarly a range of image formats or of audio formats can occur
in any one publication.
Long term maintenance of 'look and feel' of archived publications is
an issue that has not yet been fully resolved. Resource implications suggest
that it will be impossible to maintain it forever for all materials. A critical
decision will be whether preserving the information itself is more important
then preserving the publication's look and feel. 'look and feel' may only be
essential in a small minority of cases where the value of a publication as an
'artefact' outweighs the value of the information it contains. However, given
the newness of Web publications, and the developments taking place in
developing innovative Web styles of publishing, the 'look and feel' - at least
for a selected number of publications - will be essential for the PANDORA
archive.
At this experimental stage of the archive, it is appropriate to
attempt to preserve the 'look and feel' of all publications in order to
quantify resources needs for the long term maintenance of the archive. In order
to maintain the full functionality of the archived publications in their
original formats, a database of all possible browsing and interpretive software
will need to be established and updated as needed. If the browser has been
modified by the publisher in some way, it should also be captured and
validated. Ideally, the archived publication will automatically invoke the
repository version of software when it is accessed. Even if a decision is
ultimately made to retain only a narrow range of formats, a database of
software will still be required, but on a smaller scale.
It will also be necessary to experiment with conversion to alternative
formats. In the short term, for experimental purposes, it is thought
appropriate to ensure that all publications wherever possible, are held in at
least the two main text formats.
Where a publisher creates a publication in a format which is deemed to be non-standard, options for the Library are to:
- negotiate with the publisher to provide a copy of the publication in a standard format
- ask for a copy of the software which created the publication
- convert the publication to one of the archive's standard formats.
The issue of whether to convert every publication in the archive to
one or more standard formats has been considered. The following table
summarises current thoughts.
1. Conversion of all publications to one standard set of formats
| Advantages | Disadvantages |
|
|
|
|
|
|
2. Maintaining all existing formats
| Advantages | Disadvantages |
|
|
|
3. Storing all publications in both PDF and SGML-like text formats
| Advantages | Disadvantages |
|
|
|
|
7. Issues of archive management
Some general considerations for the management of the archive
include:
| PURL resolver service and link maintenance |
|
| Resource discovery metadata |
|
| Administrative metadata |
|
| File structure for the publications within the PANDORA archive |
|
8. Summary of progress and future directions
As at June, 1997, a small experimental archive has been established
and made available to key stakeholders through the Library's OPAC. Over 1800
publications have been considered by the Electronic Unit for archiving in
PANDORA, and around 180 have been selected. Work practices for selection,
evaluation and contact with publishers have become standardised and are
undertaken as part of routine work flows.
An overall business process model for the archive has been derived and evaluated by Library stakeholders in a review session. Further workshops have been held on important issues identified at that first session. These issues were:
- allocation of unique identifiers
- cataloguing procedures
- national access to information
- other issues, including tangible electronic publications and document supply.
The PANDORA business process model will be modified based on the
feedback given by the stakeholders at these workshops. A preliminary logical
data model has been established, which will similarly be modified as
necessary.
The next steps will be to:
- complete the description of the technical infrastructure required to support the PANDORA archive management facility and review it
- seek management approval for acquisition of recommended systems
Work will also be undertaken on implementation of prototypes to
support the PURL resolver service and the metadata server
For further information contact:
Attachment - Treatment of ephemera in the PANDORA archive
Ephemera is considered as a separate category to other online
electronic publications, because of its different characteristics. Library
collections of ephemera are usually chosen on the basis of topic or subject and
not at the publication level.
The major categories of materials to be collected as ephemera in PANDORA are those that:
- are on topical issues which receive special attention for a finite period of time, eg gun control, euthanasia.
- cover similar topics to those collected in print ephemera, such as the performing arts, tourism
- demonstrate innovative use of the online publishing medium
SCOAP guidelines indicate that an electronic publication will be
treated as ephemera when it is not sufficiently substantial to be preserved and
catalogued in its own right. A one off snapshot only will be taken and it will
not be regularly updated. A lower level of publisher negotiation is undertaken
for ephemeral items. Publishers are advised that the item is to be archived
unless they notify the Library otherwise.
Because ephemera is not under bibliographic control at the publication
level, the lack of an entry in the catalogue means that the material is not
easily accessible to users. A trial to catalogue at subject item level was
undertaken for a sample of ephemeral items. Cataloguing at this level mirrors
the treatment currently given to print ephemera. It proved the most successful
approach for small single online documents.
As a result of this trial, ephemera is treated in the PANDORA archive in one of two ways:
- collective catalogue records are created on ABN at the subject level for titles that do not stand well on their own. These items are gathered together into a single archive location under the most appropriate thesaurus term. Ephemera thesaurus terms are maintained with and are searchable on ABN. They are indexed and displayed on the Library's OPAC. Each subject is further organised for regularly published items on a given subject by a corporate body (eg. The Right to Life Association on euthanasia). Access is provided from the collective entry in the OPAC to a title entry screen that lists individual items and gives a hot link to them.
- a full catalogue record is created on ABN for each item that stands well on its own and can be catalogued under a meaningful title or author entry. Access to the item is provided in the archive from the OPAC catalogue record via a title entry screen - the same as the main archive. In addition subject thesaurus terms are given to help retrieval.
