Commonwealth Government Metadata Pilot Paper

Introduction

Good metadata helps people discover the resources they seek. In the context of Government metadata it allows direct resource discovery by Australians and their libraries, facilitates the sharing and exchange of data between Commonwealth agencies, and supports the information management needs of Commonwealth agencies.

Since 1937 the National Library of Australia (NLA) has supported the role of all agencies in making Government information available to all Australians. By capturing and recording metadata on government publishing output in the Australian National Bibliographic Database (ANBD), the library has ensured that printed government publications can be identified and accessed by all.

In more recent times, the Library has actively pursued the acquisition of metadata for resources residing solely in electronic format or with links to full-text content. Government publishing is one such resource, moving online since 1995 and steadily becoming the predominant publishing format for Government information. Regardless of the format changes the library's objective remains the same - capture metadata for key government resources, include it in the ANBD, share it with all Australians.

To fulfill this goal, the National Library's Commonwealth Government Metadata pilot project was commenced in January 2003 with the objective of improving access to Commonwealth Government publications through increased coverage of electronic publications in the Australian National Bibliographic Database (ANBD).

Resource discovery metadata created by agencies during the project will be accessible through the ANBD (Libraries Australia's) search engines, harvested by search engines from agency sites, and exposed to services such as the Register of Commonwealth Government Publications. The accessibility of such metadata will make it easier for agencies to meet mandated requirements such as their Online Information Service Obligations under the Government Online Strategy, their legal deposit requirements under the Copyright Act 1968 and archival requirements of the National Archives of Australia.

The formal goals of this Commonwealth Government Metadata pilot are to:

  1. Enhance access to Australian government online resources.
  2. Facilitate identification, harvesting and preservation of electronic Commonwealth publications through services such as the NLA's Pandora Archive for electronic documents.
  3. Establish recommendations for best practice in metadata creation and maintenance, and workflows for contributing records to the ANBD.
  4. Review and test means of metadata harvesting and exchange

There are many issues surrounding metadata acquisition and loading that have been identified by the project to-date.

What is "Published"?

The ANBD is predominantly comprised of published materials in that the majority of the metadata contained therein describes items that are for general public distribution or access. When focusing on electronic resources it is widely accepted that online documents are considered `published' if the intent is to make them widely available to the public, either for free or at a cost.

Government Agencies in this pilot will use a definition of "published" established by the agency when creating their own metadata. A common definition of what is "published" and what is not has been sought but generally whichever online objects the agencies choose to take the trouble to attach metadata will be included. To advise agencies of resources that might be the most valuable and appropriate to describe and contribute to the ANBD refer to Guidelines for in scope contributions (Appendix A)

How much is enough?

To what level of detail should agencies go in describing their web objects with metadata- Web tools exist which allow every page created to have its own metadata description. In practice metadata should be used sparingly to describe whole objects or occasionally classes of objects or vary rarely a distinct part of an object.

Certainly published works will only be a fraction of all the items to which an agency will wish to attach metadata so it will be necessary to filter out items which are not appropriate for the ANBD. For more detail on these issues see, AGLS metadata elements (Appendix B)

Standards

Government agencies will adopt the AGLS metadata standard developed to promote consistency of metadata application across the Commonwealth public Service. However the Nwill also consider the acquisition of files already created under different standards such as Dublin Core, XML, and MARC21. Departments are encouraged to consider their obligations regarding online resources and metadata creation and consciously promote and apply the AGLS standard.

Quality of metadata

Also important is the quality of the metadata. Low quality metadata is worse than no metadata at all. Whether agencies choose to adopt the more complex library standards or adhere to local standards it is important that agencies consistently apply that standard across the whole organization.

Agencies may use the latest edition of the Anglo American Cataloguing Rules and Library of Congress Subject Headings (LCSH) directly or as a guide to creating good quality metadata.

Data harvested from-different agency sites will be processed onto the ANBD as discrete files. To ensure reliable and predictable outcomes for users searching the ANBD via Libraries Australia, a Public Libraries portal like InformationAustralia or a standard web search engine, each agency should supply data that is at least consistent in form and content across the whole agency. This quality control and consistency is not only important for resource discovery by Australians and information sharing needs between agencies but also vital for information management needs within agencies.

For further information on metadata standards and quality please refer to Applying metadata to Commonwealth Government Publications (Appendix C)

Record exchange mechanisms

Records created by the various agencies may be transferred to the ANBD through online data entry, the FTP of batch files or by web harvesting controlled by Harvest Control Lists (HCL). Workflows need to be tested, established and where possible automated to make record exchange between government agencies and the ANBD simple and efficient.

Preservation and archiving

In order to promote good preservation and archiving practices for these government publications, Libraries Australia will supply PANDORA with information which will enable it to identify electronic resources whose capture is within its terms of reference. Libraries Australia will provide this information through a regular listing of the most recent online Commonwealth Government publications. Agencies should consult Keeping government publications online : a guide for Commonwealth Agencies (2002) for further information and guidance on maintaining long-term access to online publications.

Conclusion

Provision of good quality metadata for Commonwealth government publications improves access to these resources through library resource discovery tools including the ANBD and Recent Australian Publications (RAP), government resource discovery guides such as Register of Archives and Manuscripts (RAAM) and internal search engines.

The metadata acquisition and loading issues identified by the project can be resolved. Existing documents (e.g. Keeping government publications online : a guide for Commonwealth Agencies (2002) ) provide adequate guidance to agencies regarding what is and isn't in scope and regarding metadata content and quality. Data standards need to be applied consistently in order to result in quality information for resource discovery.

To-date the scope of the ANBD has usually been defined in terms of the source of the data (i.e. Australian libraries and national cataloguing agencies) or in terms of its use (e.g. supporting resource discovery and interlibrary loans). But the scope of the ANBD can also be defined in terms of the resources that are represented: Now are also represented published resources that are part of Australian library collections or to which Australian libraries wish to provide access. Resources included in the ANBD are those that have substantial content, lasting value and where there is a commitment to ensure ongoing access.

New modes of record transfer technology such as Web harvesting should be utilized to ensure efficient and timely updating of access to items newly published, amended or deleted in the volatile world of electronic government publications.


APPENDIX A : Guidelines for in scope contribution to the National Bibliographic Database

Although agencies are required to create metadata for various objects not all such objects will be appropriate for inclusion in the National Bibliographic Database (ANBD).

What is a publication?

In Guidelines for Commonwealth Information Published in Electronic Formats (Version 1.1)/Auslnfo, a unit of the Department of Finance and Administration, where the obligation to preserve government publications is discussed the following statements infer a definition of published that is in keeping with conventional wisdom:

"A document intended for distribution to the general public, even if only a few copies" and "A document for online publication, which will only be available on the Internet, or documents as exposure drafts available online for public comment".
(p.17 of the Guidelines offers a definition of what comprises a "document".)

According to Keeping government publications online : a guide for Commonwealth Agencies (2002) a publication is information made available to the public, free of charge or for a fee.

Recommended list of publications in-scope for the ANBD

Building on the definitions in the afore-mentioned publications we can construct a list of the categories of objects that are of interest to the ANBD. Practically this is interpreted as the following categories of online publications:

Italicised entries are currently under review. `Documents' are defined as "largely text-based, but in the future likely to include a higher proportion of visual and audio formats, including multimedia aggregations or "objects". [Guidelines for Commonwealth Information Published in Electronic Formats, Version 1.1]

According to the Commonwealth Implementation Manual: Australian Government Locator Service (AGLS) Metadata, version 1.1 (July 2001)

government agencies should apply metadata to a minimum set of resources in order to comply with online information services obligations (OSIOs) as part of the Government Online Strategy. For a fuller description please refer to the Implementation Manual but a summary of the elements is provided her. These are:

a. Home pages (major entry point to an organization or distinct business) g. Major formal publications (e.g. annual reports, etc)
b. Topics/services in high demand by the community that the organization serves h. Media releases (every release on a site or most recent ones)
c. Information required by agency clientele to understand their entitlements to government assistance or obligations i. Major entry points or indexes and menus to a range of closely related topics, programs or policies
d. Pages that provide an actual online service to the public j. Information about agency powers affecting the public, and manuals and other documents used in decision-making affecting the public
e. Pages required to meet a prescribed community/legal/service obligation by the organization k. Substantial descriptive or marketing information about agencies, their services, activities and collections.
f. Entry points to specific online services and indexes

The list above indicates that not all the metadata that is being produced describes items of interest to the ANBD even though agencies are obliged to create it to fulfil other governmental obligations. For example only items 'g' and 'I' from this list would be of interest. This raises the issue of filtering out unwanted metadata that could be harvested from agencies sites.

For agencies that expose their metadata via a Harvest Control list, it will be necessary to prepare more than one HCL to exclude links to objects listed above. For agencies that can support either FTP mechanism or an OAI harvest, it will still be necessary to only stage for collection or harvest those files that contain metadata for in scope items.


APPENDIX B: AGLS metadata elements

AGLS element Commonwealth agency obligations
Creator Mandatory
Date Mandatory
Description Mandatory
Title Mandatory
Type Mandatory
Function Mandatory for collection level resources
Mandatory for service description
Mandatory if no Subject element
Subject Mandatory (if no Function element
Availability Mandatory (for descriptions of offline resources)
i.e. no Identifier element
Identifier Mandatory (for descriptions of online resources)
(i.e. no Availability element)
Publisher Conditional
Mandatory except for service descriptions
Audience Conditional
Mandatory when the target audience of the resource is not "All"
Coverage Conditional
Mandatory when the spatial coverage of the resource content is
not the whole of Australia
Language Conditional
Mandatory when the resource is in a language other than English
Contributor Optional
Format Optional
Mandate Optional
Relation Optional
Rights Optional
Source Optional

From Commonwealth Implementation Manual: Australian Government Locator Service (AGLS) Metadata, version 1.1 (July 2001), p.20, Table 1.


APPENDIX C : Applying metadata to Commonwealth Government Publications: an ANBD perspective.

Level of description

Currently agencies apply metadata at different levels, for example for the whole publication, for each issue of a publication, or for each chapter or page of a publication. In each instance the metadata can be either embedded in the object or separated from but linked to the object. In each case it is a business decision of the agency creating the data. The Project will investigate the issues related to the transfer and conversion of such different formats to the ANBD.

Whole of item

The ANBD to-date has consisted of whole of item descriptions. For example catalogue records are contributed for a book, a picture, a journal title or a movie.

In the online environment the level at which resources or information is described varies with local practice. Generally described at the object level it is possible to insert descriptions on each page of an object depending on the type of software tool used to construct the metadata. Online metadata may describe a class of objects or a part of an object. For example, a website may have its own description while various parts of the same site dealing with specific topics may have their own description as well.

Collection level

While the concept of collection level or multi-item description is not new to libraries, it has not been employed extensively outside the area of manuscript and ephemera collections where the description of individual items would be both complex and inefficient in relation to resource discovery. In many instances libraries have had to progress to whole of items to accommodate digitization initiatives. Collection level descriptions are discouraged as it is desirable to give direct access to the described resource via embedded URLs in the metadata descriptions for each item.

Analytic or sub-item cataloguing (eg chapter or article level descriptions) is not broadly encouraged in the library sphere and its usefulness depends on the availability and use within a system of sophisticated linkages between dependent parts.

It is not possible with the AGLS scheme to link related objects in any but a very simple way. We would also generally discourage the use of Analytic descriptions for these reasons.

It is also desirable from the point of view of keeping the description of the resource up to date to have the resource described discretely at the object level. This would allow the metadata descriptions to be updated independently whenever the object is updated rather than having to update part of a description or a class of descriptions.

Objects described by metadata other than at the object level will generally be out of scope for inclusion on this pilot.

Metadata Description standards

Metadata formats

It is desirable that the content of the metadata descriptions be standardised to allow both ease of conversion to other formats and the matching of data where that is necessary. For this project the main transpositions will be between MARC21 environment of the ANBD and the AGLS environment of web metadata.

XML

Agencies who encode in XML using the AGLS schema will also be able to export this data to the ANBD via Libraries Australia's Record Import Service service. Conversion of XML data to MARC21 format for loading to the ANBD is currently being tested by Libraries Australia with good results so far.

AGLS: Contributors should undertake to adhere to the AGLS standard supported by the National Archives of Australia. Described in Commonwealth Implementation Manual: Australian Government Locator Service (AGLS) Metadata, version 1.1 (July 2001)

Dublin Core: It is of course recognised that agencies may already have adopted Dublin Core or a variation thereof as a local metadata standard. Not wishing to prevent such agencies contributing it would be possible to negotiate with those agencies as to defining a specific transformation for their data for loading into the ANBD.

MARC21

MARC21 is the current standard for data exchange between libraries in Australia. Agencies whose libraries catalogue their print and electronic publishing output on a regular basis and whose local systems can export MARC21 data can currently export that data to the ANBD via Libraries Australia's free Record Import Service service.

Agency specific formats

Agencies may be using a home grown standard in relation to their metadata and records from such sources would have to be evaluated on a case by case basis to see if transformation of their data would be considered economical.

Metadata descriptive standards

The environment into which the harvested metadata records are to be imported is one to which stringent rules relating to content have been applied for many years in the library environment.

The principle being the Anglo-American Cataloguing rules 2nd edition (AACR2), the Library of Congress Subject Headings, and the MARC21 coding sequence. Government agencies may have developed local rules, thesauri and formats regarding content that are specific to their needs.

AACR2

The National Library of Australia's Libraries Australia network maintains a minimum level record standard for contributors which is based on AACR2 and is located at :
http://www.nla.gov.au/librariesaustralia/network.html#minimum

Subject descriptive terms

In the library environment the most widely used thesaurus is Library of Congress Subject Headings along with its sister publication Library of Congress Name Authorities. Though these tools are not widely used in the IT sector what we would like to emphasise at this point is the utility of using controlled vocabulary in metadata records and of giving an indication in the records which thesaurus is the source of the term.

Quality of meta description

Context: Record updating

The ANBD currently contains over 14 million records for bibliographic materials. When adding metadata derived records it is essential that the level of description of each object be sufficient to uniquely identify that record so that it may be retrieved for the purposes of maintenance and updating. In order to facilitate this agencies are requested to adhere to the Commonwealth obligation status of the 19 AGLS metadata elements listed in Appendix A.

Quality of metadata description

In order to create quality, consistent and descriptive metadata it is important that a common set of guidelines be adopted such as Guidelines for the Creation of Content for Resource Discovery Metadata / prepared by the National Library of Australia and the State Library of Tasmania (www.nal.gov.au/meta/metaguide.html). This document describes the typical content of AGLS fields, describes what to include and what to leave out of the meta description.

Further details relating to metadata quality are contained in Commonwealth Implementation Manual: Australian Government Locator Service (AGLS) Metadata, version 1.1 (July 2001), p. 30 onwards.

Persistent identifiers

The use of persistent identifiers is to be encouraged along with their inclusion in metadata. Agencies quite often move their data around inhouse but the use of persistent identifiers will make this transparent to their users. Where persistent identifiers are not used updates of the metadata records will be required for the ANBD every time the URL of an object is changed.

See Guidelines for Commonwealth Information Published in Electronic Formats(Version 1.1)/ AusInfo, a unit of the Department of Finance and Administration, p.79 where there is a discussion of the National Library's PURL Resolver Service.

Version control

Each differing version of an item is to be considered a new item. Where differing versions of items exist they should be clearly labelled as such. The Title and Date fields are often used for this purpose see Guidelines for the Creation of Content for Resource Discovery Metadata /prepared by the National Library of Australia and the State Library of Tasmania (www.nal.gov.au/meta/metaguide.html) pp4,9.

Format types

This Project will concentrate on the acquisition of metadata for monograph objects. Currently the NLA's online harvesting software cannot distinguish between monograph and serial items. This means that it would create a new record for every issue of a serial published. Agencies can apply filters in their Harvest Control List though exactly to what degree will be an issue for the Project to explore.