Staff Papers
Z39.50 Search and Retrieve Protocol - An Update
Paper given by Andrew Wells, NSP Project, at the Standards Australia Seminar, "Matching Discovery and Recovery" August 1997
Introduction
This paper provides a short overview of the protocol, including its development and functionality. Two specific applications of immediate interest to the library community are described. First, the use of the protocol in building virtual union catalogues is examined, and comparison made with the centralised union catalogue model. Second, the Union Catalogue Profile will be outlined. This profile is proposed to allow updating of centralised union catalogues and local systems by cataloguers working on local systems.
2. The Protocol
2.1 History
The current version of the protocol was published in 1995 and is called Information Retrieval (Z39.50): Application Service Definition and Protocol Specification. In this form it is an ANSI/NISO standard (American National Standards Institute, National Information Standards Organization). Previous versions were published in 1992 (version 2) and 1988 (version 1, now considered obsolete). It is now in the process of becoming (or has become) an international standard with ISO number 23950. The texts of the ISO and ANSI versions are identical. The standard is maintained by the Z39.50 Maintenance Agency at the Library of Congress. A copy of the standard can be downloaded from the Agency's home page at http://lcweb.loc.gov/z3950/agency. At this site, there is information about current developments of the standard and instructions on how to join the Z39.50 Implementors' Group (ZIG) discussion list.
2.2 Basics
The protocol "specifies data structures and interchange rules that allow a client machine (called an 'origin' in the standard) to search databases on a server machine (called a 'target' in the standard) and retrieve records that are identified as a result of such a search."1 The protocol confines itself to interactions between the client and server machines, and does not address interaction between a human user and the client machine. The standard is designed to facilitate interoperability between computer systems. The communication described in the standard is connection-oriented and stateful.
In the protocol, the client application initiates searches on the responding server application. This communication between the client (origin) and server (target)) takes place via a Z39.50-Association (Z-association) within an application association (A-association). The services covered by the protocol are carried out by messages passing between the origin and target. The origin searches and retrieves information from target databases. In an implementation, the origin and target convert their local forms of messages and responses in and out of Z39.50. This means an origin can maintain a consistent user interface for searching targets which support Z39.50, because the client machine's searching syntax can be mapped into Z39.50 queries. In this way, the origin extends the local interface to search external targets. On the target or server side, this requires considerable conversion and can be a complex mapping exercise. Databases differ considerably in structure, so the protocol employs a common, abstract model for describing databases. In each implementation, a database must be mapped to this model to enable communication between origin and target.
The abstract database model has the following elements:
- Database refers to a collection of records
- A database record refers to a local data structure which represents the information in a particular record
- Access points are associated with each database and are used in specifying searches
- Elements may be retrieved from a database record
Z39.50 should not be interpreted as a database indexing standard.
2.3 Search and Retrieval Facilities
Facilities consist of services. The Search and Retrieval facilities are the core functions of the standard.
A Search request can be put to one or more databases at a target system and contains a query. The records retrieved as the result of a query is called a result set. Several query-types are defined, including Type-1 for Reverse Polish Notation (RPN) and Type 101, which extends Type-1 for proximity searching and restricting result sets by other attributes. The protocol gives full support and mandates use of the type-1 query which consists of a single access point clause, or several clauses linked by logical operators. For example,
In the database named "Library" find all records for which the access point 'title word' contains the value 'glass' AND the access point 'author' contains the value 'white'
The attributes used in searching belong to a particular attribute set, "whose definition is registered, that is, assigned a unique and globally recognised attribute-set-ID, an Object Identifier, which is included within the query."2 The attribute set used by the bibliographic community is called bib-1. There are various attribute types and values. For example, the Use attributes provide values for elements in a record, such as Conference name having value 3. Z39.50 targets map the Use attributes to indexes in the databases on the server. Other types include Relation (e.g., less than), Position (e.g., first in field, any position in field), Structure (e.g., phrase, year, numeric string), Truncation and Completeness.
Retrieval consists of two Z39.50 services: Present and Segment. In a Present response, the origin requests response records in the result set. The origin may specify a preferred syntax, and Z39.50 supports a number of such syntaxes, ranging from the familiar MARC syntax to the general, but very complex GRS (General Record Structure) syntax. If the target cannot support the number of response records requested by the origin, the target segments the response.
2.4 Other Facilities
Initialization. With the 'Init' service, the origin establishes a Z-association.
Result-set-delete. The origin deletes specified or all result sets created during a Z-association using the Delete service.
Access-control. This allows a target to interrogate an origin about its authorisation to establish a Z-association or carry out a specific operation.
Accounting/Resource Control. Services in this facility allow origins and targets to communicate information and warning about resource consumption. For example, the origin may submit a query which exceeds price limits. The target relays this information to the origin.
Sort. With the Sort service, the origin can request the target to sort a result set.
Browse. This employs one service called Scan, which enables scanning of ordered term-lists, such as subject headings.
Explain. This facility allows an origin to query a target about implementation details, e.g., which databases are available, attribute sets and record syntaxes used. The facility is "intended to permit the development of clients that to at least some extent are dynamically self-configuring as they encounter various servers."3 The Explain facility is not widely implemented. However, there is an existing need for origins to find out what targets do support in order to enable successful implementation. The National Library of Australia is investigating ways of creating a register or directory of Z39.50 targets in Australia. The need for this was raised at the March 1997 meeting of Standards Australia Committee IT/19. In Canada, a self-reporting directory has been established (the register can be found at: http://novanet.ns.ca/vCucadm.html)
Termination. This allows an origin or target to immediately terminate all operations and initiate termination of the Z-association. The service used is called Close.
2.5 Extended Services
Version 3 introduced a new service called Extended Services, which permits tasks to be performed outside a Z39.50 session. These tasks are more complex than a search and retrieve operation, and will usually be carried out after the Z-association initiating the task was established. The progress of the task can be checked using other Z39.50 services. "The Extended-Services (ES) service allows an origin to create, modify, or delete a task package at the target. The target maintains task packages in a special database."4
The Extended Services defined by the standard can be found in Appendix 8. They include:
Persistent Result Set. This enables the origin to request the target to save a search result.
Persistent Query. In this service, the target saves a Z39.50 Query in response to a request from the origin.
Periodic Query. The origin requests the target to save a query and run it periodically, according to a schedule specified by the origin.
Item Order. The origin can submit a document delivery request to the target
Database Update. The origin requests that the target update a database by insert new records, replacing or deleting existing records, or by modifying elements within records.
Export Specification. The origin specifies the format, delivery mechanism and destination of records from result sets.
Export Invocation. This service lets the origin request delivery of records, according to an Export Specification.
This is a very brief outline of this complex standard.
3. Z39.50 and Access to Distributed Union Catalogues.
The facilities of Z39.50 have been put forward as a way to create a decentralised union catalogue. In this scenario, an origin sends a Z39.50 query to multiple Z39.50 targets. The query is sent either serially or concurrently. The targets then returns the result set to the origin. In some implementations, the result set is the number of records found at each target, e.g. Library A had 10 matches, Library B had 23 matches, and so on. In other cases, the results are listed in brief citation form. Duplicates can be retrieved, but result sets are not merged. A study at Cornell University indicated that this would not satisfy user requirements, and suggested that client software will be needed to integrate different result sets from servers into a single, non-redundant record list.5 This is a complex operation for client software, because the number of records involved can be large, and can come from databases of different composition and structure (e.g., citation and bibliographic databases). Removing duplicates where matches can be made on control numbers such as the ISBN is relatively straightforward.
In Canada, there has been significant research on the use of Z39.50 in accessing distributed union catalogues. The project is known as VCUC, the Virtual Canadian Union Catalogue (see http://www.nlc-bnc.ca/resource/vcuc). The context for this study is the use of union catalogues for resource sharing, giving the retrieval of location, holdings and availability data a particular focus. The study is not aimed at use of union catalogues for supply of cataloguing data.
An early problem has been the provision of location, holdings and circulation information in response to a Z39.50 query. There is no agreement within the Z39.50 Implementors' Group on how to solve this problem, and the issue will be discussed at the August 1997 ZIG meeting in Copenhagen. The problem is compounded by the lack of consistency in the storage of holdings data in local and union catalogues. In Australia, there has not been widespread implementation of the USMARC formats for holdings and location data. USMARC also allows summary information for holdings to be embedded in bibliographic records in tags 850 and 852, but there is not much evidence of this practice in Australia. Generally, holdings data is held in locally defined 9XX MARC fields. Consistency in the recording of location and summary holdings would facilitate interoperability. Even so, this would retrieve summary holdings only. Detailed holdings could be held in another place on the network, such as a different database on the same server. Retreiving these detailed holdings would involve two step searching. It has been proposed that the target return the URL for the detailed holdings as part of its response containing bibliographic and summary holdings information.
Other methods for retrieving location, holdings and circulation information are currently being proposed. The use of the OPAC record syntax 'wraps' clusters of bibliographic and holdings records and permits the transfer of circulation information.6 However, the implementation of this syntax is complex.
The retrieval of bibliographic information in decentralized union catalogues has met problems in search precision. The mapping of bib-1 Use Attributes to database indexes varies from server to server, and can sometimes be too broad. For example, the Author attribute may be mapped to other indexes which may contain authors' names, such as notes and titles: searching on this attribute may retrieve the name, but not within the user's specific context of searching for the author of particular works. The use of a diagnostic message indicating that the target does not support the attribute may be preferable. It is clearly worthwhile for vendors to agree on a common set of Use attributes.7
There are challenges in attempting to make access to distributed catalogues emulate a union catalogue, and it is worth questioning the distributed approach. The National Library of Canada believes that the two approaches will be complementary. Union catalogues do provide consistent indexing and searching of records which come from multiple sources, although this often comes at high costs of background processing. Union catalogues integrate or consolidate records from multiple sources far more effectively than current implementations of Z39.50 broadcast searching approaches. While union catalogues scale well, the scaling of virtual union catalogues needs further investigation: the number of targets being searched, slow responses at servers and loads on servers at small libraries are factors which could affect performance. Lynch believes that virtual union catalogues will find use in scholarship and research, particularly when the user can select the targets to be searched. Union catalogues will continue to support resource sharing. He concludes that "while Z39.50 is limited in its ability to support the dynamic federation of databases for distributed searching today, it has been highly successful in the more limited role of extending familiar local user interfaces to remote databases outside a local system, particularly if this is done as a crafted implementation rather than just an ad hoc incorporation of random external databases."8
4. Union Cataloguing Profile
The interest in virtual union catalogues possibly lies in the potential capabilities of Z39.50 and the advantages of using a local system for all library operations. There are extra costs in having several systems for collection management and resource sharing, and it is clear that some libraries are interested in developing and maintaining autonomy through managing all operations through one local system. At the same time, there are good reasons to maintain a national union catalogue, particularly for resource sharing. A centrally maintained database of location and holdings information is an important piece of resource sharing infrastructure and assists in timely service to users.
In recognizing that many libraries are interested in using their local system for cataloguing, the National Library of Australia is assisting the development of the Union Catalogue Profile (see http://www.nla.gov.au/2/abn/ucpzdrft.html). The Profile employs the Update facility of the protocol's Extended Services to enable a client to update both a local system and a union catalogue. Various workflows can be devised, but one which combines searching and updating of both local and union databases is desirable. The advantages of the Profile include the single interface, and simple configuration for hardware and software. It is proposed that the Profile covers straightforward search and enquiry functions, as well as complex global update and merging transactions. The text of the Profile outlines many operations , and shows how the profile would accomplish tasks. Two examples are:
- Add holdings to a bibliographic record on a union catalogue as part of the process of obtaining a copy for the local system
- Delete a holdings record on the local system, ensuring this is reflected on a union catalogue as part of the same process
The Profile also proposes use of other Extended Services for regular supply of data. This could be used for collection development purposes (e.g., send all monographic works added to the database published in Queensland) or for collection management purposes (e.g., send records with my library's holdings which have been amended). The Profile proposes these operations as follows:
- Place standing orders from another server for records based on specific criteria (through use of Persistent Query, Periodic Query Schedule, Export Specification and Invocation with the other server as target and the local system as the Export Destination)
- Submit a list of search keys to another server and obtain copies of any matches, with the option of rerunning the query at regular intervals for non-matches (through Persistent Query and Periodic Query Schedule)
The Union Catalogue Profile was developed by Standards Australia Committee IT/19, and has been presented by Australian representatives at meetings of the Z39.50 Implementors Group (ZIG). There are a small number of issues expected to be resolved at the August 1997 ZIG meeting. These include a secure mechanism for global update and an optional scheme for record locking, which is required for utilities where simultaneous editing of a record by more than one cataloguer is not allowed. Due to the large number of operations covered by the Profile, it is likely that implementation will be gradual.
References
1. Clifford Lynch, "The Z39.50 Information Retrieval Standard. Part 1, A Strategic View of Its Past, Present and Future".In D-Lib Magazine (http://sunsite.anu.edu.au/mirrors/dlib/dlib/april97/04lynch.html) [online]. April 1997.
2. Z39.50 Maintenance Agency, Information Retrieval (Z39.50): Application Service Definition and Retrieval Specification (Washington, 1995), p. iii
3. Lynch, "The Z39.50 Information Retrieval Standard"
4. Z39.50 Maintenance Agency, Information Retrieval (Z39.50), p. 26
5. Sandra D. Payette, Oya Y. Rieger, "Z39.50, the User's Perspective" In D-Lib Magazine (http://sunsite.anu.edu.au/mirrors/dlib/dlib/april97/cornell/04payette.html) [online]. April 1997.
6. Carrol Lunau, Fay Turner, Issues Related to the Use of Z39.50 to Emulate a Centralized Union Catalogue (http://www.nlc-bnc.ca/resource/vcuc/ezarl2.htm) [online] (Ottawa: National Library of Canada, 1997
7 Lunau.
8. Clifford Lynch, "Building the Infrastructure of Resource Sharing: Union Catalogs, Distributed Search and Cross-Database Linkage" In Library Trends , Vol. 45 (1997), p. 457
