Go to National Library of Australia Home PageNational Library of Australia Online

SEARCH:

Networked Services Project
Request for Tender for NLA Networked Services

Appendix D - National Bibliographic Utility Work Flows; Duplicate Resolution Specification

  • National Bibliographic Utility Work Flows in a Z39.50 Environment
    • Traditional Cataloguing Workflow
    • Workflow Involving Use of Z39.50 Without the Update Extended Service
    • Workflow Involving Use of Z39.50 Update Extended Service
  • Illustrative Example of Specification for Duplicate Detection and Resolution

D.1 National Bibliographic Utility Work Flows in a Z39.50 Environment

As more Customer Libraries install Z39.50 capabilities in their local systems, a greater range of workflow options will become available. Three workflows are described below, as an indication of the range of workflows that the Library wishes the National Bibliographic Utility and the Cataloguing Client to support.

D.1.1 Traditional Cataloguing Workflow

The Customer Library:

a) Searches the NBD using the NBD search facility
b) Finds the bibliographic record on the NBD
c) Adds holdings to the NBD; transmits a copy of the bibliographic records + holdings in real time to the Customer Library's local system, or
d) At some later stage selects the bibliographic record + holdings for transfer eg by ftp or magnetic media.

D.1.2 Workflow Involving Use of Z39.50 Without the Update Extended Service

Work flows of this type would occur where the Customer Library's own ILMS supports Z39.50 but not the Union Catalogue Profile of the Z39.50 Extended Service for Update.

The Customer Library:

a) Searches the NBD using Z39.50 client capabilities on the Customer Library's own ILMS, or using a Cataloguing Client with Z39.50 capabilities
b) Finds the bibliographic record on the NBD
c) Captures a copy of the NBD bibliographic record using the Z39.50 Search and Present capabilities
d) Loads that bibliographic record to the Customer Library's ILMS, either directly or via the Cataloguing Client
e) Adds holdings data to the bibliographic record and transmits it back, with holdings, to the NBD at a later stage, through ftp or magnetic media.

D.1.3 Workflow Involving Use of Z39.50 Update Extended Service

Workflows of this type would occur where the Customer Library's own ILMS supports the Union Catalogue Profile of the Z39.50 Extended Service for Update.

The Customer Library:

a) Searches the NBD using Z39.50 client capabilities on the Customer Library's own ILMS, or using a Cataloguing Client with Z39.50 capabilities
b) Finds the bibliographic record on the NBD
c) Captures a copy of the NBD bibliographic record using the Z39.50 Present capabilities
d) Adds holdings data to the bibliographic record, either on the Cataloguing Client, or on the Customer Library's own ILMS:
i. if on the Cataloguing Client, transmit simultaneous updates, containing the bibliographic record + holdings, to the NBD and the Customer Library's ILMS, which under this scenario can accept records sent using the Union Catalogue Profile of the Z39.50 Extended Service for Update
ii. if on the Customer Library's ILMS, transmit the bibliographic record and holdings to the NBD.

A number of variations of these workflows are possible, including cases where the Customer Library captures a record and does not transmit any holdings to the NBD and cases where a book supplier or cataloguing agency creates cataloguing on behalf of a Contributing Library; the above three descriptions are simply three of the possible work flows.

The first workflow includes the equivalent of the present ABN 'downline loading' function. In the second workflow, downline loading is replaced by the Z39.50 present function. Data is transmitted back to the NBD using 'upline loading' functions. In the third workflow, data transmission in all directions occurs through Z39.50.

The Library wishes to support all three types of work flow, though expecting that, as more libraries' ILMSs implement Z39.50 and the Union Catalogue Profile, workflows of the first type will disappear.

D.2 Illustrative Example of Specification for Duplicate Detection and Resolution

To give some idea of the Library's complex needs for duplicate detection, one possible method of detecting and resolving duplicates is described below.

Allow an NBD Manager to choose the data elements on which detection of duplicates is to be based. Allow detection to be based on a single data element, eg ABN immutable (field 019 1), or on a combination of multiple data elements, eg LC Number (USMARC field 010), ISBN (020 sub-fields a and x), ISSN, title (USMARC 245 $a), GMD (245 $h), place of publication, year of publication, publisher, pagination and several other data elements.

Allow data elements to be given a weighting that indicates their reliability as indicators of duplication, eg a match on LC number (010) might be given a weighting of 9 (highly reliable), but a match on author might only be given a weighting of 3 (not very reliable). Allow an NBD Manager to vary the weighting for each data element from time to time. Allow an NBD Manager to set different weightings depending on the source of the incoming record, and on the type of record (eg serial or monograph). Allow higher weighting to be given where the match occurs on a greater number of characters (eg a matching ten-word title should receive a higher weighting than a matching two-word title).

Calculate the reliability of a decision that a pair of records are duplicates, by adding up the weightings of data elements that have the same content, eg the weightings for ISBN, author, title, place of publication, year of publication, and pagination.

Allow NBD Managers to set conditions for subsequent actions eg:

a) If score is over 40 and holdings are attached, allow the incoming record to replace the existing record (subject to a comparison of quality)
b) If score is 30 to 40 and no holdings are attached, allow the incoming record to replace the existing record (subject to a comparison of quality)
c) If score is 30 to 40 and holdings are attached, flag the pair of potential duplicates for human review
d) If the score is less than 10, add the incoming record to the database.

Allow use of fuzzy matching techniques when comparing two data elements, and allow a lower weighting to be assigned to cases where the data elements are similar but not exactly the same.

In cases where multiple records on the database are identified as potential duplicates of the incoming record and are flagged for human review, present the multiple records to the human reviewer in "most similar first" order.

Provide optional reports that sample the results of the duplicate resolution process and enable NBD Managers to assess the reliability of the decisions made by the system.

After determining that a pair of records are duplicates, choose the better quality of the two, according to a logic that compares record source, encoding level, date of creation/update and presence or absence of particular fields, sub-fields and MARC field indicators.

Before rejecting an incoming record, or replacing an existing record, check if the record to be discarded contains particular data elements that should be retained, eg ABN "immutable" numbers (019 1), or MeSH subject headings. Transfer these data elements to the surviving record. Allow managers to specify, and change from time to time, which elements should be retained.

Depending on the results of the match and quality check process, perform one of the following actions:

a) Add the incoming record to the database as a new record
b) Reject the incoming record, but load any holdings data that is attached to it
c) Replace an existing record with the incoming record
d) Merge the records
e) Flag the incoming record and its potentially matching records(s) for human review.

If holdings data is to be loaded, first check the holdings data that is attached to the database record. If that holdings data is already present on the database record, do not add that incoming holdings data.

There may be cases when an incoming record contains holdings data for a particular location, eg for University of Melbourne, but the content of the holdings data differs. Examples:

Existing:	VU	On order 19 May 98
Incoming:	VU	En f 631 ME

For each incoming file, allow NBD Managers to choose whether holdings data should replace existing holdings data for the same location.


Return to Start or Continue to Appendix E
About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us