RIS processes & the match/merge algorithm

RIS Processes

The three basic processes involved in the batch upload of records to the ANBD are: file receipt, conversion and loading. These processes are outlined below. An overview of matching and merging is also provided. The functions described for record conversion, matching and merging are indicative. The software is configurable and modifications are made as the need arises or requirements change.

1. File receipt

Files are received in 4 ways:

  • Some files are automatically collected from remote sites by FTP. This includes all National Agency files and some files provided by library suppliers on behalf of their customers.
  • Files of up to 10Mb may be uploaded from the RIS section of the Libraries Australia Administration system.
  • Customer files can be placed on the Libraries Australia RIS server by FTP.
  • Files can be received as email attachments. Libraries Australia staff save these to a local drive and upload them to the Record Import Service gateway by FTP. These are usually test files, but a few customer files arrive this way.

2. File conversion

MARC conversion is performed on every RIS file before it is loaded to the ANBD.

The majority of conversions are generic and performed automatically on every file that is supplied to Libraries Australia. In addition, there are specific conversions available for National Agency files and customer files based on profiles set up for each organization. These conversions are done to change minor variations in the MARC formats supplied and to ensure that all bibliographic and holdings records loaded to the ANBD meet data quality standards, internal data storage formats and RIS load program validation routines.

Successfully converted files are sent for loading. Any unsuccessfully converted records are place in an error file for review by Libraries Australia staff or the contributing library.

3. File loading

Files are forwarded to the RIS load program and are generally loaded in the order of priority assigned to them. The highest priority is assigned to all National Agency files and current files from Libraries Australia customers. Lower priority is generally assigned to retrospective customer file loads, and large file loads (in excess of 20,000 records).

During the file loading process incoming records are either:

  • Added to the database as new records
  • Matched to an existing database record and merged with it
  • Tentatively matched with database records and sent to a review file, or
  • Rejected due to problems with the incoming data

Any tentatively matched incoming records are sent to the Libraries Australia error logs for review by Libraries Australia staff. Staff will resolve these tentative matches by adding them to the ANBD or matching them to an existing ANBD record. Records are very rarely rejected but occasionally there can be a problem in the format of a particular field that was not detected during the conversion process.

Error logs and file load reports are monitored and checked against previously kept statistics to see that the files are fully loaded and to check for any other signs of data problems.

Customers receive email notification of completed file loads and availability of reject files. They can then view summary loading statistics online for files that have been loaded and can download files of rejected records. This is done from the RIS section of the Libraries Australia Administration system.

Matching and merging

The RIS loading program's basic functions are to:

  • Detect matching records using an algorithm, which takes into account many bibliographic data elements including key control numbers, access points and coded data
  • When a match is confirmed, identify the preferred record.
  • When the preferred record is determined, amalgamate the records by in principle retaining data in the preferred record. However, some data elements in the non-preferred record may be transferred.

1. Matching

The RIS match/merge algorithm uses a number of different elements or matching keys to determine whether the incoming record has a possible match with an existing database record. The most commonly used keys include:

  • Local system numbers supplied in field 035
  • ANBD numbers in field 001
  • Library of Congress control numbers supplied in field 010
  • Standard numbers supplied in field 020, 022, 024
  • Title or shortened title key

Control number matching is done first (i.e. LSNs, ANBD or LC numbers)

If a match with one database record is found, the incoming record is passed to the merging process. If there is a multiple match, the record is rejected by the algorithm. These system numbers should be unique and not duplicated on the ANBD.

If no control number match is possible, the program will locate all possible matching database records by searching for other data elements such as the standard number or title key.

If a single match is found at this stage the process may move to the merging program. if the incoming record matches more than one database record, the program performs a more detailed comparison of these records. It aims to identify the database record with the highest level of similarity to the incoming record. If the incoming record fails to match any database records then it is added to the database.

2. Comparison

This is part of the matching program which compares incoming records with Database records. The comparison is done across a very comprehensive range of bibliographic data fields. All the bibliographic fields prescribed for a minimum level ANBD record are compared. Additional elements are also checked when appropriate to the type of material.

3. Merging

Once the best match has been determined, a separate merge algorithm identifies the preferred record by assessing which record is the best record. The concept of 'best' can be defined and configured by weighing up the relative value of various factors, including record source, encoding level, which tags are present etc.

As a general rule records created or edited online are preferred over uploaded records all other factors being equal.

Information in the preferred record is generally retained.

Fields from the un-preferred record are added if they do not exist in the preferred record.

When the un-preferred and preferred record share fields in common, under limited conditions it is possible that a specific field from the un-preferred record can overwrite its' corresponding field in the preferred record. This generally occurs only when these fields are thought to be similar enough (in cases where fields are repeatable) and also considered to contain 'better' data i.e. the data is thought to be of higher quality, more comprehensive or longer (therefore more complete).

Some examples of fields from the un-preferred record that may overwrite include:

  • Serial dates of publication (in field 362)
  • Citation (in field 510)
  • Electronic location and success (in field 856)
  • Projected publication date (this field is deemed 'better' if the un-preferred record has a higher encoding level)

Under very limited conditions, some contributors may send upgrades. Their records will overlay corresponding ANBD records, apart from some core data elements in the database record that are never overwritten. These records must meet certain quality criteria to prevent poorer records overlaying good quality database records.

These merging rules can be modified. However contributors should note that the rules are kept simple to minimize the risks of over-writing valid data. Assessing the data quality for the content of each field or for a record in its entirety is a complex and challenging task. Contributors who want to make specific changes to the content of a field may use the Libraries Australia Cataloguing Client but should note that the conventions for cataloguing in a shared network apply.