Guideline
Federated Search Protocol
Introduction
This project will investigate the feasibility of supporting federated search across the collections of Australian cultural heritage institutions. The initial implementations of federated search are likely to be those developed by the Collections Australia Network (CAN), the Powerhouse Museum, and the National Library of Australia (in Libraries Australia and PictureAustralia). Other Australian collecting institutions are currently examining the feasibility of their participation.
Existing systems and federated search requirements
The National Library's library management system is currently based on the Voyager product from Endeavor. The National Library also manages the Libraries Australia service which uses TeraText from SAIC for its search database. The National Archives have developed their own collection management system called RecordSearch. The National Gallery use the EMU collection management system from KE Software. The Collections Australia Network service is based on Linux PostgreSQL servers.
The federated search protocol used in the pilot needs to be able to:
- Operate over http and support distributed searching across collections
- Easily integrate with a wide range of existing systems
- Be simple to implement to reduce barriers for participating instiutions
- Support simple keyword searching
- Optionally support fielded searching via simple unqualified Dublin Core
- Support searching for collection and/or item level records
- Support relevance ranking of results
- Support the application of persistent URLs for results records
Consideration was given to building a centralised metadata index for searching by harvesting (via the Open Archives Initiative Protocol for Metadata Harvesting) the collection records of all institutions contributing to the service. Although this model has been used successfully by the National Library in services such as Libraries Australia, PictureAustralia, MusicAustralia and ARROW, it was decided that the volume of metadata records, for example 4 million from the National Library and 6 million from the National Archives would make this architecture difficult to implement and not desirable.
The preferred architecture for the service is to do distributed searching across collections via a light-weight web protocol to build a single results page through which each collection's results can be accessed. Note that some aggregation of metadata records, for example from smaller institutions, would be compatible with this architecture in order to reduce the number of nodes being searched to a managable level.
Candidate protocols
It is proposed to build on existing protocols already in use and available on the web. Options examined include:
- SRU
- OpenSearch
- Australia's Fauna protocol based on the Darwin Core
The barriers to adoption of each protocol were considered. SRU promises the power of the Z39.50 information retrieval protocol packaged up using web services and http. Although SRU is relatively easy to implement, its wide ranging functionality still presents a considerable barrier to adoption by non-library cultural agencies.
The most light-weight protocol is OpenSearch. However, OpenSearch is designed for free-text searching and the OpenSearch 1.1 protocol does not support fielded searching. To meet this optional requirement, the OpenSearch protocol would need to be extended.
The Collections Australian Network protocol is currently unpublished but its author has indicated that it is particularly designed to support fielded searching of natural history museum's collection data via the Darwin Core metadata schema. The protocol is based on search via URLs and XML responses similar to those of OpenSearch and SRU.
On balance the OpenSearch option is considered to be the least risk and technically easiest to implement. The pilot project will also investigate the feasibility of extending the OpenSearch 1.1 protocol to support fielded searching.
Proposed federated search protocol
The protocol is based on OpenSearch 1.1.
Query syntax
The OpenSearch query syntax 1.1 includes these parameter values:
| Name | Value | Definition | Comments |
|---|---|---|---|
search |
{searchTerms} |
The terms to be searched for | URL-encoded search terms |
count |
{count} |
The desired number of results. | A positive integer |
startIndex |
{startIndex} |
The offset, in individual search results, of the first search result | A positive (non-zero) integer |
startPage |
{startPage} |
The offset, in pages of search results, of the first search result | A positive (non-zero) integer |
The OpenSearch query syntax 1.1 does not define any standard names for the query parameters. It is proposed to adopt for this project the parameter names listed in the first column of the above table.
There has been some discussion by the developers of OpenSearch and SRU about extending OpenSearch to support fielded searching via SRU's Common Query Language (CQL).
OpenSearch services created for this project can optionally support CQL with a minimal level of compliance defined as follows:
- Support for search by serverChoice (ie search against indexes determined by the remote server), dc.title, dc.creator and dc.subject with the default being searching against serverChoice;
- Support for phrase (indicated using "") as well as keyword AND (indicated using 'all') searching with the default being keyword AND searching;
- Support for boolean operators: AND, OR, NOT with the default operator being AND; and
- Support for additional parameter: level=c (for collection level records), level=i (for item level records) and level=a (for all records) with the default being level=a.
The definition of serverChoice will vary for each remote databases but will generally be designed to approximate free-text searching and will search across multiple fields within the database.
Some example searches are:
Example 1: Search for any level records with the keywords daisy AND bates
- http://some.domain.name/fedsearch?search=daisy%20bates&count=100&startPage=1
Example 2: Search for item level records with the phrase "daisy bates" in the Dublin Core subject field
- http://some.domain.name/fedsearch?search=dc.subject%3D%22daisy%20bates%
22&level%3Di&count=100&startPage=1
Example 3: Search for collection level records with the keywords daisy AND bates in the Dublin Core title field
- http://some.domain.name/fedsearch?search=dc.title%20all%20daisy%20bates%2
2&level%3Dc&count=100&startPage=1
Note that special characters within {searchTerms} such as :, ", ?, =, & etc need to the encoded to conform to the URL specification RFC 1738.
Results syntax
Results of searches will conform to the OpenSearch 1.1 Response. This specification allows results to be returned (OpenSearch response) in RSS 2.0 or Atom 1.0. This project recommends use of RSS 2.0 (as extended by OpenSearch) for results because, although it is not as technically sophisticated as Atom, it is simpler and currently more widely deployed on the web. Results sets from each remote target should, if possible, be relevance ranked. The results response from a remote server is an XML document. The OpenSearch response from the NLA server node to the second example search might look like this:
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<channel>
<title>Search of Libraries Australia items for "daisy bates" in dc.subject</title>
<link>http://nla.domain.name/fedsearch?search=dc.subject%3D%22daisy%20bates%
22&level%3Di&count=100&startPage=1</link>
<description>Search of NLA catalogue items for "daisy bates" in
dc.subject</description>
<copyright>&copy;2006, National Library of Australia</copyright>
<opensearch:totalResults>65</openSearch:totalResults>
<opensearch:startIndex>1</openSearch:startIndex>
<opensearch:itemsPerPage>10</openSearch:itemsPerPage>
<opensearch:link rel="search"
href="http://nla.domain.name/opensearchdescription.xml" type="application/opensearchdescription+xml"/>
<opensearch:Query role="request" searchTerms="Daisy Bates" />
<item>
<title>Daisy Bates</title>
<link>http://nla.gov.au/anbd.bib-an401513</link>
<description>[1st American ed.]. New York, Coward, McCann & Geoghegan
[1972, c1971]. xvi, 266 p. illus. 22 cm. Bibliography: p. [251]-253.</description>
</item>
<item>
<title>The long shadow of Little Rock, a memoir</title>
<link>http://nla.gov.au/anbd.bib-an712277</link>
<description>New York, David McKay Co. [1962]. 234 p. illus. 21 cm.</description>
</item>
<!-- ... -->
<item>
<title>Mrs Daisy M. Bates who is in charge of the Native Camp, Ooldea,
East-West Railway, 1921 Aug. 27 [picture].</title>
<link>http://nla.gov.au/anbd.bib-an26280040</link>
<enclosure url="http://nla.gov.au/nla.ms-ms365-88-466-t" length="10051" type="image/jpeg" />
<description>27 Aug. 1921. 1 photograph : b & w ; 12.6 x 11.1 cm. Part
of the collection:
Papers of Daisy Bates.
Also available in electronic version via the Internet at: http://nla.gov.au/nla.ms-ms365-88-466.</description>
</item>
</channel>
</rss>
The above item links go to Libraries Australia. Bibliographic record details are displayed along with options to get the item, online or via a library, document supply service or online bookshop.
The RSS <enclosure> element can be used to include a link to a thumbnail representing the item. This is an optional element. Fields displayed within the RSS <description> element include: publisher, publication date, description, summary and notes from the bibliographic record.
User Interface
This specification does not mandate any particular user interface. The federated search service user interface will allow users to do fielded or free-text searches of the catalogues and collection management systems of Australian cultural agencies. Returned results will be in XML and will be transformed to HTML before being returned to the user. Search results from each target will be accessible on a single web page but results from each source will not be merged.
Conclusion
It is recommended that a pilot project is undertaken to test the feasibility of the approach outlined in this document.
Tony Boston, October 2006, version 0.4.
