[Commonwealth of Australia]
Functional Requirements for a Whole-of-Australian-Government Search Architecture


1. Introduction

1.1 Terms of Reference and Membership

The Commonwealth Government Search Engine Working Group (SEWG) was formed in October 1997, under the auspices of the Office of Government Information Technology (OGIT). The Group was asked to report by mid-January 1998 on the following:

The membership of the Group is given in Appendix 1.

1.2 The Information Management Steering Committee Report

In the past two years Commonwealth and State Government agencies have discussed the desirability and means of providing a seamless means of online access to Australian Government information.

The Information Management Steering Committee (IMSC) considered these issues during 1996. It issued a draft report in late 1996, and a final report in mid 1997, entitled Management of Government Information as a National Strategic Resource. This report addressed the broad principles of information access from both the client point of view and the technical and change management points of view. It examined some of the requirements for a seamless access environment, and identified the need for appropriate resource description of Government information, both electronic and paper based, and the need for an appropriate framework to search, locate and interpret such information through a single official Australian Government entry point.

1.3 Working Group on Government Information Navigation

In 1997 a Commonwealth/State reference group, the Working Group on Government Information Navigation, considered the issues involved in both online information access and online service delivery. In July 1997 this group completed a report entitled Improving Access to the Information and Services of Australian Governments. This report also examined specific aspects of the whole-of-Government online environment such as browse access to information, means of notification and highlighting of new material, technical issues involved in broad-based searching, resource description or metadata standards, and directories of people and functions.

1.4 The Role of the Search Engine Working Group

To attempt to move towards the implementation of the above reports, the Government Technology and Telecommunications Committee (GTTC) formed the Commonwealth State Navigation Working Party. This Working Party acts on behalf of the Commonwealth and States and steers a Consultancy aimed at developing a plan for the implementation of seamless navigation of government online services, including a functional specification, a business case and proposed management structures.

The Search Engine Working Group (SEWG) contributed to this process by developing this national position in relation to the functional requirements for a whole-of-Australian government search facility, based on principles developed in the documents referred to above. The issue of browse access to government websites was beyond the terms of reference of the SEWG.

The SEWG effectively reports to the Commonwealth State Navigation Working Party. That Working Party also received a report from the Australian Government Locator Service (AGLS) Working Group 2., which was charged with the formulation of locator metadata standards for Australian Government information.

1.5 Use of Public Search Engines

At present, those who are searching for Australian government information on the World Wide Web are obliged either to browse for information using starting points such as the Australian Governments'’ Entry Point or are obliged to use a public search engine such as AltaVista, Lycos, HotBot, Yahoo! or Anzwers. Most search engines do have advanced search features which allow searches to be limited to specific domains such as “gov.au”. However, a number of Australian government websites do not use the “gov.au” domain.

The report of the Working Group on Government Information Navigation commented as follows on this issue.

Most current popular search engines rely on the harvesting of text words from the HTML pages of World Wide Web sites. Whilst allowing a considerable amount of relevant information to be discovered, these search engines can have unsatisfactory results for a number of reasons:

Although it is possible with some of these search engines (eg AltaVista, Anzwers) to limit searching to a particular domain such as "gov.au" and thus retrieve only Australian Government information, such a search is likely to miss relevant information. Continuing dependence on these search engines for the discovery of Australian government information is not therefore recommended. To ensure the retrieval of all relevant information and to limit the amount of irrelevant information retrieved, two approaches are recommended:

1.6 Centralised or Distributed Search Architecture

The report of the Working Group on Government Information Navigation went on to observe:

A key issue dealt with in the Technical Group report was that of search architectures. Essentially the choice is between a single central search engine which harvests the metadata and/or text words from all of the sites in the domain, or a distributed architecture in which the search engine at the Entry Point operates in cooperation with search engines at the local sites. The Technical Group report commended the distributed approach, but it is recognised that this model may have to be implemented over the medium term, with a central search engine providing an appropriate model for the short term.

This issue is dealt with further in Section 4 of the present report.

2. Business Issues

Whilst the focus of the Search Engine Working Group (SEWG) has been on addressing the technical aspects of a whole-of-Australian Government search facility, the following business issues will need to be considered.

2.1 Definition of a ‘Government’ Site

For a user to be confident that they were searching across the whole of government in Australia, the search facility would need to identify all Australian Government sites. Under the current practices of domain name registration, this cannot be guaranteed as not all government sites are registered under the domain “gov.au” (eg theAustralian Broadcasting Commission (ABC), Commonwealth Scientific and Industrial Research Organisation (CSIRO) and Cinemedia). Therefore it will be necessary for all Australian government sites to register with the search facility, as it is probably not feasible to mandate that all government sites use the “gov.au” domain name.

2.2 Variable Search Support Across Government

Government agencies are at varying levels of size and sophistication in the provision of information on the Internet. They include large and sophisticated sites such as SCALEplus; subject entry points such as ERIN and EdNA which act as aggregates by providing services to smaller sites; and some very small sites which do not yet have their own search engines. Any whole of government search facility will need to accommodate searching across the whole range of sites. The whole of government search facility therefore will need to access local indexes by utilising existing search engines at sites which have them, but will also need documents on sites which lack search engines. Moreover the search engine will need to search for documents in different data formats (eg HTML, XML, SGML, PDF) and across a range of spaces (eg gopher, FTP sites etc).

2.3 Individual Agency Search Engines

As stated in Section 2.2, Australian government agencies vary in size and the extent to which they have developed their Internet presence. Agencies which have not yet deployed search engines on their sites could usefully be provided with assistance in the selection of search engines. Further implementation of the whole-of-Australian government search facility will be easier if a narrower range of search engines are used across government agencies.

The SEWG has not tested the market place for the availability of search engines which meet the functional requirements listed in Section 3. These functional requirements, whilst developed for the whole-of-Australian government search facility, (ie the middle layer) could be modified for use as a checklist for individual site search engines.

2.4 Metadata

The use of metadata has been proposed as a means of improving the precision of searching for Internet resources by explicitly encoding the fact that certain words are key subject terms, the name of the author, and so on. In addition, metadata can be used to describe resources other than HTML text files (including printed documents), thus enabling them to be retrieved by a search.

In the case of HTML files, the metadata can be embedded within the document (using the HTML

tag) but a more satisfactory and flexible approach is for each agency to create its own metadata repository or database, linked to its various information resources. This data may then be either searched in a distributed system or in a merged database.

During the last year, considerable progress has been made with the development of the ‘Dublin Core’ metadata element set. This standard is being promoted as a generic metadata standard for use by libraries, archives, government and other publishers of on-line information. At present, the standard is a set of semantic elements to describe basic document like objects.

The AGLS Working Group has recommended a metadata set that is Dublin Core compliant for use by whole-of-Australian government agencies for metadata creation.

2.5 Resource Implications

Large volumes of metadata may not be generated by individual agencies until tools become available to simplify and streamline the process of metadata creation and maintenance. To achieve these aims the National Library of Australia and the Distributed Systems Technology Centre (DSTC), together with the Australian Defence Force Academy (ADFA) and Charles Sturt University (CSU), are involved in an important new project which aims to develop and disseminate such tools. The specific tasks of the MetaWeb Project include the development and promulgation of a number of tools which can facilitate the creation and maintenance of metadata. The project will also develop and test a generic Australian metadata repository model and associated software.

Even with the advent of automated tools for the creation of metadata in the form recommended by the AGLS Working Group, all government agencies will need to consider both the resource and training implications. A business case will need to be developed to reinforce the importance of agency compliance with standards. It is likely to be some years before all agencies will have the capacity to create Dublin Core metadata and to deploy search engines on their sites.

2.6 Transition Mechanism

The search model defined in Section 4 defines a staged process of implementation to enable some access to be provided immediately whilst more sophisticated mechanisms are being developed both at individual sites and by the on-line industry. Ultimately when all agencies have reached a level of maturity and sophistication in the deployment of standards on their web sites, a distributed search facility is considered to be the best solution.

During the transition phase it may be necessary to have a model which uses the local index in the case of sites which are more developed but also allows the creation of a common index for sites without their own search engines.

3. Functional Requirements

3.0 General Issues

It is anticipated that the integrated search facility for the whole-of-Australian Government information will initially run across federal government information but ultimately extend to the state and local jurisdictions. The nature of the online environment and technological development will mean that the functional requirements of the search facility will continue to evolve. The success of the product is, therefore, dependent on continuous feedback and development.

Overriding principles are that:

The functional requirements for a whole-of-Australian Government search facility are specified below.

3.1 Scope of Searching

Users should be able to conduct cross-jurisdiction, cross-agency and subject searches for information on Australian government WWW sites. They should be able to search:

3.2 Data Elements to be Searched

Users should be able to search, using keywords or phrases, on the principal metadata elements such as:

These facilities should also draw upon the resources of various Australian government thesauri; directory services; functional descriptors; geographic indexes and data sets (eg GIS data sets, post codes, electoral and other boundaries) for the provision of search limiting.

3.3 Improving Search Precision

The search facility should be able to improve precision of searches with the following features:

In addition to these search features, users should be able to refine/limit their searches (with minimal guidance) by:

3.4 Displaying Search Results

Results yielded from a search should be presented to the user:

3.5 Data Formats to be Searched

Users should be able to access all data formats available via the search facility.

The search facility should be flexible enough to accommodate new data formats as they emerge.

3.6 Currency

Users should be able to access to current information as soon as it is made available. Currency of the search facility and the data included would be assured by the inclusion of:

3.7 Quality of the Data

Quality and authority of the information indexed must be assured. The quality of data supplied to users via the search facility could be maintained in the following ways:

By the whole-of-Australian government information search facility through:

By individual Australian government site administrators through:

3.8 Language

The search facility must be presented in accessible language. Languages other than English represented in Australian government information sources should be accommodated by:

English is the first priority language for the search facility.

3.9 Scalability

The search facility must be able to cope with the range and volume of resources made available by Australian government agencies. To this end, the facility must:

3.10 Help Support

Provision must be made for online help support which would include:

4. Search Architecture

4.1 The Model

The SEWG recommends a three-tier model in which the user, or client, accesses the resources created by the various agencies with the assistance of mediating software described as the middle layer.

The middle layer assists distributed searching across multiple sites by providing value-added functions such as query expansion (using thesauri), query refinement or filtering, and query routing or pre-selecting target sources tailored to a user query. Reliance on site metadata in conjunction with functional directories, such as the ‘Keyword AAA Thesaurus’ devised by Australian Archives, raises the problem of interpretation and translation of terms (ie the bureaucratic terminology of government may not have meaning for average users and the application of terms within different agencies can have very different meaning).

The use of thesauri to filter queries to specific search engines may also be problematic because of the reliance upon software to make value judgements about sites which may be applying thesaurus terms differently or using different thesauri.

The middle layer is intended to deal with the scenario of wide area searching over target sites of unpredictable size, as with the Internet environment. This may be contrasted with the searching of large databases over proprietary networks using-defined data sources, for registered users.

Figure A represents such a distributed search model. The agency layer, or the resource creator, is loosely coupled with the user or resource access layer (usually provided via a WWW browser). Between these two is the middle layer which mediates to match user requirements with the available services and resources.

The agency layer can comprise a single layer or multiple sub-layers as in Figure B. Consortia which serve as subject entry points (eg EdNA, DPIE and ERIN) are an example of the latter category.

The advantage of the proposed three-tier schema is that it allows a high degree of autonomy to an agency in terms of its data creation and information management mechanisms, while allowing its information to be visible and searchable with that of other agencies as part of a general search architecture.

Part of this autonomy involves the acceptance of responsibility on behalf of an agency for the creation of its own resource descriptions and metadata. It is therefore assumed in this model that resource description or metadata creation is part of the agency layer (Figure C), which is in keeping with the provisional recommendations of the AGLS Working Group.

4.2 Technical Issues

4.2.1 Indexing in Relation to the Three-Tier Model

The indexing of Australian government web sites is currently performed in a variety of ways. Some agencies create their own indexes and deploy their own search engines, some agencies are indexed by a consortium they belong to, and for some other agencies the only indexing is done through public search engines such as AltaVista or Anzwers.

In other words, the indexing function is currently carried out in different layers of the three-tier model. This issue was examined at length by SEWG and some alternative options were considered, which are discussed below.

4.2.2 Responsibility for the Indexing Function

The SEWG considered the issue of how to distribute the search and indexing functions between the agency layer and other layers such as the middle layer or a common indexing and searching service for Australian Government information.

While the three-tier architecture is seen as the best approach for the medium and long-term, the SEWG recognises that a shared approach may be appropriate in the short-term. The shared approach would involve creation of AGLS metadata by agencies or consortia in the agency layer. This metadata would be gathered by a cross-agency mechanism into a single repository, and accessed through a cross-agency search engine.

By contrast, the distributed approach would involve creation and indexing of AGLS metadata by agencies or consortia in the agency layer. A central (single, or later distributed) over-arching search mechanism might then utilise the agency indexes by invoking the respective agency or consortia search engines. Result sets from these individual search engines could be merged into a coherent result set by the central whole-of-government search facility.

The distributed approach would rely on significant standards and software development. It is essentially a parallel searching mechanism that is able take a user query in a single query syntax, translate it to multiple syntaxes of the underlying (agency) search engines in question, and translate the results (and the relevance rank values) back into a coherent result set. This demands that query and result syntaxes, and the ranking algorithms of each individual search engine, be made known to the whole-of-government search facility.

4.2.3 Scalability

SEWG also discussed the issue of scalability, both in terms of the total size of the search domain and in terms of the range of formats and data types. One of the key reasons for a three-tier architecture (as opposed to a closely coupled architecture without the intervening middle layer), is to accommodate what is likely to be very significant growth in the total number of government web sites and volume of data.

The SEWG estimates that about 5,000 Commonwealth, State and local government agency sites could be expected. However, the key issue is the flexibility of the architecture to accommodate increases in the scale of the resources covered.

4.2.4 Language

The SEWG considered the possible need for multilingual presentation of information. Current browsers are capable of supporting multilingual and multiscript services: for instance, Java has UNICODE support. However, on the government scene there is very little data stored in multiple languages at the server end. The creation of multilingual data is an issue for data providers. The user interface aspect of this issue will be dealt with by the Consultancy referred to in Section 1.4.

Footnotes:

  1. This position was endorsed by the Commonwealth State Navigation Working Party on 6 January 1998 and is presented here as the national position.
  2. Formerly known as the AusGILS Working Group.
  3. For indexing/gathering/searching purposes.


Return to the Index

Last updated 13 January 1998