Go to National Library of Australia Home PageNational Library of Australia Online

SEARCH:

Government Access Architecture Workshop

23-24 September 1996, National Library of Australia, Canberra
Detailed Discussions and Outcomes


The discussions below have been aligned to the questions set out before in this document.

Question 1 - Metasearcher

As mentioned above, opinion was divided on whether a metasearcher option ought to be adopted in the short term as a mechanism for a whole-of-Government search arrangement.

In broad terms, a metasearcher is defined as a mechanism that provides a unified interface to, and a uniform view of, multiple document sources even if these sources use their own disparate internal search engines. Specific features attributed to a metasearcher vary, depending on which definition is used. For the purposes of the Workshop, the definition used was that adopted by the STARTS initiative. By this definition, the attributes required of a metasearcher include:

  • an ability to accept a single query syntax for multiple document sources, and to merge the query results from these sources;
  • an ability to evaluate query results at these sources; and
  • an ability to choose the best sources for a query.

While the first attribute may be achieved partially without the incorporation of any technical metadata, today, as with the NLA pilot metasearcher demonstrated at the Workshop, it does call for technical metadata (e.g. query syntax and result syntax of search engines) for a fuller exploitation of the capabilities of individual search engines and for a more meaningful and complete merger of multiple result sets.

The second and the third attributes can not be achieved at all without the establishment of some of the technical metadata standards outlined above.

The arguments in favour of a Government metasearcher in the short term were:

  • that some results could be immediately available, with existing tools, and therefore at minimal additional costs; and
  • that for many agencies that are at relatively early stages of Internet efforts, such tools could serve to demonstrate the benefits of, and clarify some of the issues in, the use of a single search mechanism over a broad-range of disparate data sources. This could in turn assist in further formulation of requirements.

The main arguments against a Government metasearcher approach in the short term were:

  • that in the absence of appropriate technical metadata to assist in merging and ranking, the search outputs would become unusable and prone to 'noise', especially as the number of target databases increases; and
  • that a central deployment of metasearcher will be problematic as its performance is likely to decline with rising size and numbers of target databases.

However, there was more of an agreement in terms of a medium term vision of a metasearcher model. This model was envisaged as a distributed metasearcher model, with multiple sites of implementation, with the 'middle layer' (as defined in the Summary above) acting as a filter, based on sets of metadata, to identify the logical areas within government for further in depth searching. Such metadata sets would need to have elements that would allow searching via:

  • a subject approach
  • a functions/agency approach
  • a collections approach.

In addition to metadata sets, thesauri would be employed in the filter layer. Two thesauri that could be used are the Australian Archives Functions Thesaurus that monitors what functions agencies are responsible for over time and the Triple A Thesaurus which is being nominated for use across government. Once the 'filter' has identified relevant sites for further detailed searching, the query would be addressed by parallel searching by a metasearcher, on the selected sites.

The advantage of such a filter based metasearcher, as opposed to a native metasearcher or metasearcher without filters, is the ability to return smaller and more focussed retrieval sets to the user, and to achieve network efficiencies by reducing avoidable traffic.

It is to be noted however, that while the filter model provides a better logical design than a native metasearcher, the physical deployment issues are the same in both cases, in particular, a central deployment of the filter would create performance bottleneck and a single point of failure situation.

While no consensus was reached on whether or not to deploy a centrally coordinated Government metasearcher in the short term, a number of agencies expressed interest in obtaining copies of the pilot metasearcher demonstrated at the Workshop with a view to adopting it as an immediate starting point for a search strategy within their own agencies. Some data/service providers - such as AustLII - expressed willingness to provide metadata information regarding their query and results syntaxes, for incorporation into metasearchers, and a number of organisations espoused the view that a lead agency approach should be adopted to take forward and examine solutions in the arena of metasearchers and related activities.

On balance therefore, an optimal approach for the short term could be to make the NLA metasearcher pilot software publicly available through a self-help Web site and to leave the decision to individual agencies as to whether to start using it now or to wait for the implementation of a filter model.

In this scenario, no follow-up support will be undertaken by the originating agency, i.e. the NLA, due to the temporary nature of the arrangement. But interested agencies with appropriate IT resources will be free to instal, run and maintain the software on their own hosts.

This approach would also pre-empt some of the potential performance bottlenecks of a central implementation. In parallel with this, a preferred option would be to have a lead agency nominated throught the IMSC or an equivalent process, to carry out further examinations and trials and to catalyse future implementation processes.

* Recommendations *

  • that in the immediate short term no central whole-of-government metasearcher be implemented, but instead, the existing pilot metasearcher software demonstrated at the Workshop be made publicly available by the NLA on a non-maintenance basis, for interested agencies to instal and run on their own hosts should they so choose;
  • that concurrent with the above, a lead agency be nominated through IMSC or an equivalent mechanism to progress things further in this area. The responsibilities of the lead agency would include:
    • a lead/coordination role in developing the 'filter' layer of subject, function/agency and collection information for whole-of-government data;
    • a role to track developments in technical metadata standards such as STARTS and the uptake of such standards in major search tools/services, and to enhance the present pilot metasearcher, or equivalent, accordingly;
    • a role to liaise with Australian search engine and service providers to seek metadata specific to their environments, and to incorporate these in the pilot metasearcher, or equivalent;
    • a role to report to an apex body such as the Office of Government Information Technology, on the progress of the above processes, and on new emerging options and their implications on whole-of-government access mechanisms.

Question 2 - Metadata

It was generally agreed that if thousands of government sites were available search results would be more accurate and relevant if a form of metadata was employed. Discussions covered collection level metadata, document level metadata, agency and function specific metadata, and mechanisms of metadata production.

Metadata at collection level

Collection metadata describes collections of resources. Basically, it identifies the theme of a collection, and was considered of the highest priority in the present context, as a government department may contain several collections and this is where visibility would be of most importance.

Points discussed included:

  • role and value of metadata in making government information collections more visible
  • role and value of Government Information Locator System (GILS)
  • Metadata sets currently available
  • AusGILS (an 'Australianised' version of GILS, specified in the IMSC-TG report), and how AusGILS would be used in the 'filter' layer

* Recommendations at collection level *

  • that metadata is required to describe and hence make visible government information at collection level;
  • that an appropriate metadata set called AusGILS be developed as recommended in the IMSC-TG report;
  • that AusGILS be adopted by all agencies so that standard descriptions of government collections are available;

Metadata at document level

In this context the word document encompasses text as well as other types of information relevant to government activities i.e. tables, images and other multimedia sources.

Document metadata describes the document and may identify, for example, the author, title, and subject of the document.

Points discussed included:

  • whether there was a role and value in the use of metadata at the document level, in terms of effectiveness of government information retrieval;
  • adoption of a suitable core of document metadata in the Australian Government context. It was anticipated that such a core would bear similarities to other general purpose document level metadata sets such as Dublin Core, but could differ from these as needed to suit the context. One of the suggestions was that this core be referred to as the 'Parkes Core', a name to be used in the rest of the present document.

Discussions also included whether metadata at document level would be best embedded or kept as a separate metadata database. This issue will be taken up under the Question - Metadata Storage.

It was generally agreed that in addition to needing metadata at the collection level, which would assist in the location of relevant sites for further in depth searching, quality of retrieval of documents across government would be enhanced if standard metadata descriptions were associated with the documents.

It was acknowledged that agencies at present have specific metadata sets, thesauri headings, subject terms etc that they assign to documents, and these would remain valid metadata options for assisting an agency site search. An example is the recently adopted geospatial metadata set for ANZLIC (Australia New Zealand Land Information Council) databases.

It was generally agreed that a minimal set of standard metadata - Parkes Core - will be developed, and agencies will be encouraged to add at least this minimal set of metadata to each document they own. This core would exist in addition to agency specific metadata, and would assist in retrieval of documents across multiple government agency sites.

The Workshop preferred that such a standard metadata core - Parkes Core - be treated as highly desirable as opposed to mandatory, and that accordingly, agencies be alerted to the value of its use. It was recognised that it would be an agency decision whether they assigned standard metadata at document level or not. Agencies would presumably consider this a worthwhile investment for important documents or ones for which they wanted users to have accurate retrieval.

* Recommendation at document level *

  • that OGIT develop guidelines outlining an optimum metadata set for use at document level by government agencies should they wish to do so.

Other Metadata issues

Other metadata issues discussed included the role and need for agency specific metadata and function specific metadata. The Workshop felt that there is a need to see if - via collections of metadata - we can support agency autonomy and effective whole-of-goverment searching/publishing.

The Workshop also discussed the place for intellectually assigned metadata versus automatically assigned metadata. It was acknowledged that tools will be needed to support the user in semi-automatically assigning metadata to docuemnts. This will have dependancies on the publishing system used within each agency, which will need to be taken into account.

Question 3 - Metadata storage

Metadata storage issues were discussed both at collections level and document level. It was acknowledged that while at the collection level, metadata would almost of necessity have to be stored separate from the data in question, at the document level, there will be a mixture of storage scenarios in the long run. Some documents will have metadata stored with them as this ensures that the metadata is copied with the document. For such documents, encoding rules for different formats - such as HTML - will need to be adopted or developed. Some document formats - e.g. pictures - on the other hand will possibly have metadata stored in separate databases. It implies that there will be different mechanisms for harvesting, identifying and managing metadata for different document formats.

At the document level, metadata can be implemented with two different approaches and is dependent on the agencies' resources and policies. The first is as a separate database containing the metadata records with pointers to the actual documents (either physical or electronic). The database should support a common information retrieval protocol to access the records (such as Z39.50). The second option is to embed the metadata into the document. This is appropriate for HTML files using the META tag. While the second option could be simpler to implement and manage, it has the disadvantage of not being as flexible as the first option in terms of migration or transfer to other sites etc.

* Recommendations for Metadata storage at collection level *

  • that the Australian Archives develop an AusGILS template for agencies to use in creating AusGILS locator records;
  • that agencies create and maintain the AusGILS locator records on their own web servers or on their host agencies' web servers.
  • that AusGILS locator records use SGML syntax;
  • that locator records be made available as SGML marked up text files (one for each collection) and be available in the top-level 'AusGILS' directory on the agencies' WWW server.

* Recommendations for Metadata storage at document level *

  • that the method of storage of document metadata be left to the individual agencies’ discretion, compatible with any data interchange and indexing requirements at a whole-of-government level;
  • that a lead agency be nominated through IMSC or an equivalent mechanism to develop and adopt procedures regarding harvesting, identifying and managing these different forms of metadata for use at a whole-of-goverment level, in keeping with international developments

Questions 4 and 5 - Logistics and siting of Indices, and Full-text indices

As with the Metasearcher issue (Question 1), issues in indexing had consensus in some areas and differences in some other areas.

The principal area of agreement was that in the longer term there would be no full-text indexing at a whole-of-government level, though it was acknowledged that individual agencies could choose to have full-text indices if they so chose. Previously it has been recommended that agencies produce metadata at collection level in the form of AusGILS locator records. A case for metadata at document level has also been presented. Additionally it was asked if there was value in indexing the full text of government documents available through the Net. This was not seen as a viable long term option. An index of full text terms collected from across all government sites would become too large and possibly result in imprecise retrieval sets. Retrieval time would also be affected and excessive network traffic generated. Hence it was not recommended.

However, there was divergence in terms of opinion on short term government-wide indexing architecture. It was generally recognised that the development of a metadata based government-wide architecture outlined above would take at least 9 to 12 months to implement and would involve reaching agreements with agencies prior to implementation. It was seen as a longer term solution and hence an immediate solution was also discussed. There were two main opinions:

  • one opinion was that as an interim measure, a web search engine should be set up specifically for indexing Australian Government Web pages, with or without document metadata, i.e. it would index on the basis of document metadata where such metadata is available, or alternatively, on full text documents where document metadata is not available;
  • the other opinion was that it would be a better use of collective efforts at this stage to focus on developing and implementing the recommended metadata structure, so as to set the foundation for a proper distributed indexing architecture in future, based on multiple gatherers, brokers and indexers interchanging metadata information in an agreed format. According to this opinion, the short term indexing requirement could be adequately served by worldwide indexing services such as Alta Vista that are currently indexing Australian Government material anyway.

Though a resolution could not be reached between the above two opinions within the available time at the Workshop, it was generally agreed that both approaches have their own disadvantages and advantages.

The disadvantages are that: a central indexing model would have the same scalability problems as a central Metasearcher model (Question 1); and a free indexing service such as Alta Vista would have the disadvantage that the government would have little control over the indexing process, e.g. over ranking of government pages or over how often or when the indexing information was being updated.

The relative advantages are that a (temporary) central indexing model could offer an opportunity to analyse and map Government holdings, thus facilitating the process of establishing the filter layer recommended earlier (Question 1), comprising subject, function and collection information. The advantage of a (temporary) wordwide indexing approach is that as the entire mechanism is transparent to the Government, the efforts of the Government can be fully devoted to metadescriptions and their logistic issues, and to the subsequent long term access architecture/mechanisms. Also, such worldwide indexing services (e.g. Alta Vista) are now beginning to support some of the META tag based metadata and other metadata conventions promulgated by initiatives such as STARTS, and could therefore be expected to offer a gradually improving quality of retrieval.

A suggestion was made by the Workshop that a nominated lead agency maintain and manage both the 'single government entry site' as proposed by the IMSC and the central index created by the search engine, should a central indexing model be adopted.

It could be mentioned in this context that the NLA had, until recently, been carrying out an experimental gathering and indexing exercise on Australian government domain information, using Harvest software. Should the central indexing approach be adopted as a short term whole-of-government indexing option, it could be worth considering a resumption of this exercise with Harvet or other suitable tool with an appropriate re-focussing on data.

The Workshop also noted that a risk in adopting a short term central indexing approach is that it may deter agencies from expending the effort in being more involved, an aspect that will need to be addressed in the collaborative liaison mechanisms.

On the issue of full text indexing, there was agreement that it was a local decision within agencies as to whether or not agencies would use full text indexing tools on their Web sites. It was anticapated that many agencies need such a capacity. From a whole-of government point of view, full text indexing was not considered essential. Agencies could choose to provide full text searching if resources allowed and it was appropriate to do so.

On other agency level indexing, the Workshop agreed that agencies would index the AusGILS metadata created at collection level, and would register the URL of their AusGILS with a lead agency, with a standard naming convention such as http://www.tax.gov.au/ausgils/. For document level indices, agencies would have the discretion of providing whole indices or summaries of indices.

* Recommendations *

  • that for a longer term indexing strategy, a lead (multiple) agency approach be adopted for the purposes of AusGILS registration of individual agencies and for high level visibility of meta-information on the basis of which a scalable distributed indexing and searching mechanisms can eventually be put in place;
  • that for a shorter term indexing strategy, IMSC or an equivalent mechanism nominate a lead agency that is to liaise with key players and decide upon the more cost effective of the two options discussed above for creation of Government indices in the immediate future;
  • that individual agencies be encouraged to index and maintain, either on their own server or on server of their chosen host agency, at least their collection level metadata or AusGILS records, and preferably some summary information on their document level indices; and
  • that a lead agency be nominated through IMSC or an equivalent mechanism to assist in technical aspects of the above recommendation.

Question 6 - Persistent Locators of resources

Discussions iscluded description of what URL's, PURL's (Persistent URL's) and URN'S (Uniform Resource Nmaes) are, and to what level will URL's be assigned for government information.

The conclusion was that it would not be possible to have central management of PURL's government wide. This would be better managed at agency level. Relatively permanent addresses for documents were considered desirable. Of the options available, some form of PURL was considered achievable in the shorter term.

There was discussion of the problem of agency name changes, abolition and creation, which results in changes to domain name components of document addresses, but no solution was identified.

* Recommendation *

That OGIT develop guidelines for the syntax and structure of URLs to allow them to have a high level of persistence

Question 7 - Benefits

Points raised included: better government information; Open government; streamlining government; streamlining business; efficiencies to be gained; publishing costs reduced.

There are technological and policy reasons why governments need to reposition themselves with regard to information management and delivery. A business case can be made that outlines the advantages to government in improving public access to government information and which at the same time highlights the efficiencies gained in the use of better information management.

There was agreement that Information management is as much about the need to control information flow as to release it.

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us