Staff Papers

Indexing APAIS and AMI: Implications of the National Document and Information Service

Paper by Sandra Henderson, Manager, National Bibliographic Publications presented at the 1st International Conference 'Indexers: Partners in Publishing', Marysville, Victoria, 31 March - 2 April 1995.


Although the focus of this paper is the NDIS and what it will offer our indexing services, it may help a little if I start by describing the current systems used for creation of the APAIS and AMI databases.

APAIS is an index to Australian scholarly periodicals in the social sciences and humanities, while AMI is an index to health and medical periodical literature in this country. Both are produced in the National Bibliographic Publications section within the National Library of Australia.

The records, as you can see in the examples, bear a striking similarity to the records from any periodical indexing service.

                                                                       
APAIS                              AMI                                 
 ISSUE  = 9407                     AU - Joshua DE                      
AC  A94076692                      TI - Current perspectives in        
IN  9407                                myeloma.                       
TI  Electronic developments in     AB - The author looks at            
    Australian state libraries          chromosomal changes, bone      
CO  tables                              disease, hypercalcaemia,       
AU  AWCOCK, Frances                     prognosis, factors affecting   
SO  Australian Library Review,          tumour proliferation, host-    
    v.11.no.1,Feb 1994: (59)-66         tumour interactions, renal     
MI  Acquisitions (Libraries);           function, and treatment of     
    Cataloguing; Laser discs;           myeloma.(non-author abstract)  
    Information retrieval;         MH - Bone Diseases/COMPLICATIONS;   
    Library circulation;                Bone Marrow Transplantation;   
    Computer networks                   Human; Hypercalcemia/          
MI  Libraries, state; Computer          COMPLICATIONS; *Multiple       
    applications                        Myeloma/COMPLICATIONS          
ID  Indexing; Video discs; CD-          /GENETICS/THERAPY ;Prognosis   
    ROM; Collection development;   AD - Haematology Dept., Royal       
    Local area networks (LANS)          Prince Alfred Hospital,        
                                        Sydney                         
                                   SO - Fellowship Affairs 1994        
                                        Nov;13(4):25-6,31-2            

APAIS has been produced in the National Library since 1945, and the current processing system was installed in the 1970s. It was imported from the British Library, and uses mainframe-based word processor and publishing software. The software is unfriendly, is little used within the Library (therefore systems support is difficult to maintain), and modifications are almost impossible.

APAIS incorporates 11,000 - 12,000 citations per year. These are drawn from the National Library's intake of monographs, serials and newspapers. Material for indexing is selected by a librarian, and an indexing sheet is completed for each citation, with appropriate citation details handwritten onto each form. The material is batched, and taken away by our contract indexers. All database indexing is now done by contractors, rather than by Library staff.

The contractors complete the indexing form, assigning thesaurus and other subject terms, checking personal and corporate names against ABN and the existing APAIS file, and transcribing details onto the sheet as necessary. These are returned to the Library, and the data entry undertaken as a separate task. There is no automatic data validation, and proofreading is a tedious and time-consuming task. Data correction of the master word processor file is possible, but because it is not closely linked to the database system by which APAIS is made available to users, it is not possible to make retrospective corrections to errors which find their way into the databases (Ozline, AUSINET and AUSTROM).

The situation is a little better for AMI, although we still rely on handwritten data sheets for each entry. Validation and data correction procedures are more advanced, but no facilities for batch entry (for example from diskette) exist.

The NDIS

In late 1993, the National Library of Australia and the National Library of New Zealand issued, to a short list of tenderers, a Request for Tender for a redevelopment of the current ABN, Ozline, NZBN and Kiwinet services, with improved and expanded functionality. The new system (unnamed at the time of writing, but referred to by the project name NDIS - National Document and Information Service), will offer not just replacement of the current online services, but will, additionally, provide us with sophisticated data creation and maintenance facilities, as well as output of products to meet varying publication needs.

In 1994 the contract for development of the system was awarded to CSC Australia, who are now working with a team from both libraries, in the early stages of system development.

Some of you may have seen the Request for Tender document, or attended some of the meetings which have been held around Australia to introduce ABN and Ozline users to the concepts being addressed in the NDIS. The detailed analysis phase of the project has resulted in some amendments to the requirements listed in that document, but it is largely unchanged. For those of us creating databases, the changes promised by NDIS will completely revolutionise the way we go about our work.

The NDIS will be based on Oracle software, with client-server functionality, and will run on a massively parallel IBM SP2 computing system. Development will take place over the next 2 years. There will be opportunities for users and others to comment on various aspects of the development at different stages during the project- for example, sample screens will be on show at the ABN/Ozline Annual User meeting in Melbourne in July, and small groups of experts are being formed to review certain parts of the system. There will be a group looking specifically at the indexing functions.

The goal of the NDIS is

"To facilitate nationwide access to documents held by libraries and other information providers in Australia, New Zealand and overseas."

This is a major contributor to the stated strategic directions of both libraries for the next 5 - 10 years. One of the strategic directions given in the RFT document is

"to improve access to information about Australia and New Zealand through electronic means, including indexing and imaging"

This paper concentrates on some of the specific requirements listed in the Request for Tender (RFT) document, and what they will mean for the production of the APAIS and AMI indexes. The intellectual work of choosing thesaurus terms to accurately reflect the intentions of the authors, and selecting the appropriate articles for indexing within APAIS will alter, but the more tedious work will be revolutionised.

Data loading

RFT 5.3.3
(a) will support the intake of resource details continuously throughout the day without significantly affecting online response times
(b) will support the intake of resource details delivered via magnetic media
(c) will support the intake of resource details uploaded from an external source

Data loading for both databases will be more flexible. At present the only option open to us is data entry inhouse, into the separate data creation systems for APAIS and AMI. This makes it impossible to ask our indexers, who all work from home, to do with this task, so they have to handwrite the data forms. There is no facility for batch loading, so we cannot accept diskettes of records into either APAIS or AMI. Not only would our contractors be able to index online, or submit indexing on diskettes, but the possibility of accepting indexing from other sources becomes possible, and I will mention this at the end of the paper.

Data conversion

RFT 5.3.3
(d) will provide a ...facility to support the definition to the system of conversion routines to the NDIS internal format for incoming resource details which are:
(i) in a standard bibliographic format based on MARC;
(ii) tagged and delimited or comma delimited; or
(iii) marked up using SGML

Although at this stage the other data providers to the Ozline service do not intend to index online, the RFT specifies a number of standard input formats, designed to cater for a range of incoming data. In most cases the data for other databases comes to the Library on diskette. The range of formats to be supported by the conversion routines should avoid the situation we have at present, where every new database coming to Ozline requires separate conversion routines.

Input facilities and data validation

RFT 5.3.4

The Input/Validate process will:

  • provide users ... with access to customised workforms...and sophisticated text editing facilities
  • identify, reject or correct and report on invalid data conditions, providing instant feedback to online users and diagnostic reports to offline users

RFT 5.2.6

The NDIS will validate all incoming data according to sets of rules developed for each... Validation may include spell checking, a check for the presence of specified elements, processing against edit rules and so on.

Data validation routines should greatly cut the time needed presently for APAIS proofreading and editing. APAIS descriptors and source journal details will be checked automatically, routines will be set up to validate the format, if not the content, of other fields, and we are very much attracted by the idea of a spell checker based on previous entries in the database, which is already used by Index New Zealand. Apart from spell checking of free text fields, adequate validations already exist for AMI, so we will attempt to duplicate most of those checks in the new system.

Thesauri

RFT 5.2.4

(c) (ii) will support the following thesauri:

  • Medical Subject Headings (MeSH)
  • the dictionary from the INNZ input system
  • the APAIS Thesaurus (for use by both APAIS and INNZ)

The thesauri used, the APAIS Thesaurus and the Medical Subject Headings (MeSH) will be installed in the new system, to assist both the indexing operations, and searchers of the databases. Our current experience with online MeSH as an indexing aid and search assistant allows us to predict that this will result in improved productivity from indexers, and more flexible searching.

Text editing

RFT 5.3.5

(a) must allow authorised users to create, maintain or delete data online as part an NDIS session (ii) and (iii) must be able to retrieve and copy a full record or selected details into a workform to be updated...or to form the basis of a new record

(c) must provide a screen editor ...with standard text processing facilities...

The RFT allows for definition of input workforms for each database, and the kind of text editing facilities we have become familiar with in PC-based word processors. Such features as "cut-and-paste", copying of workforms to allow for new records which are substantially the same as previous ones, and copying of data which remains constant for a group of records will all make the job easier and help eliminate errors.

Global change

RFT 5.3.13
The system will provide NDIS managers with a facility for globally reprocessing existing text... Data will need to be changed to accommodate changes to attribute sets, validation rules or standards or to improve the quality of data already loaded.

Closely allied to this is the requirement for a global change facility. Not only will this make it easier to rectify any common faults in incoming data, but would allow us to undertake some cleanups of the older data in APAIS. Since the data creation software and the APAIS databases are completely independent, errors detected after loading of data into the database cannot be corrected, and modifications to groups of records, such as changes to thesaurus terms or names, cannot be undertaken. With NDIS providing us with a data creation process directly linked to the public view of the data in Ozline, we will be able to both correct and improve records, and, where it is deemed necessary, add or delete information in many records with minimal effort. For example, the addition of ISSNs to APAIS has been on our agenda for some time, and has not been feasible in the existing system, but we will be able to make such large scale amendments in NDIS.

Other major requirements include a statement about data matching, to exclude duplicates. This is being eagerly awaited by the cataloguing users of our services, and will have some application in the databases, where we are aware some duplicate records exist as a result of occasional failures in our serials records information.

Links to document images is a venture we are already making some progress toward, in preparation for the NDIS. Images of all documents indexed for APAIS from the January 1995 issue of the index are being captured, in anticipation of the setting up of these links. In time other Ozline databases could be included in this project, with AMI the next under consideration. The difficulties faced in this development are varied. Apart from technical issues such as image quality, compression, storage and access, there have been prolonged discussions with the Copyright Agency Ltd about the legal aspects. Not all issues have been satisfactorily resolved at this stage - we are yet to tackle the more difficult technical and legal problems associated with newspaper articles, and some material has proved impossible to copy to an acceptable standard.

The search side of the new system will be the most visible feature to many users, particularly in the early stages of the phased implementation. Further information about this is available in documents available from the Library's web server, but the improved search facilities will also prove useful for indexers and cataloguers, who often need to locate existing information on the database to copy or verify details.

One requirement that is of great interest to us, since APAIS is one of the flagship bibliographic publications of the National Library, is the output of records in formats appropriate for publication. This has been the subject of much discussion, as we try to firm up the requirements here, because not only do we want to produce a product much the same as the one we already publish, we want the option to produce more flexible products in the future - perhaps bibliographic products specified for individuals or groups of users, produced for electronic downloading in SGML format for printing at the requester's end.

Linking of authority information to bibliographic records, is important in maintenance of the databases. Those of you familiar with MeSH would know hundreds of new terms are introduced each year, and many more change slightly. We will want to be able to load a new version of MeSH, and have the changes in terms apply to all records which include those terms, in the same way that the National Library of Medicine does for its MEDLINE database on which AMI is modelled. The changes to the APAIS Thesaurus are less numerous, but we are embarking on a process of user consultation about the suitability of terms in specific subject areas, and this will lead to some thesaurus changes, which we will want to reflect in the database.

What are the main improvements the users of our indexes will see? Apart from the ones which will apply to all users of the NDIS, such as cross-database searching, we are hoping the system will lead to improved database currency. When we surveyed subscribers to the printed APAIS two years ago, a fairly common complaint was the time taken for journals to be indexed, and for the index to appear. Although we have taken steps recently to improve currency by streamlining the proofreading and editing stages, it is still an area of some concern and one where there are improvements to be made when we move to a more efficient system. The changes at both the indexing, editing and output stages should overcome many of these problems.

The validation procedures will lower the error rate generally, as well as saving us time. Staff time that is now devoted to checking the spelling of thesaurus terms, can go instead to improved thesaurus revision, by relieving us of some of the more tedious work.

The availability of images is seen as a crucial factor in future marketing of our database services. Users are no longer content with lists of citations (were they ever?), but the technology now exists to allow us to create links from the citations to the document images to allow the users to retrieve immediately items of interest.

While there have been some advances in the last few years in terms of output options, NDIS will provide a range of flexible output, both in terms of data format and content, and the method of delivery.

Finally there is the issue of contributed data to our databases. For years now libraries all around Australia, and New Zealand, have contributed data to build the national bibliographic databases in each country, and have been rewarded via a system of credits. This has not been without problems, as all ABN users would realise - duplicate records and some lesser quality records being just two of these, with a large scale Quality Improvement Program addressing some of the problem areas in recent years.

Accepting outside contributions of data to the citation databases has not been tried. While there has not been software in place which would allow it on a wide scale as happens in ABN, it has been other issues which have dominated our thinking. Issues such as quality and consistency of indexing have been uppermost in our deliberations. With NDIS, and the fact that cataloguing and indexing will be side-by-side operations on the database, we will be taking a closer look at what the costs and benefits there might be in accepting data from other sources. Obviously there are libraries who are better qualified than ourselves to index specialist journals in some cases, and in some cases they are handling the same journals for other database indexing projects. There has always been material we would have liked to include in APAIS but have not had the resources to index. If authorities are online, and items can be tagged for central review (as they are in ABN), the hurdles may not be insurmountable as in the past. It is not inconceivable that a number of libraries will become contributors to APAIS, with a specified set of titles to index, and that they will receive some recompense for their contributions.

Top

About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us