Staff Papers

Digital Libraries: Some Implications for Government and Education from the Australian Development Experience

Paper presented by Eric Wainwright, Deputy Director-General, Singapore1996.


CONTENTS


Abstract

In the past 2 or 3 years there has been considerable progress in several countries towards realising the concept of the digital library, ie, a service through which collections of documents can be organised and retrieved remotely through online networks. These collections may be very varied, as 'document' has come to describe many types of digital objects, and there are considerable problems in developing successful searching techniques to search across very large heterogenous and distributed collections held in different formats.

It has not been generally understood that the concepts of the digital library are applicable to many other areas. As the World Wide Web becomes a universal means for wide-area inter- connection to information, governments, educational institutions and commercial enterprises are all seeking to present documents in a way that allows them to be visible and retrieved over the Internet.

This paper explores some of the implications of present developments, in the light of experience in developing national library services, and in developing information management strategies for the Australian government.

Introduction

The aim of this paper is to examine current trends in the development of digital libraries, with particular reference to Australia, and to explain how digital library concepts are important for wider developments on the Internet. Digital library solutions are likely to be at the heart of the successful development of access and retrieval systems for all large-scale purveyors of content.

Definition

First, we need to define what we mean by a "digital library". The term "digital library" appears now to be preferred for a concept which is also described variously as :

  • virtual library
  • electronic library
  • library without walls

These terms are essentially used interchangeably.

But the concept of the digital library needs more careful definition. In the term "digital library" are two important concepts :

  • Digital, ie, information in any digitised format
  • Library, ie, a total mechanism for obtaining access to, storing, organising and delivering information

A digital library remains a library, with the same purposes, functions and goals as a traditional library. The digital part of the term indicates merely that the material is stored and accessed digitally. A digital library is therefore far more than a digital collection, particularly a collection, as is illustrated in the overwhelming number of the World Wide Web sites on the Internet, which consists of relatively volatile current information.

One definition of a digital library is :

"A digital library is an assemblage of digital computing, storage, and communications machinery together with the content and software needed to reproduce, emulate, and extend the services provided by conventional libraries based on paper and other material means of collecting, cataloguing, finding, and disseminating information. A full service digital library must accomplish all essential services of traditional libraries and also exploit the well-known advantages of digital storage, searching, and communication". (Gladney, H.M. et al, 1994).

The organisations involved in the development of digital libraries have been ones who face considerable difficulties of storing, organising and accessing very large amounts of information, distinguishing between different versions of the information, and enabling long-term access to the material.

Typically these organisations now include libraries, archives (including sound and visual archives), and the records management sections of larger organisations. And most recently, the issue of digital libraries has started to become important to governments, as government agencies, somewhat belatedly in most cases, start to be serious about providing information over both internal "Intranets" and to each other and to the public over the Internet.

Digital Library Development

Digital library development is only very recent, and rests on our ability to fully replace the characteristics of analogue formats such as print. Print replacement has of course been predicted for at least four decades and has not occurred to any great extent until recently. This is not surprising - print replication cannot be expected to occur until :

  • Authors are all producing information in electronic form.
  • The electronic information produced can be accessed globally, regardless of the particular way the information is stored.
  • Communication networks have sufficient bandwidth and protocol standardisation for transmission of text and visual images at the rate required by users.
  • Potential users of the information have the ability to access it effectively.
  • Potential users of the information have the capacity to continue to store and manipulate the information locally.
  • Electronic information can be preserved over a long period.
  • Electronic text can be read over an extended period as easily as print.

Clearly we are some way from these conditions being fully achieved.

The realisation of digital libraries has rested on the affordable availability of several technologies :

  • Multi-level user interfaces (including GUI's)
  • Substantial work-station capabilities at user level
  • Fast retrieval from massive data stores
  • Mechanisms for linking related documents
  • National/global interoperability
  • Standard search/retrieve protocols
  • Transaction capabilities

all of which are now available.

Traditional libraries contain documents. Both traditional documents and the complex retrieval systems created by libraries contain mechanisms that link documents that have some characteristics in common, such as their author or subject. Any digital library must also be capable of providing such linkages between "like" documents. This is a complex task for any large library covering more than a narrow subject field. It is hardly surprising then that the hypertext capabilities of the World Wide Web have provided a catalyst for the growth of digital libraries, combined with the availability of the other technologies listed above. The World Wide Web, particularly as its client-server architecture starts to support the easy transmission of legacy data into a form which is searchable and deliverable across the Internet, now provides a serious means for developing digital libraries, in which digital documents can be located in many different places, yet be searchable through a single mechanism.

The word "document" may here connotate for many people too narrow a concept. I am using the word digital document to mean all document-like objects which are stored in digital format - these will include a wide variety of multi-media materials with audio and visual characteristics as well as text, but have nevertheless functions in common in acting as information delivery objects.

Digital library research and development has grown very rapidly since about 1990. One of the first large-scale experiments was the digitisation of the Spanish Archives relating to Spain's early colonisation of the Americas - the "Archivo General de Indias", conducted as a collaboration between the Spanish authorities and IBM in celebration of the sesquicentennial of Christopher Columbus' 1492 voyage to the Americas. The project involved microfilming vast tracts of the archives, and then converting the microfilm into digital form for ease of access.

But it was the establishment of the United States National Science Foundation's Digital Library Initiative, in conjunction with the National Aeronautic and Space Administration (NASA) and the Advanced Research Projects Agency (ARPA), that gave digital libraries a major boost. The scale of the six NSF projects, each with budgets of over $US 3.0M, has allowed exploration of some of the major issues confronting the establishment of digital libraries on a large scale. The six projects :

have included information in a wide variety of formats, not just textual documents but also visual resources, numerical and spatial data.

Just as important, the Digital Library Initiative signified the seriousness with which the United States was approaching the issue. There is little doubt that the initiative has drawn attention to the problems of large-scale research storage, organisation, dissemination and retrieval of digital objects. It has led to a very considerable number of conferences focussed on digital libraries from 1994 to the present, aimed at defining the elements that are likely to make digital library operations on a large scale successful. The initiative has also had ramifications in a number of other countries, particularly in Europe, where the European Union has responded with two major programs - the Telematics for Libraries Programme, and the Info 2000 Programme concentrating on multimedia. In the United Kingdom also, a major series of programs has been developed as a result of a review of university requirements under the Electronic Libraries Programme.

Nor has this research and development been confined to academic and government areas. IBM, for example, has had a major program of research and development concentrating on digital library concepts, which has led recently to the launch of its InfoMarket commercial development. InfoMarket provides a range of services to digital content providers who wish to receive payment for the distribution and use of their products, and rests on IBM's development of the cryptolope. Cryptolopes are essentially envelopes for digital content, which require a key, and thence automatic payment, for opening to use the enclosed digital information. The cryptolope and similar developments clearly offer a means for copyright holders to distribute information on a wide scale, without losing control of their copyright content.

IBM has analysed the elements needed to provide digital library services, viz :

  • Creation/capture
  • Storage
  • Access
  • Dissemination
  • Electronic rights management

and has developed strategies to address these (see http://www.software.ibm.com/is/dig-lib/).

Australian Digital Library Developments

In Australia, digital library developments have been on a smaller scale. Major current developments include :

  • Australian Cooperative Digitisation Project (http://www.nla.gov.au/ferg/ferg2.html)
  • Images 1 : the National Library of Australia's Documentary Image Collection (http://www.nla.gov.au/imagecoll/)
  • Indian Ocean Rim Region Virtual Library Project (http://www.cowan.edu.au/library/iorr/)
  • Italian-Australian Records Project (http://www.nla.gov.au/niac/meetings/npo95imo.html)
  • Our Story : Australia's Heritage Online (http://www.dca.gov.au/)
  • Scholarly Electronic Text and Image Service (http://www.library.usyd.edu.au/Ejournals/)
  • World 1 : National Document and Information Service (http://www.nla.gov.au/2/NDIS/)
  • Distributed Systems Technology Centre (DSTC). Resource Discovery Unit (http://dstc.edu.au/RDU/)

Most of these projects have concentrated on the problems of digitising existing text and visual representations, and have in some case built on earlier experiments in which analogue video disk technologies were used, as is the case of much of the material in the National Library of Australia's Images 1 service - now covering over 15,000 visual documentary history items illustrating people and places of importance in Australia's development.

The Australian Cooperative Digitisation Project 1840-1845 is a major project involving both preservation microfilming and digital scanning of the whole body of Australian fiction publication in the period, one of significance in the emergence of a separate Australian identity.

The Italian-Australian Records Project is an interesting, being supported by both the Australian and Italian governments, to record significant archival material relating to the Italian experience in Australia. Both it and the Australian Cooperative Digitisation Project comprise what are described as "hybrid" approaches, resulting in outputs both of microfilming for long term preservation purposes and also digital versions for current access.

Our Story : Australia's Online Heritage is a very recent government initiative, which will result in the digitisation of significant items in the Australian national cultural institutions, such as the National Library, National Museum, National Gallery and the National Film and Sound Archive - linking these digital collections seamlessly as part of the new National Cultural Network.

The National Library's major current development World 1 is aimed at building the full infrastructure needed to deal with the dissemination of both traditional and electronic documents from the network of the nation's libraries. It will commence service in 1997 and will provide a mechanism for searching and retrieving from a very large database of document metadata (over 15 million items), locating them in Australian libraries, and then ordering or directly accessing them over the Internet, depending on whether they are in print or digital form.

As well as these projects for the creation of digital library content from existing material, there are many projects concerned with the creation of new digital documents, particularly electronic journals. The Australian universities have a combined program of investigations into electronic publishing across a wide range of fields (see http://online.anu.edu.au/caul/elec-pub.htm).

Other research has concentrated on the problem of resource discovery in large distributed sources. The National Library has been working with a number of universities to investigate the use of Harvest (a distributed system for gathering and brokering information collected from different online sources) to index World Wide Web sites in the National Library, universities and some other government organisations. A key organisation is the Australian research and development effort relating to digital libraries is the Distributed Systems Technology Centre in Brisbane (http://dstc.edu.au/RDU/). DSTC has a wide range of projects relating to networked information discovery and retrieval, and has developed a number of products.

Australian Government Developments

The National Library has also been heavily involved within the Australian government in developing systems for effective access to Australian government online information. Most Australian government agencies now have some World Wide Web presence on the Internet, but many as yet do not have large quantities of online documents accessible. Those that do have come to realise that once the amount of information available becomes large, there are considerable problems in designing and maintaining the menu hierarchies, the links between different documents, and particularly the resource discovery systems which will allow relatively unsophisticated users good access to government information services.

In many ways, government, as an information source, may be regarded as one large digital library, in which the creation and storage of information is distributed over many agencies. The task for governments is to present their online services and search systems in ways which do not require people to know how the government is structured at any given point in time (government departments and agency structures tend to change regularly, particularly when there are changes of government). In Australia we have developed a conceptual architecture for access to the government as a digital library :

1 This architecture rests on the idea that all government information resources are of three basic types :

  • people
  • documents (including multimedia)
  • structured databases

The challenge for government is to develop a means for accessing this large digital library from anywhere in the country, through the development of a series of search engines which are capable of accessing stores of documents, databases and directories of people and agencies held in a variety of formats, and then aggregating the results of searches in a way that permits them to be communicated and presented through a single mechanism, ie, the Internet and World Wide Web browsers, at the users own convenience.

Current Problems for Digital Libraries

The Australian government's Information Management Steering Committee, which I chair, has identified a number of problems to be overcome if public access to government information is to be achieved.

Archiving/Long-Term Access

As with any large-scale digital libraries of material in a wide range of formats and on many topics, this is a non-trivial task. Because of the evidential and accountability requirements of government, government records have to be kept accessible and as far as possible free from later alteration over many years, as is the case with written and printed archives.

Most of the large investment in Internet development at present is aimed at the areas of information presentation, access and communication. Even the extended IBM effort is not concentrated principally on the longer term problems of preservation of electronic information. This is a key issue which distinguishes digital libraries from other collections of digital material, in which most of the information has a very short life. It seems very likely that digital libraries will adopt some of the newer information warehousing concepts for older and less used data access, while implementing the opportunities of the rapid Internet developments for improved access to more heavily used current data. But the longer term preservation of electronic information presents problems to which there are currently no total solutions. There is a general consensus that there is no current format for electronic storage that can be guaranteed to last as long as we have become accustomed to with printed documents, particularly those printed on acid-free paper. But the physical preservation task is in practice likely to be outweighed by the present absence, in most organisations, of effective procedures and processes to ensure the retention of copies of electronic documents, particularly where those documents exist in several versions. The ease with which digital documents can be deleted is likely to mean that we will have significant gaps in our historical record of the current and perhaps the next decade, while electronic library and records management systems continue to develop the facilities now required. This is of particular concern to organisations such as national libraries. As we start to create relatively large digital collections as part of Australia's national cultural network, we need to be sure that the digitisation effort is not to be dissipated by future problems of ensuring longevity of access.

The National Library of Australia has recently been involved with the Commission on Preservation and Access and the Research Libraries Group in the United States in a major study of these longer term problems, which has led to the report of the Task Force on Archiving of Digital Information (Commission, 1996). This is an excellent report on all the issues, and concludes that while the problems of migrating electronic documents from one format to another will be significant, equally significant are the problems of organisation on a national basis to ensure that important digital documents are archived successfully.

Legacy Data

For any major collection, successful digital library technologies are crucial to its future. The capabilities of the new information technologies, particularly as all these capabilities can be provided remotely over the Internet, offer all large content providers a tremendous opportunity to extend access and reach. Yet at the same time, this process automatically ensures the loss of local monopoly power - since unless an institution holds the only copy of a document, or owns and is able to exercise the copyright - any other institution in the world is able to provide the document concerned also. This presents existing collecting institutions with a major problem of choice in knowing what to digitise. Costs are still relatively high if high resolution delivery is to be required, and organisations need to consider carefully their strategy for making older data available.

Many existing documents are in some digital format; in particular, in various popular word-processing formats, such as Microsoft Word, and in database formats such as Oracle and SQL. The task for the digital library is to enable documents in all these formats to be searched through a single search mechanism and for them to be delivered over the Internet using the World Wide Web's http protocol. The National Library is working on this issue at present in conjunction with other government organisations. Major work has also been going on for some time in Canada through the Government Information Finder Technology (GIFT) project (http://www.gc.ca/), and the United States Government has recently commenced a major new investigation through its Advanced Search Facility Project.

Fortunately, there has also been considerable development in the marketplace, aimed at providing "plug-in" facilities to World Wide Web browsers that enable users to view documents in a variety of formats over the World Wide Web. Many publishers now provide documents in Adobe Acrobat's Portable Document Format (PDF) - Acrobat readers are becoming common-place, as are readers from Microsoft Word documents. But products now exist with multi-format reading capabilities, such as QuickView Plus allows access to over 200 document formats through a Web browser.

Resource Discovery

But transmitting and reading a document is only possible after it has been located. As the volume of material in a digital library, and on the Internet as a whole, becomes very large, then the problem of finding the "needle in the haystack" rapidly multiplies. The problem basically has four components :

  • Metadata, ie, a description of a document which enables that document to be indexed easily by automated indexing tools, or created easily by human beings so that the document can be found by people using natural language approaches. Librarians have, of course, done this for years, creating catalogues and indexes. But this is an expensive process. Unless documents have good quality metadata, then it is likely that they will not easily be found later. Attempts to simplify the metadata creation process so that it can be used more easily by authors are being made, particularly through the Dublin Core project, while metadata for the description of collections of documents or databases is being pursued in a number of countries through the Government Information Locator System (GILS) (Turner, 1996). However, inadequate metadata is at the heart of the failure of many search engines to produce the most suitable documents in return for a search.

  • Search Engines. Considerable advances have been made in search engines for the Internet, and there are now several of major power covering the whole Internet, such as AltaVista, InfoSeek, OpenText, Excite, etc. Impressive as the engines are in coverage, they are still relatively unsophisticated compared with the indexing tools created for professional information managers and librarians over the past 30 years, and rely more on breadth of coverage than their ability to target individual documents effectively.

  • Distributed Databases and Scaleability. It is proving a difficult task to design search engines and directory structures so that different databases can be searched as if they have the same search structure, and the results sets brought together for ease of use. An important development here is the Z39.50 standard for search and retrieval. Many library-oriented systems now have Z39.50 client and server capabilities, and Z39.50 capability is also built into both the United States and Canadian approaches to Government Information Locator Services (GILS) which is likely to be followed in Australia. Although World Wide Web to Z39.50 gateways exist, the complexity of the Z39.50 standard has so far deterred mainstream Internet developers from widespread implementation. Attention is now being given to whether a streamlined subset implementation "Z39.50 lite" can be more easily built in into the next generation of World Wide Web browsers.

  • Address Stability. Lastly, there are major problems with digital objects changing their addresses on the Internet. Every World Wide Web user knows the frustration of identifying a potentially useful source only to find out that the uniform resource locator which represents its address is no longer current. Much research is going on to develop a means of separating digital documents from their immediate locations, ie, to give each digital document a consistent name which may be associated with one or more URL's. One experiment is a resolver process developed by OCLC in which persistent uniform resource locators (PURLs) are created. Users seeking these documents do not then need to keep track of any location changes, which are recorded by the resolver. However, more fundamental solutions are needed, and work in the Internet Engineering Task Force and at the Corporation for National Research Initiatives in the U.S. in particular is directed towards various schemes for defining uniform resource names and other characteristics for digital documents.

Electronic Rights Management

Finally, there are the issues of ensuring that individuals and organisations are able to receive a fair return for the distribution and use of their intellectual property where they desire this.

The IBM cryptolope and similar products from companies such as Wave Systems, and techniques for giving digital documents a watermark or electronic signature which indicates their origin or ownership, are keys to the future operation not only of digital libraries but of the commercial dissemination of content generally across the Internet. These developments seem likely to lead to the realisation of "superdistribution" techniques. Superdistribution means the very wide dissemination of documents or other digital objects across the Internet, with a very small payment being made automatically for each use of the document. Such a distribution system is very different to present and traditional publishing frameworks, in which sales of books and journals are generally very low, with a relatively high price being charged for each work. What is unknown as yet is what combination of traditional and superdistribution methods will result in maximising revenue for copyright owners or for distributors of material out of copyright. The latter includes a large volume of material in major libraries, and it seems possible that as more of this material becomes digitised, superdistribution methods may well allow libraries to realise a return towards the costs of their investment in digitisation of older and lesser used, but still valuable, materials.

Conclusion

To sum up, digital libraries are now developing rapidly, and the tools being developed have a much wider implication than is generally realised - for government, and for any large organisation which requires a large amount of digital content to be searched successfully online. The major problems on which further research and development is needed, if such large collections are to be accessed over the long-term, have been described above.

But the challenge of what Clifford Lynch (1995) has called "deep semantic interoperability", ie, the ability of the user to access, consistently and coherently, similar classes of digital objects and services, distributed across heterogeneous repositories, with software compensating for site-by-site variations, remains. It is an extraordinarily difficult task, but one that has to be achieved if digital libraries are ever to reach their full potential, within the complex legal, social and economic structures that make up our society.


Eric Wainwright, Deputy Director-General, November 1996.
Last updated 9 July 1998

Back to the top