Paper by Warwick S. Cathro, National Library of Australia, for the Information Online & On Disc Conference, Sydney, 19-21 January 1999
Abstract
The
Development of the Digital Library
Digital Libraries
and National Libraries
The Selection Process
The Acquisition Process
The Control and Access Process
Controlling the Use of the Collection
The Preservation Process
The NLA's
Digital Services Project
Conclusions
Acknowledgements
References
A digital library, like any library, is a service which is based on principles of selection, acquisition, access, management and preservation, related to a specific client community. This paper examines some of the key challenges which these processes encounter when dealing with digital collections, with particular attention to the issues which are raised for national libraries. Examples are the challenge of selecting significant national digital publications, the challenge of how to acquire efficiently those digital publications which are selected, the challenge of integrating access to digital and traditional information resources, the challenge of ensuring reliable delivery of digital publications given their changeable physical location, and the enormous challenge of how to preserve digital publications. The paper refers to the National Library of Australias Digital Services Project, which has developed system requirements in the light of these issues and challenges.
The term digital library began to be heard in the early 1990s, as universities and other institutions began to build discipline-based collections of information resources in digital form, and to provide access to these collections through local and wide area networks. Today, hundreds of services which might qualify for the description digital library have been developed, and it is possible to survey what has been achieved by such services and what challenges have been identified.
The Digital Library Federation has proposed the following definition [1]:
Digital libraries are organizations that provide the resources, including the specialized staff, to select, structure, offer intellectual access to, interpret, distribute, preserve the integrity of, and ensure the persistence over time of collections of digital works so that they are readily and economically available for use a by a defined community or set of communities.
This definition emphasises that a digital library, like any library, is more than a mere aggregation of information resources: it is a service which is based on principles of selection, acquisition, access, management and preservation, related to a specific client community. All of these principles are relevant when we consider the meaning of a digital library and the practical issues involved in service delivery.
A digital library collection may include two types of information resource. One type comprises the digital original resources, which are sometimes referred to [2] as resources which are born digitally. The other type comprises digital surrogates, which are created from traditional information resources through format conversion. While both types of resource have the same access and management requirements, they raise different issues of selection and acquisition, and their preservation imperatives are also different.
Some definitions of digital library are broad enough to embrace services which integrate access to digital and traditional (e.g. print) materials [3]. In the interests of clarity of terminology, the use of the term digital library in this paper will be confined to services which are based on digital materials. On the other hand, it would be appropriate to use the term digital services to encompass the use of electronic services designed to improve access to traditional library collections as well as digital collections.
We should bear in mind that the library community has a responsibility to collect and deliver information resources regardless of format, and indeed to strive to put in place mechanisms which will promote integrated access to all formats. This will become more important as an increasing proportion of information resources are available only in digital form. As Lynch and Garcia-Molina [4] have observed:
[The] objective is to develop information systems providing access to a coherent collection of material, more and more of which will be in digital format as time goes on, and to fully exploit the opportunities that are offered by the materials that are in digital formats.... There is, in reality, a very strong continuity between traditional library roles and missions and the objectives of digital library systems.
Integrated access to diverse materials is usually accomplished through services which allow the relevant metadata for all materials to be searched simultaneously.
Integration can also be realised in the delivery process which follows discovery and access of the information resources. For example, it is now possible to deliver, in digital form, information resources from traditional library collections to remote users, using document delivery services such as Ariel. In this case the key difference to the user between a digital service based on a digital collection, and one based on a traditional collection, is the length of the delay that occurs before the collection item is delivered.
The emergence of the World Wide Web in 1993, and its rapid development thereafter, has allowed developers to provide universal access to digital libraries. Previously, access to digital collections was supported by proprietary networks or by local or campus-wide networks. These services also depended on a variety of end user software and hardware. By contrast, access through the Web is based on open standards (such as the HyperText Transmission Protocol) and on widely available browser software which can be used from anyones desktop computer. Because of the Web, place is not a barrier to access to digital libraries.
Amongst the earliest examples of pre-Web digital collections, we should recognise the construction in the 1970s of databases of full-text documents, supported by software such as STAIRS. These collections were single format, and access was limited by proprietary communication protocols and rudimentary interfaces.
A relatively ambitious pre-Web attempt to build a digital library was Project Mercury (1989-1992), a joint development of Carnegie Mellon University and OCLC. It developed software for uniform access to textual and image databases, including page images of journal articles. Access was to be confined to the university campus, with X Window interfaces.
The TULIP Project (1993-1995), which was planned prior to the emergence of the Web, facilitated access to materials science journals. Each of eight US universities developed their own solutions for access to the electronic versions of these journals. The project revealed a host of practical problems with content delivery, content storage, lack of integration with other services, printing, and authentication [5].
Any examination of digital libraries must recognise the achievements of the National Digital Library Program (NDLP) of the Library of Congress and its predecessor, the American Memory Project (1990-1994). This Project aims to make five million collection items accessible online by the year 2000, through digitisation of materials in the collections of the Library of Congress and other cooperating institutions. Key features of the NDLP are the attention given to selection, the quality of presentation of the digital surrogates, the use of quality cataloguing data, the standards and facilities which have been developed to support discovery and access, and the depth of the technical documentation made available on the project site [6]. The NDLP was one of the first examples of a publicly accessible, Web-based digital library service.
Some of the key digital library possibilities and challenges are being explored by the Digital Libraries Initiative (1994-1998), which comprises six research projects funded by the National Science Foundation, in partnership with the US defence and space communities. These projects will build testbeds of digital collections in a range of disciplines, evaluate their use, test various indexing and searching techniques, explore interoperability standards, and examine authentication and security issues [7]. The research results are being shared within the group and are being progressively reported to the public.
While the development of digital libraries is motivated by the imperative of improved information delivery for users, most of these projects also have a research aspect, as we have observed with the Digital Libraries Initiative projects. Alongside these pilot projects, research is also proceeding to develop conceptual models of the digital library, and to clarify the technical framework.
In one of the key early research papers, Kahn and Wilensky described a framework for digital libraries, and for distributed digital object services in general. This paper [8] established some basic definitions and terminology which are used to describe digital library services. The framework takes account of the fact that while content in a digital library takes a wide variety of forms, it is possible to define a digital object which applies to all such forms. The digital object is conceptually contained within an envelope which informs the access software how to unpack it, what standards it conforms to, and so on. Furthermore, a digital object consists of digital material plus a unique identifier for this material, sometimes called a handle, and access to the objects is supported by a distributed system which supports the discovery of these handles.
A paper by Arms, Blanchi and Overly [9] provides an excellent summary of the issues relating to the information architecture for digital libraries, including the Kahn/Wilensky framework.
The framework has been extended by others, including Payette and Lagoze, who have described an architecture for storing and disseminating digital library content [10]. This architecture provides a more complex model, for example by recognising the ability to have a range of different interfaces to the same digital object, and the ability to associate rights management schemes with the dissemination of the object. The architecture is being applied to specific projects such as The Making of America II, a Digital Library Federation project which is assembling digital information resources concerned with transport in the United States between 1869 and 1900.
This paper is particularly concerned with the issues raised for national libraries in the delivery of digital information resources. What is the role of a national library in the digital age? And how should a national library facilitate the delivery of national digital services?
The role of national libraries involves:
In the context of these roles, the following are amongst the challenges for national libraries which arise from the rapidly expanding world of digital publications:
These challenges will be examined in the context of the key library processes of selection, acquisition, bibliographic control, access control and preservation. It is recognised that a number of these challenges (particularly those of acquisition and access control) are not ones for national libraries exclusively.
The National Library of Australia has previously identified these challenges in a number of discussion papers [11-14]. It is continuing to discuss these issues with other libraries, particularly the Australian state libraries and other national libraries.
National libraries have a responsibility to collect and preserve the nations information resources, in all formats. In doing this, they must take account of the expanding world of digital publications, particularly given the rapid development of the World Wide Web since 1993. The fact that the Web has allowed large volumes of digital material to be published creates a particular problem for national libraries. The high cost of publishing in traditional formats has meant that publishers have effectively undertaken a filtering role, selecting only quality works, or those with high market appeal, from the many manuscripts submitted by authors. However, the Web has allowed many authors to find an alternative, lower cost, publishing channel. For national libraries, this presents a significant challenge in identifying and selecting the publications to be preserved, and in identifying these in the absence of traditional selection tools and legal deposit provisions.
A statutory deposit system is often used to support the building of a comprehensive or highly intensive collection of resources relating to the country or jurisdiction concerned. In Australia, at the national level, the Copyright Act does not mandate legal deposit for digital publications, but the National Library is pursuing the development of amendments to the legal deposit section of the Copyright Act which will make electronic publications, both physical format and online, subject to legal deposit. Of course, even when these amendments to the Act are made, the Library will not wish to retain all Australian electronic publications, just as it does not retain more than a small sample of printed ephemera.
It is also necessary to distinguish between selection for current access (including online reference services and licensed information services) and preservation for future access. In the first case (and unlike the traditional library) the library does not need to collect the object, but only provide the means for discovery and access. In the latter case, the library must either collect the object and take responsibility for its ongoing access through technological change, or make sure that some other institution has taken responsibility for this.
In Australia, through its PANDORA Project [15], the National Library has created, and is continuing to develop, an operational proof-of-concept archive of selected Australian Web publications. The Library has developed a set of guidelines for its selection of online Australian publications [16-17]. These guidelines take account of issues such as: Australian content, Australian authorship, the authoritative status of the author, research value, whether the publication exists in hardcopy form, whether there has been any quality control process in the online publishing, public interest in the subject matter, and whether the publication is indexed by a recognised indexing service.
One way of by-passing the selection problem is to collect and preserve everything published on the Web. In this context, national libraries have an interest in recent efforts to do just that. The Internet Archive is a well known project to archive the entire World Wide Web and some other components of the Internet, such as the Gopher hierarchy [18]. The Archive was founded by Brewster Kahle in April 1996. In October 1998, the archive for the months of January and February 1997, containing two terabytes of data, was deposited with the Library of Congress.
In 1997 the Royal Library of Sweden initiated the Kulturarw3 Project, in order to archive the Swedish domain of the Web. To date, this project has made three comprehensive harvests of the Swedish web domain. The result is a file of about 200 gigabytes covering 31,000 Web sites [19]. This attempt at a comprehensive collection of digital publications is consistent with the Royal Librarys practice of collecting printed ephemera through a legal deposit framework under which printers, rather than publishers, deposit their product with the Library.
Both the Internet Archive and the Kulturarw3 projects have stated that they will attempt to use data migration to maintain the readability of the documents. However, there must be doubts about the costs of storage and migration involved in this non-selective process, given that it involves successive snapshots of a large and rapidly expanding volume of information. There must also be doubts about the practicality of supporting access to each snapshot through an appropriate index.
A digital library collection may be built through either or both of the following processes (in addition to creation of content by the library itself):
In many cases, the materials selected for the digital library will consist of non-contemporary heritage materials, or other resources which exist only in non-digital form. In such cases a digital conversion process, such as imaging or OCR scanning, or a combination of both, must be used if it is desired to deliver the resource through a digital service.
The digital conversion process is a costly one because it is laborious both in processing time and in project management overheads. It is also a process which requires careful planning to ensure that the full informational value of the original material is preserved. Chapman and Kenney argue strongly for a strategy of full informational capture, in which the conversion process is matched to the informational content of the original, in those cases where the digital conversion is undertaken primarily for preservation purposes [20].
The National Library of Australia has had direct experience of two major digital conversion projects in recent years. One of these projects has developed IMAGES1, which now contains more than 20,000 digital surrogates of items in the Librarys pictorial collection. Access is delivered through the Web, through the Librarys catalogue and in the future through the Kinetica Service. The digital collection is updated through a routine digital conversion process as new items are acquired for the pictorial collection.
The Australian Cooperative Digitisation Project [21], also known as the Ferguson Project, provides access to digital surrogates of printed materials published in the 1840s. This Project has proved to be significantly more complex than IMAGES1, for several reasons. Unlike IMAGES1, the Ferguson project had a preservation as well as an access objective. Microfilming was part of the preservation process, and this introduced complexities into the imaging process. There were stringent resolution requirements, because there needed to be sufficient resolution to enable the digital surrogates to be read comfortably while also being downloaded in a reasonable time. Finally, the use of OCR scanning, used to improve search access to the Ferguson materials, added to the complexity of the Project.
Even where contemporary materials are involved, some digital library projects have created collections through a combination of imaging and OCR scanning from print originals, such as journal articles. On the surface this appears to be unnecessarily costly and inefficient, given that these contemporary resources, during their preparation, existed as machine readable text. Moreover, the process of OCR scanning re-introduces errors, violating authors' moral rights unless they are painstakingly corrected.
The builders of digital libraries would benefit greatly if they could obtain the authoritative version of any document in machine-readable form, complete with structural markup. By flagging the logical components of a document, structural markup allows descriptive metadata (author, title, abstract and so on) to be extracted efficiently from the document text. The processes of structural markup have been defined by the international standard framework known as SGML (Standard Generalised Markup Language). Use of this standard during the complete publishing process would support the more efficient integration of documents from various publishers into the digital library. It would also support more powerful and precise searching of the contents of the digital library, by allowing searches to be focussed on the descriptive metadata, the abstract, executive summary or other appropriate parts of the document.
There has been limited success in pursuit of this goal. One example is provided by the role of SGML markup in the Illinois Digital Library Project [22]. This project has constructed a multiple-publisher testbed containing tens of thousands of full text journal articles in the disciplines of physics, engineering and computer science. The project has been able to utilise SGML markup provided by the publishers, although it has converted this into a standard SGML format for federated searching.
The Illinois Project has been fortunate to obtain, from publishers, the authoritative texts with structural markup already in place. Most publishers perceive it to be too costly to treat their publications in this manner, and the prospects for adoption of SGML in the publishing community appear to be limited to specialised areas (such as the markup of case reports by legal publishers) or situations where it might be mandated (such as university theses).
Web documents already possess a weak form of structural markup in the HyperText Markup Language (HTML). The recently developed Extended Markup Language (XML) will increase the extent of structural markup for documents published on the Web. Some observers have expressed the hope that XML will be taken up by the commercial publishing community, helping to deliver all the power of SGML without all the complications [23].
Another method of acquiring documents which are already in digital form is to harvest or gather them from the Web, with their limited HTML markup. There is no need to undertake this process unless you wish to ensure the preservation of the materials concerned, since you need only to link to the resource to support current access. Since national libraries have a strong preservation mandate, gathering from the Web has been the approach which the National Library of Australia has used in its PANDORA Project.
The gathering process itself raises a number of challenges. To preserve fidelity to the original resource, it is necessary that the archive copy should replicate the directory structure, file names and internal links of the original resource, in parallel form. On the other hand, links to external resources will need to be disabled and replaced by a suitable message, unless these external resources have also been archived or continue to exist unmodified with permanent names. For efficiency reasons, it is desirable that the gathering software should automate these processes, while also collecting administrative metadata associated with the resource. Needless to say, gathering software with all these features does not yet exist.
It was noted above that the world of digital publications has created challenges, for national libraries and others, such as:
Bibliographic data (or metadata) is the key to integrating access to digital and traditional collections. A searcher often needs to discover information resources in all formats relating to a given subject, or which have been created by a given author. Services such as subject gateways, which have been developed primarily to support access to quality information resources on the Web, have the potential to support browse and search access to distributed collections of both digital and traditional materials.
Integrated access can be a particularly desirable goal when only part of a collection has been digitised, which will usually be the case for any library with a large collection of original materials. For example, a national or state library with a large pictorial collection may offer a service which allows a searcher to discover a digitised image of a picture or photograph which meets a particular information need, if the library has digitised this picture. But what if the picture that meets this need is one of the majority in the collection that have not been digitised?
The answer to this question is to offer a digital service which supports access to all of the librarys pictorial collection. The elements of this service might be (a) the original collection items, (b) the digitised images, (c) the metadata about the entire collection, and (d) a digitisation on demand service which supports the progressive digitisation of the collection in response to user requests for access to collection items.
One potential barrier to integrated access is the divergence of metadata standards between the world of traditional publications (based mainly on the AACR2 and MARC standards) and the world of digital publications (which may be based on Dublin Core or other non-MARC standards).
The Dublin Core metadata standard has been developed during the past few years, with two main purposes [24]. One has been to support simple resource discovery for digital collections. The second has been to provide a kind of lingua franca, with the potential to integrate access to the digital services of libraries with those of institutions such as museums and archives, and with access to the wider universe of Web based resources.
The development of this standard continues to be beset by debate within the community of implementors over the degree of structure and rigour which the standard requires. This discussion intensified at the most recent Dublin Core Workshop in Washington in November 1998, where the key debate focussed on the question of a more rigorous data model which might guide the development of a Dublin Core Version 2. Representatives of the commercial Web publishing community argued that the development of such a model would encourage the widescale implementation of the standard in the publishing and rights owner communities. However, others argued that development of a new version at this stage would undermine confidence in the standard, and would not be necessary given the basic purposes of the standard.
As well as being a lingua franca across the sectors, the Dublin Core standard may be useful in supporting search access across various collection levels. For example, the experience of the library and information industry has exhibited a divergence of standards between those used to describe and support access to the whole item, on the one hand, and those used to describe and support access to part of the item, on the other. This traditional dichotomy between the cataloguing community and the abstracting and indexing community has the potential to create access barriers. The Dublin Core standard may assist in bridging these barriers, by providing a basic set of descriptive elements to which the cataloguing and indexing data elements can be mapped.
For both traditional and digital collections, users have a need to navigate up or down through the levels of a collection. This may apply, for example, to items within a manuscript collection, articles in a journal, photographs in a collection, or individual pages within a book. In the physical environment it is straightforward enough for the user to manage this browsing process. However, in the digital environment the linkages between the components of the collection (or collection item) must be made explicitly.
There is no clear consensus on a preferred or standard method of supporting navigation through collection levels, given the wide variety of published and original materials which the digital collection might need to support. While the use of the Dublin Core Relation element is one possibility, another is a generalisation of the system of component part identifiers which has so far produced the Serial Item Contribution Identifier (SICI) and Book Item Contribution Identifier (BICI). These identifiers, while uniquely naming a component part of a serial or book, are structured so as to carry information about the serial issue, serial title or book. A more generalised standard might be able to carry similar relationship information for any multi-level resource, including collections of original materials.
At the lowest level, the part-of-item level, structural document markup also has a role to play in supporting access to text-based resources. For various purposes, searchers may wish to confine their searches to the abstract, the methods section of a scientific article, the citations, and so on. Research by the Illinois Digital Library Project has drawn attention to the potential for this kind of markup to support the disaggregation of journal articles into separate components [25].
The Digital Library is likely to consist of a distributed set of digital objects. But is distributed metadata the best way to provide access to the objects? There is no simple answer to this question, and existing projects reveal a continuum from:
A digital service may attempt to support integrated access to diverse categories of information resource, or to resources from diverse sources. The more diverse these categories or sources, the more likely it is that a centralised model (based on a uniform metadata standard) will be required, because the less likely it is that there will be sufficient alignment of the entire set of protocols that are needed to support the fully distributed approach.
This issue was discussed last year in the context of government information by the Search Engine Working Group, which was charged with developing a model for metadata based access to Australian government information in digital form. As the Working Group commented [26]:
The distributed approach would rely on significant standards and software development. It is essentially a parallel searching mechanism that is able take a user query in a single query syntax, translate it to multiple syntaxes of the underlying (agency) search engines in question, and translate the results (and the relevance rank values) back into a coherent result set. This demands that query and result syntaxes, and the ranking algorithms of each individual search engine, be made known to the whole-of-government search facility.
In this case, the Working Group was dealing with a relatively uniform collection of information resources - Australian government documents - but still judged it more practical to rely on a single central index to the distributed metadata until standards and the software were available to support a more fully distributed approach.
It was noted above that the Kahn/Wilensky Framework assumes a highly reliable system of unique identifiers, called handles, to support basic universal access to digital information resources. The handle and its supporting infrastructure would ensure that a resource can still be accessed even if it changes its location. The standard for Uniform Resource Names (URNs) as proposed by the Internet Engineering Task Force, together with a network of URN resolver services, are compatible with the requirements for a universal system of handles.
In a 1996 paper [27], James Miller drew attention to the important role of national libraries, as institutions of long standing, in relation to handles or permanent names:
There must be one or more entities that take institutional charge of the issuing and resolving of unique names, and a mechanism that will allow this entire set of names to be moved forward as the technology progresses..... The Digital Library community must identify institutions of long standing that will take the responsibility for resolving institutionalized names into current names for the forseeable future.... I propose that a small consortium of well-known (perhaps national) libraries could work to provide the computing infrastructure.
The National Library of Australia has indicated [14] that it favours the use of Universal Resource Names (URNs) in any national registration system. Consequently, the Library is exploring the implementation of a national mechanism enabling publishers of digital publications to register them, and in the process to apply for Uniform Resource Names. Publishers would contribute metadata as part of the registration process, and the metadata repository would be maintained as an Australian URN resolver service.
As commercial publishers increasingly make use of the Web as a publishing medium, important issues are raised in the relationship between publishers and libraries, including national libraries. Publishers have been comfortable with the role of libraries in relation to traditional publications (including the legal deposit system), knowing that the physical format provided natural limitations on access to the publication. However, there would be a threat to the remuneration of publishers and copyright owners if unlimited access to commercial digital publications were provided by libraries.
For these reasons, libraries must deal with the challenge of reaching agreement with publishers on reasonable access conditions for digital publications, including those received on deposit. They must also deal with the challenge of managing and controlling these access conditions. Because of the difficulties posed by these challenges, many digital library projects have concentrated on the digitisation of original materials which are not subject to copyright or which have little commercial value.
The access conditions could theoretically involve a wide range of options such as:
There has been insufficient dialogue between publishers and libraries for agreement to have been reached on the above options. For their part, libraries in Australia will not wish to see any agreement which weakens the fair dealing principles, especially given the recent report of the Copyright Law Review Committee on the Simplification of the Copyright Act. The Committee recommended that the Copyright Act be amended to consolidate the fair dealing provisions into a single, technology neutral provision which would, amongst other things, uphold the place of fair dealing in the digital environment [28].
If a satisfactory agreement were negotiated, there would remain the issue of managing and controlling the agreed access conditions. Currently, some major projects are underway to better define this management process, and to develop clearer models for commercial traffic in digital information resources. For example, the IMPRIMATUR Project has made some progress in developing a business model. The recently commenced INDECS (Interoperability of Data in E-Commerce Systems) Project [29] will attempt to create a genre-neutral framework among rights-holders for electronic [Intellectual Property Rights] trading so that companies [such as] record companies, film companies, book and music publishers can trade their creations in a coherent single marketplace. INDECS is funded by the European Commission, and the trade associations of almost all the major rights sectors are participating.
The control process demands the availability of software which can allow or deny access to a digital object depending on the rights conditions associated with the object, and possibly also depending on the category of user. The software should also notify potential users of conditions of use applying to an object, and support the online payment of copyright, reproduction or delivery fees. There is also a need to log usage of each object and report this to a designated copyright collection agency where applicable.
The process of controlling the use of the digital collection includes the requirement to recognise and authorise users, either as individuals or as members of a category. The issues involved in user authentication and authorisation have spawned some major projects. An example is the CNI Program on Authentication, Authorization and Access Management, which issued a key discussion paper [30] in April 1998. This paper is the focus of continuing discussion amongst information providers and institutions, who wish to see an efficient process through which end users can gain access to registered information resources. In the United Kingdom the ATHENS Project, which is based on a central database of authorisation information, is providing what some observers have judged to be a satisfactory interim solution [31].
The Web has become a powerful information tool despite its chaos, partly because its open, standards-based system of hypertext links has given it a genuine universality. Many of the existing digital library services, and those under development, are based on free and unfettered access to the collections. As these services are joined by digital libraries which impose user charges or other access conditions, the universality provided by the Web is weakened. To quote from Clifford Lynch [32]:
As long as all resources are free, the move from one sphere of control to another is not so important to the user. But as transfer from one system, which may be free, to another, which charges for use, becomes commonplace, the notifications of these transitions may weaken the sense of a coherent information space. Indeed, the user may want a less coherent presentation to make cost implications more visible.
For institutionally based users, this problem can be addressed in part through the use of licence arrangements for access to the commercial services, so that the user could effectively search these services and free Web resources through a common subject gateway, with no metered charging applied to either. However, this option is not available to the independent scholar or the casual user, and its implementation for institutional users depends on solutions to the challenges of authentication and authorisation.
This paper has surveyed many of the challenges which are involved in building and managing a digital library. These challenges are all the subject of continuing research or standards development. But none of them is as serious or as potentially intractable as the problem of digital preservation.
It takes an effort to recall that the personal computer is barely 20 years old. Computer technology is changing so rapidly that the design, the interfaces, the technical standards and the file structures of the computers of 2020 are very likely to be quite different from those used today. For national libraries and other research libraries, which have a tradition of building collections to meet the needs of scholars many decades or even centuries into the future, this presents a very formidable challenge. The ability to access and read digital information in the future will depend on strategies such as migration (in which the data is migrated, if technically feasible, to new operating systems and data structures) or emulation (in which modern computers emulate the operating systems and data structures of previous eras).
A National Library of Australia position paper on electronic publications observed [12]:
[There] are unresolved technical issues to be confronted. These include the necessity to reformat material involving conversion from one format or medium to another.... Details of how a continuous program of updating, through migration or refreshment, as a digital preservation strategy might work remain to be developed. For multimedia publications it may not be possible to keep the "look and feel", and interactive or dynamic aspects of these publications may be lost.
The many aspects of this challenge have been documented on the PADI (Preserving Access to Digital Information) web site which is maintained by the National Library of Australia, and the interested reader is urged to explore that site thoroughly. The issues were also explored in depth by a key report commissioned by the Commission on Preservation and Access and the Research Libraries Group [33].
Another report from the Commission on Preservation and Access [34] has discussed the importance of structural markup, using SGML standards, to the long term accessibility of digital materials:
When viewed from the perspectives of the preservation community and the digital librarian, SGML appears to be the best choice for archival document storage. SGML is standardized, portable, flexible, modular, and supported by the market. [Its] vendor neutrality, extensibility and ability to manage large information repositories make it an attractive choice for both archival and retrieval purposes.
Some less optimistic observers have questioned whether, given our short experience with digital information formats compared to that of paper, the quest for digital preservation has any realistic chance of success. Michael Gorman presses this point [35]:
I would suggest that we should work out collective schemes to make copies of such documents on acid-free paper, catalogue them, and add the resulting records to local, national, and international databases. I insist on saying that the only practical manner in which we can preserve our present for posterity is to create print on paper archives and to create an enduring bibliographic structure to sustain those archives. All suggestions of massive electronic archives are confronted with insuperable economic, technological, and practical problems.
Peter Graham has also expressed serious doubts about the practicality of digital preservation [36]:
The investment necessary to migrate files of data will involve skilled labour, complex record-keeping, physical piece management, checking for successful outcomes, space and equipment. A comparable library [project for] data migration cost and complexity at approximately this order of magnitude would be the orderly photocopying of books in the collection every five years.
These remarks rightly warn us of the depth of the challenge that this issue presents. However, it would be premature to assume now that the problem is incapable of a solution. Many international research efforts (including the CEDARS Project in the UK, the work of Jeff Rothenberg on emulation, and the work of the Digital Library Federation) are actively pursuing the digital preservation question from a wide range of perspectives. There is a great imperative to find a solution, or a combination of solutions.
Since it began to deliver information through the Web, the National Library of Australia has developed digital library services through projects such as IMAGES1, the Ferguson Project and PANDORA. The Library now needs to find better systems tools to enable it to meet the twin challenges of managing its present and future digital collections, and of supporting shared access to digital collections. To this end, it has commenced a project which it is calling the Digital Services Project.
The Library expects that the system resulting from the Digital Services Project will enable it to collect, organise and preserve its digital materials, and to support access to them which is integrated with access to its traditional collections. These services will apply to items that are born digital and those which are reformatted from non-digital originals. The system would enable the Library to manage, in a robust, high performance systems environment, collections such as:
The Library also expects that the system will enhance the national digital infrastructure by enabling the Library to:
Theoretically, Kinetica could be enhanced to support the second of these goals, and this remains a possibility. However, Kinetica is limited in terms of the types of metadata that it can support. Consequently, the Library may decide to support the cooperative access services outside Kinetica.
In any case, the Library has stipulated a number of technical standards, to ensure that the resulting systems can interoperate with other national and National Library systems such as Kinetica and the Librarys web catalogue, and also to ensure that the Library is able to load, store and search a wide range of structured digital documents, representing formats including text, sound and image.
The Library has allocated capital funding to initiate this project. In December 1998 it distributed an Information Paper setting out its detailed requirements for the Project, with the aim of encouraging comment from libraries and the IT industry. The Information Paper is available from the Librarys web site.
The Library is planning to release a Request for Tender in the second quarter of 1999, after taking account of the comments received in response to the Information Paper.
This paper has discussed key challenges faced by the developers and providers of digital library services, with emphasis on the challenges for national libraries. The challenges span the entire range of library functions: selection, acquisition, access, management and preservation. In many of these areas, research efforts are attempting to pilot possible solutions, develop better conceptual models, or formulate improved standards. However, the solutions to these challenges cannot be left only to the researchers. They will require a response by all of us who are attempting to build new or improved digital services.
This paper has identified some themes which might influence these responses. One theme has been the importance of standards to the processes involved in digital services. More effort is still required in the development and implementation of standards. For example:
A standards issue which has been raised a number of times in this paper is the importance of structural markup of digital publications. For example, structural markup can:
Another of the themes of this paper is that many of the responses to the challenges of digital services appear to require a more concerted dialogue with publishers, and with the rights owner community. For example:
The development of digital libraries has opened up an exciting new world of information delivery for the researchers and citizens of tomorrow. It is our responsibility to improve these services by addressing these challenges in concert with our colleagues around the world.
I wish to acknowledge the ideas and suggestions from my colleagues at the National Library of Australia which have been incorporated into this paper. In particular I wish to acknowledge the contribution by staff who, over the past few years, analysed and debated most of the issues which are discussed in this paper, and developed models for the Librarys digital services.
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |