Staff Papers
Report on Senior Executive Fellowship to Research Digital Archiving in National Libraries
This report provides an account of the study trip undertaken by Pam Gatenby (Assistant Director General, Collections Management) on the Senior Executive Fellowship awarded to research developments in digital archiving among national libraries. The Fellowship enabled attendance at a major international library conference and visits to libraries active in the area of digital archiving. The study trip was undertaken between August 16 and September 10 2002.
Introduction
This report provides contextual background to the subject of the research, an outline of the main activities being carried out by the institutions visited, a summary of the main observations and conclusions reached, and a statement of outcomes. In addition, in accordance with the requirements of the Fellowship, a section on implications of findings for the wider Australian Public Service (APS) is included.
Throughout the report, the term digital archiving is used as shorthand for the full range of activities associated with collecting and managing publications in online form for access for instance; selection, collection, quality control, cataloguing, file management and preservation for on-going use. The Library is used to refer to the National Library of Australia. The terms online resources and online publications are used interchangeably to refer to the range of information types on the Internet such as PDF and hyper linked documents, and multi-media web sites and parts of web sites that are used to disseminate information to the public. Online refers to dissemination via the Internet while digital can refer to both online and physical format resources in digital form.
Background
A large part of the documentary heritage of countries is now published in online form only. Important information is being disseminated via the Internet from a wide range of sectors, including traditional commercial publishers, government agencies, academic institutions, the private sector, public bodies and community groups. Unless steps are taken early in the life of online information to prevent it from disappearing, its availability for use will be short-lived and future generations will be denied access to an important component of their documentary heritage.
There are many complex issues involved in ensuring important information in online form remains available for use in the long-term. They relate to:
- the ephemeral nature and rate of growth of information on the Internet;
- deciding what is important and worth saving for historical and cultural reasons;
- the rapid rate of change of the hardware and software on which digital resources depend;
- the costs involved in taking on responsibility for collecting and preserving the resources;
- the legal issues involved in copying and providing access; and
- organisational, procedural and technical systems issues.
An excellent overview of the issues from the cultural heritage perspective, is provided in an article by Peter Lyman, School of Information Management and Systems, University of California, Berkeley, titled Archiving the World Wide Web (http://www.clir.org/pubs/reports/pub106/web.html).
The role of national libraries
Most national libraries now accept that their traditional role of collecting and safeguarding access to information published in their country must extend beyond print based publications to those in online form. However, national libraries are at very different stages in developing their responses to this new responsibility. Many have not yet committed to the challenge while among those that have, approaches vary but the problems they are facing are the same.
Digital archiving at the National Library of Australia
The National Library of Australia has a statutory responsibility to collect and preserve significant Australian publications regardless of their format.
The Library has been at the leading edge of world developments in collecting and archiving cultural heritage information resources in online form since 1996, when it established the PANDORA (Preserving Access to National Documentary Online Resources of Australia) project. Archiving is now a mainstream activity, involving collaboration with the state libraries and with ScreenSound (the National Film and Sound Archive) to develop a national collection of Australian online resources. This collection, which resides in the PANDORA archive, now includes over 3,000 titles - many of which have already disappeared from the Internet - and is growing at a rate of around 900 titles per annum.
The Library's digital archiving activity is well regarded by other libraries around the world and the Library is often invited to participate in international conferences and projects.
Detailed information on the policies and procedures that underpin the national collection of online resources and the Library's digital archiving activities are available at http://www.nla.gov.au/initiatives/digarch.html. The archive itself can be searched via the Internet at http://www.nla.gov.au/pandora.
In brief, key features of the Library's approach to digital archiving include the following:
- Resources that are archived are selected according to detailed selection criteria.
- Titles are collected and archived with the permission of their publishers, as the Library does not yet have legal deposit rights for online resources.
- All types of online resources are collected - plain text, multi-media, interactive documents, parts of web sites and whole web sites.
- About 1/4 of the archived resources are collected on a regular basis, with previous versions retained.
- Public access is a prime objective: all titles are catalogued on the National Bibliographic Database (NBD) [1] and public access is available to the collection via the Internet.
- Some titles (mainly commercial in nature) have public access restrictions applied for a certain period of time, which is negotiated with the publisher.
- Digital preservation requirements are integral to our planning and routine procedures.
The National Library of Australia accepts a special responsibility for collecting and preserving the published output of Commonwealth government agencies. It is a priority for the Library to develop collaborative approaches with agencies to manage online publications, and to address issues such as how to guarantee the authenticity of what is archived and how to provide persistent public access in the face of broken URLs. As the government sector is now a major producer of online content in Australia, such issues are pressing.
Collaboration with other creators and publishers of digital content is also an essential and integral part of the strategy being pursued by the National Library in order to increase the range of nationally significant information that is saved. The Library is currently considering various business models for cross- sector collaboration and is aiming to establish digital archiving agreements with selected publishers.
Since 1996 the Library has gained considerable experience with digital archiving. It has developed a sound understanding of some of the issues and barriers that must be dealt with as a priority if countries are to safeguard their online documentary heritage. In brief, the main issues the Library is currently concerned about are the extension of legal deposit to online resources; how to make our routine operations more cost-efficient; how to extend the scope of our collecting through collaboration and technical solutions; and how to collect and manage online resources structured as databases. Preserving access to the resources we collect for long-term use is an overarching, ongoing concern.
The Library is now at the stage where it must determine how it will address these issues and the directions it will pursue in the near future. This challenge was a major reason for applying for the Senior Executive Fellowship.
What other national libraries are doing
Some other national libraries have also been collecting online resources routinely for a few years now while others are still experimenting with various approaches.
Those that have developed a body of knowledge through experience include the national libraries of Sweden, Finland, Denmark, and Canada and the Library of Congress. These libraries have adopted two different approaches:
Selective collecting : specific resources are collected according to selection guidelines, generally using harvesting software and involving manual quality control (Australia, Canada, Denmark, Library of Congress)
Comprehensive collecting (also known as Whole Domain Harvesting) : the whole web domain for the country concerned is gathered a couple of times each year using harvesting software (Finland, Sweden). Very little manual intervention is involved.
Each approach has advantages and shortcomings, as follows.
With selective collecting, effort is concentrated on important resources and these are collected according to selection criteria. The collecting process is generally quality controlled and specific technical problems that some resources present are resolved on a case-by-case basis. Access conditions are negotiated with rights owners if necessary and a reliable form of searching and retrieving resources from the archive is provided. However, this is a very staff intensive approach and the amount of information that any one institution alone could hope to collect is insignificant compared to the total amount of potentially eligible information on the Internet although, a lot of this could be considered not worth keeping long-term. Also, the selective approach takes resources out of the web context in which they reside and consequently they can lose some of the meaning that the context provides.
With comprehensive collecting, or Whole Domain Harvesting, the entire domain of a country is gathered a couple of times each year. No manual intervention is generally involved instead, selection is based on domain identifiers. This results in a huge amount of information to be managed and archived. The quality of what is gathered is unknown but it is likely that files will be missing, links broken, and complex sites missed completely. Restricted sites are unable to be gathered. To provide access to archives created in this way, a means of providing automated indexing must be developed. Also, legal issues have to be considered if gathering is carried out without the right to do so. While the staff requirements are fewer with this approach and a more comprehensive archive is definitely developed more quickly, the costs associated with storage, downloading and programming are high.
A purpose of my study trip was to find out more about the two approaches from the libraries with experience in using them, and about their plans for dealing with the shortcomings. In addition, I was keen to explore the reaction of other national libraries to a proposal from the Internet Archive [2], the major proponent of the Whole Domain Harvesting approach.
The legal framework within which national libraries around the world are currently carrying out their collecting and access responsibilities, varies markedly from country to country. A few - for instance, South Africa, Norway and Finland - have amended their legal deposit [3] legislation to apply to publications in digital form, while others, like Australia, New Zealand and Canada, have processes in place to do this. Copyright legislation is also a major issue for national libraries in carrying out their roles in the digital age. For many, it is proving to be an immovable obstacle that is limiting their aspirations with regard to providing long-term public access to digital resources.
Another feature of the international scene is the amount of experimentation and research that is being conducted within national libraries and in collaboration with other bodies. In general terms, the research is attempting to find automated solutions to basic needs such as focused collecting and metadata creation in order to reduce the staff costs involved in these activities. It is difficult to keep up with the nature and potential significance of this research from a distance but it is nevertheless very important to try to do so. Much of the research is local and small-scale, being carried out by national libraries that have not yet committed to a recurrent digital archiving program. However, some initiatives would seem to have the potential to deliver shareable and scaleable technical solutions to particular problems. Gaining a better understanding of the nature of the research being carried out by national libraries was another reason for applying for the Fellowship.
Purpose of Fellowship
At the broad level, the chief purpose of the Fellowship was to research recent developments among national libraries and related organisations in collecting and managing publications in online digital form and in providing on-going public access to them. In particular, I was interested in investigating developments in the following areas:
- Legal deposit and copyright
- Collecting online government publications
- Collecting complex online publishing formats
- Providing public access to restricted resources and to resources that have been collected through the whole domain harvesting approach
- Cross-sector collaboration - what makes collaboration work and what are the costs?
- Sustainable business models
All of these areas of interest are of immediate and strategic importance to the National Library of Australia and encompass complex issues that must be confronted if we are to continue to be effective in carrying out our responsibilities with online documentary heritage resources. To do this we must remain at the forefront of strategic thinking and developments. In addition, given the potential enormity of the costs involved, collaboration with a range of stakeholders including other national libraries, will be necessary. The Fellowship allowed me to explore practical opportunities for collaboration.
The Fellowship
To maximize the benefit to be gained from the Fellowship study trip, I visited key national libraries which are active in digital archiving, and OCLC Online Computer Library, a large US library consortium providing online services to research libraries in the United Sates and around the world. I also attended and participated in the annual International Federation of Library Associations (IFLA) conference, which this year had a strong emphasis on digital archiving. I had also intended to visit the US Government Printing Office (USGPO) but unfortunately the dates of my time in Washington coincided with their summer vacation. However, by visiting OCLC I was able to find out about the plans of the USGPO to start collecting government online publications as they are a member of OCLC and will be using their new digital archiving service.
The International Federation of Library Associations (IFLA) (Glasgow, Scotland)
The IFLA conference is the world's largest international gathering of librarians (generally around 5,000), providing an important forum for the interchange of developments in all fields of information management. A major value of the IFLA conference is the unique opportunity it presents to forge international networks and to meet up with colleagues from all over the world who are working in areas of common interest. At the conference I presented two papers on aspects of the National Librarys digital archiving strategies and procedures. I also attended other sessions at which digital archiving was a main focus and learnt about developments in European national libraries with which we generally have little contact, due to problems of distance and language. I also arranged some meetings with relevant colleagues in advance of the conference.
The IFLA conference was a very rewarding and exciting experience professionally and proved to be an invaluable forum for catching up with developments world wide in digital archiving.
The national libraries of Sweden and Denmark
The national libraries of Sweden and Denmark the Kongl Biblioteket (KB) and Det Kongelige Bibliotek - are doing very interesting and innovative work in the area of digital archiving and they are taking different approaches from us. Sweden has been taking snapshots of the whole Swedish web domain twice a year since 1996. They have solid experience with this approach a good understanding of its features. They provide a search service to the contents of their archive called Svesok. It searches on basic metadata extracted from the harvested resources.
Denmark has been collecting static online publications under legal deposit since 1998. They developed a system for managing the publications but access to their archive is restricted to two workstations one at the Royal Library and one at the university library, Arhus. The archive can be searched on data fields in the legal deposit form publishers use to advise the library of new static publications. The library is now considering the Whole Domain Harvesting approach for collecting non-static web resources but it is waiting for their legal deposit legislation to be amended to cover these resources.
Both libraries are also working with the national libraries of Norway, Finland and Iceland on developing a common access module for searching across the resources in all their archives. The project to develop this is called the Nordic Web Archive project (http://www.nwa.nb.no)
The Library of Congress (Washington, DC (USA))
In early 2001 the US Congress appropriated $US100 million for the Library of Congress (LC) to develop, in collaboration with the public and private sectors, a national strategy for the archiving and preservation of digital materials. This initiative is called the National Digital Information Infrastructure and Preservation Program (NDIIPP) (http://www.digitalpreservation/ndiipp.gov). LC has conducted initial consultations with a range of stakeholders and is due to report back to Congress with an action plan by the end of 2002. I was particularly interested in discussing with them their approach to developing partnerships, identifying research and national infrastructure needs, and preparing models for national collaborative action. (Useful information on these topics is now available on their web site.)
In addition to NDIIPP, LC is leading initiatives associated with the bibliographic control of electronic resources. They have recently released a national collaborative plan called Bibliographic Control of Web Resources: a Library of Congress Action Plan. They have also collected web sites on specific topics in association with the Internet Archive. This project is called Minerva and I was interested to discuss their future intentions with web archiving. (LC considers the extent of their own future commitment to collecting and preserving online resources as uncertain at this stage. They indicated that considerable internal policy deliberation is still required. In the meantime, they will continue to select web resources on a highly selective, thematic basis in association with the Internet Archive)
The National Library of Canada
The National Library of Canada (NLC) commenced archiving online publications at about the same time as the National Library of Australia. It therefore has an established archive and considerable digital archiving experience. They have a strong focus on archiving online government publications and are at a similar stage to Australia in attempting to amend their legal deposit legislation to cover online publishing. NLC is the only other national library in the world that I am aware of with a public archive of selected resources. While there are strong similarities with what we are doing there are also some differences in approaches. It was very useful to discuss their future directions and areas of mutual interest. These included more effective ways of working with government agencies on issues related to online publishing, extending legal deposit to digital resources, and collecting a wider range of web resources.
OCLC (The Online Computer Library Center, Dublin Ohio.)
OCLC is a nonprofit membership organization serving over 40,000 libraries in 82 countries around the world. It provides services to libraries and their users aimed at reducing costs and enhancing access to information.
The chief reason for my visit was to find out about the Web Document Digital Archive (WDDA) Project (http://www.oclc.org/digitalpreservation/archiving/wdda.shtm). The goal of the project is to create a service providing long-term access to web documents for customer libraries. Specifications for the service were developed with a number of US libraries with an interest in archiving government information. Phase 1 of the service was released just before my visit and it was very interesting to be given a demonstration of its functionality so that I could compare it with the Librarys system. OCLC staff also very generously offered to provide a remote demonstration to Library staff when I returned to Australia. (This has since occurred.) The Librarys infrastructure is more mature at this stage but OCLC are intending to develop theirs further. We will watch its development with interest and collaborate by exchanging information on development plans where appropriate.
During my visit the Library was also invited to join a new collaborative service called the OCLC Digital & Preservation Cooperative, which we have since done.
Observations and conclusions
The study trip was extremely useful in providing current information on the digital archiving activities of national libraries and insight into their concerns and intentions. The observations made during the trip and the conclusions I reached have helped me determine the Librarys digital archiving action plan for the coming year. The main observations and conclusions are outlined below together with an indication of the consequences (if any) for the Library.
General observations
National libraries are interested but cautious
The level of interest among national libraries around the world in protecting their documentary heritage in digital form has definitely increased quite markedly in the last couple of years. Most libraries are now aware of the issues and are considering their options for implementing action. However, it is interesting to note that with one or two exceptions, the only national libraries with routine procedures in place for collecting online resources, are the ones who commenced in the mid to late 1990s. Some other libraries are experimenting with solutions to particular issues such as how to harvest online resources in more targeted or "clever" ways, and how to automate the generation of metadata, but on the whole national libraries would appear to be cautious about committing to a course of action. This is due in some part to the fact that the national library sector as a whole is now better informed than in the early days of digital archiving when pioneering libraries were prepared to test the waters. Based on the lessons learnt by the early implementers, there is now a better understanding of the costs and issues involved, and of the barriers to be overcome.
Some national libraries are also frustrated by the absence of effective structures for collaboration and by the perceived lack of guidance by professional library bodies on some fundamental "business" principles that underpin traditional national library roles. For instance, there is no consensus and not much debate on issues such as the scope of our collecting responsibilities, our obligations with regard to public access, the continuing relevance of the national bibliographic function, and the organisational implications of taking on new functions. This frustration was expressed in particular at some of the IFLA conference sessions relating to the role of national libraries in the digital age.
An important development that could go some way towards addressing the need for guidance and debate on basic principles is establishment of a Digital Issues Working Group by the Conference of Directors of National Libraries (CDNL). CDNL is an independent body that represents the interests of national libraries around the world and which meets annually in conjunction with the IFLA conference. (The Director General of the National Library of Australia currently chairs CDNL.) At the meeting of the Digital Issues Working Group at this years IFLA conference, it was agreed that the work plan for the group would cover 3 areas of central importance to the role of national libraries - preservation, access, and legal deposit and agreements with publishers. The National Library of Australia has agreed to lead the group focusing on legal deposit and agreements with publishers.
Experienced libraries are at a crossroads
An interesting observation that arose from visiting those national libraries with experience in digital archiving was that they are all at a crossroads with regard to planning their future directions for digital archiving. This is also the case to some extent with the National Library of Australia so it was useful to compare experiences and issues. For instance, the National Library of Sweden is aware that the Whole Domain Harvesting approach has limitations and that it is desirable to supplement this approach by doing some more specific and focused collecting. Consequently, they intend to harvest some newspaper titles more frequently and are also working with publishers to collect and catalogue particular categories of online titles. However, without supportive copyright and legal deposit regimes in place they are reluctant to address the problems associated with collecting and providing access to complex and/or restricted online resources. They accept that some important resources that fall into this category will be lost.
The National Library of Denmark has been collecting static online resources under legal deposit on a selective basis since 1998 but is now considering moving to a Whole Domain Harvesting approach as they are concerned about the sustainability of their current approach when legal deposit is extended to non-static resources. However, they acknowledge that, while Whole Domain Harvesting requires less manual intervention, it is not necessarily a more affordable option and still has some very big issues to be resolved. The National Library of Canada is also reviewing the scope of its collecting with the view to extending collecting to non- document type web sites and more complex online resources.
The main concerns shared by those national libraries with digital archiving programs already in place are the scope of their collecting, the sustainability of their current approaches, and the need to automate operations.
Observations relating to specific areas of interest that formed the focus of the Fellowship
Legal deposit and copyright
Copyright is a major obstacle for many national libraries to collecting, copying and providing access to publications in digital form. The Commonwealth of Australia Digital Agenda Act 2000 is far more progressive than the copyright law of any of the countries visited and is viewed with envy by other national libraries. Review of the application of national copyright legislation to intellectual property in digital form was at different stages in the countries visited and there was some pessimism about the extent to which any revisions would support the missions of national libraries. Consequently, while national public access to documentary heritage in digital form was still considered a necessary goal for national libraries, few seem to consider it achievable and have curbed their aspirations to be in line with what they think will be realistically achievable under copyright law.
With regard to legal deposit of publications in digital form, a best practice model has not yet emerged. A few countries have amended their legislation to varying degrees. However, no country yet has legal deposit legislation in place that covers the full range of online publishing and is in a position to implement the legislation. It was useful talking to national libraries that have already or are in the process of amending their legislation, in order to see what lessons they had learnt. Denmark, for instance, amended their legislation to cover online static publications only, which is now considered shortsighted and difficult to administer. The British Library (with whom I met at IFLA) has been working on amending their legislation for several years now and has learnt that the full cooperation and support of publishers is essential to progress even if this slows down the process. Canada is following a similar process to Australia and hopes to amend their legislation early next year.
The general impression I gained in this area was that Australia is well placed on both the copyright and legal deposit fronts. Even though we do not have legal deposit in place for digital publications, a process is in train to change this situation. If requirements that have been identified during the process can be delivered, then Australia would have a favorable and progressive legal framework in place for preservation of the country's published output in online form. The approach we are taking was outlined in a paper I presented at the IFLA conference, which generated considerable interest.
Collecting government publications
A particular area I was interested in exploring on the Fellowship was collecting government publications in online form. I was especially interested in the selection criteria and workflows used, and collaborative arrangements between national libraries and government agencies. The two most useful visits for this purpose were to the Administrative Library in Denmark and the National Library of Canada. The Administrative Library has very recently commenced collecting and archiving online Danish government publications with the official support of the Dutch government. All agencies are required to notify the Library of new online publications they issue. The selection criteria they use are essentially the same as those used for printed publications. As the initiative was in its infancy when I visited, it was too soon to draw any particular lessons from it but I will monitor its progress with great interest. It will be particularly interesting to find out whether agencies comply with the requirement to report new online publications and what level of collecting activity the Administrative Library is able to sustain.
The National Library of Canada has been very successful with collecting online government publications and now has around 10,000 in its archive. Government publishing is the chief focus of their collecting unlike at the National Library of Australia where we have collected widely across a range of sectors and formats. The main reason Canada has been able to build up a large collection of online government publications is that they collect only straightforward document type publications and do not seek the permission of agencies to archive each individual title- rather they seek blanket permission from agencies. The National Library of Australia has now adopted this latter approach.
While visiting the National Library of Canada, I was also interested to learn about a collaborative initiative focused on government information that they are involved in. The initiative is being led by the National Archives of Canada and is based on development of a tool with KPMG Consulting called the IM (Information Management) Maturity Model Tool. The purpose of the tool is to assist federal government agencies to assess their information management capacities against generally accepted best practices. The model covers print based and electronic information resources. It provides a useful example of how to convey information management responsibilities to government agencies and it will be considered when the Library develops further its strategy for working with government agencies. (See Outcomes)
The other visit that was relevant to my interest in government information was to OCLC, which has just made a new service (the Web Document Digital Archive service) available to customer libraries. The service provides systems to support collecting and archiving online publications. The specifications for the service were developed in collaboration with US state libraries with a responsibility for government publications and with the USGPO. However, the OCLC service can be used for any type of online publication and the collaboration on developing the specifications did not reveal any requirements unique to online government publications. A demonstration of the service indicated that it has similar features to the National Library of Australias digital archiving system (developed in-house) but that its functionality is not as extensive as the Librarys system.
A general conclusion I reached with regard to archiving government publications was that on the whole national libraries treat government publishing the same as that from other sectors. The only example of action focused specifically on government information was the Canadian IM Maturity Model initiative. It is likely that more innovative work in this area is occurring in national archives and other organisations (such as the Danish Administrative Library) whose chief responsibility is government information.
Providing access to complex online resources
The National Library of Australia is alone so far in routinely collecting online publications in a range of formats. The Library collects the simple along with the complex - document type publications as well as multi-media and interactive web sites, for instance. To provide access to these resources in the future it will be necessary to collect and preserve the specific software on which access is dependent. The Library is not currently doing this, as it is a big undertaking that has international implications. The preferred approach is to build a software repository through collaboration with other libraries and interested parties. However, as other libraries have very little experience so far in collecting this type of online resource, I concluded from my visits that probably the most productive way to address the issue is through bodies such as the CDNL Digital Issues Group, the Internet Archive and major research programs such as the Library of Congress NDIIPP. The Library will nominate development of an international software repository as a priority issue to be taken up by the Digital Issues Group. We will also recommend research into ways of archiving database driven online resources. Many important online publications are structured as databases and as such cannot be collected using harvesting software, as this is dependent on the presence of web hyper-links. The Library is keen to develop a collecting and archiving strategy for this category of online publication as it is becoming more common but so far no library has worked out how to manage them. (I learnt at IFLA that the Bibliotheque de France has started looking at the issue.)
The resources that the Library could commit to solving these issues alone are insufficient compared to the scale of the problems involved.
Approaches to collecting online resources: Whole Domain Harvesting
The Whole Domain Harvesting approach would offer a viable alternative to the Librarys current selective approach if the technical, legal and financial issues associated with it could be addressed satisfactorily - however, this likelihood would still seem to be some time away. A major advantage of Whole Domain Harvesting would be that many more resources would be collected than is possible through the selective approach. Also, it is expected that in time it would offer a more efficient way of operating. The Fellowship provided the opportunity to find out first-hand about some interesting developments that are aimed at addressing the shortcomings of the Whole Domain Harvesting approach - for instance, the Nordic Web Access (NWA) Project is attempting to provide the means to search across archives of harvested web resources, and the Internet Archive is proposing to develop harvesting software that meets the specific collecting requirements of national libraries. However, considerable research is still required to address requirements such as validation and quality control of archived content; gathering the full range of file types that are now available on the Internet; and reliable searching of the content of archives built through Whole Domain Harvesting.
Many national libraries consider that the Whole Domain Harvesting approach to collecting is the solution to fulfilling their collecting responsibilities for online resources. However, I detected that overall there is actually little understanding of its merits and shortcomings. Much more debate is required on what national libraries actually require from it as an approach before they commit to it as a solution.
While the Library has an interest in the potential of the Whole Domain Harvesting approach as a more cost-effective way to collect online resources, we do not have the resources to invest in the necessary research ourselves and must therefore wait for solutions to emerge elsewhere. Nevertheless, we will attempt to influence international efforts by working with the main advocates of the Whole Domain Harvesting approach such as the Internet Archive to the extent our resources permit, and we will monitor related developments closely.
In the meantime, in the face of increasing resource pressures, we must explore other ways of making our current operations more cost-effective. To this end, in order to reduce the costs associated with cataloguing and providing access to the resources we archive, early next year we will investigate the automatic generation of descriptive metadata. We will draw on the work being done in this area at the Danish Bibliographic Center (visited on the Fellowship) as well as at the National Library of New Zealand (with whom we keep in touch). We will also study the indexing approaches that the Royal Library in Sweden, the Nordic Web Archive project and the Internet Archive are taking to provide access to their harvested web resources.
Cross-sector collaboration
There is general recognition by national libraries that collaboration is essential to finding sustainable technical solutions to problems that are beyond the means of any one institution to solve. It is also understood that collaboration must involve a range of stakeholders. There are several examples of national libraries working with publishers to share responsibility for safeguarding publishing output in digital form. A recent notable example is the agreement, announced at the IFLA conference, between the Royal Library in the Netherlands and the international journal publisher Elsevier Science. Under this agreement, the Royal Library became the official digital archive for approximately 1,500 Elsevier Science journals made available on the web through its ScienceDirect service.
The National Library of Australia intends to use this agreement as a model for pursuing similar arrangements with selected Australian publishers. It was useful to have the opportunity to discuss its development with relevant staff from the Royal Library who were at IFLA.
The most important example of cross-sector collaboration to emerge so far from the library sector is the National Digital Information Infrastructure and Preservation Program (NDIIPP), managed by the Library of Congress (LC). LC has identified collaboration with the private sector in particular as an essential plank in developing a national strategy for preserving cultural heritage in online form. Identifying key stakeholders and determining how collaboration might work in practice, have been major concerns of the program to date. Central strategies include using existing research infrastructure and coordinating effort at the national level to avoid unnecessary duplication of effort. The National Library of Australia will monitor progress with NDIIPP with great interest and will attempt to benefit from its outcomes. As a result of the time I spent at the Library of Congress while on the Fellowship, I gained a much clearer understanding of the outcomes they are seeking from the project.
All the national libraries I visited raised with me the need for collaboration. In fact, most considered it essential to making further progress. However, identifying exactly what to collaborate on and how to bring collaboration about requires careful consideration. The Library also considers collaboration essential to moving forward but knows from experience that collaboration can be demanding and requires a shared commitment of resources if it is to succeed. Consequently, as already mentioned, I consider the best way for the library to participate at the international level in the short term is through the CDNL Digital Issues Group. This group provides an accountable framework and has the support of several major national libraries which collectively should be able to contribute a solid skills and resource base.
Sustainable business models
A surprising observation made during my study trip was that with one or two exceptions, national libraries have done very little long-term corporate planning for their new roles in the digital age. Most recognise that they have inadequate technical infrastructure in place to support their digital collections but are unsure what to do about this. There was little evidence of attempting to integrate new activities and roles into strategic planning or mainstream operations, and there is no understanding of the costs entailed in digital archiving.
The National Library of Australia has made progress in each of these areas. While some of our work is recognised internationally, I concluded that there was scope for us to promote our activities more widely. At the same time it was gratifying to receive recognition from a number of other national libraries for our achievements.
Outcomes
The overall conclusion I reached as a result of the Fellowship study trip was that the National Library of Australia is still at the forefront of digital archiving and preservation activity among national libraries. The Fellowship enabled me to confirm that the Library's approach to building its collection of significant Australian online resources and to meeting its objectives in the area of digital archiving, is still sound. A more cost-effective approach is not yet in place at any other national library or library related organization. However, activities are taking place that we must keep up with and take into account in our planning.
While other national libraries for the main part are not ready or able to investigate solutions to some of the more complex issues associated with digital archiving, such as preserving software, collecting databases and collaborating across sector boundaries, I believe these issues will be picked up by the well endowed research programs such as Library of Congress NDIIPP, and that solutions will emerge in time. In the meantime, the Library will continue to investigate at a modest level at least interim procedures to manage the issues.
With regard to the context in which we are managing our digital archiving responsibility, I concluded that Australia has some advantages over other countries visited including a fair copyright regime, a history of successful collaboration among collecting institutions, and a practical approach to dealing with new challenges. If we are successful in extending legal deposit to publications in digital form next year, this will be another major advantage. A major disadvantage is that the National Library of Australia does not have access to external sources of funding to enable us to expand the scope of our activities or to carry out major research - our recurrent budget determines the scope of our ambition. For this reason we must be very clear about our objectives but flexible in our approach. Another disadvantage is our distance from other national libraries doing significant work in the field of digital archiving. While new communications technology helps in sharing information, it is still difficult to collaborate in a practical way from a distance. We will nevertheless, continue to pursue opportunities for collaboration but preferably through structured channels such as the CDNL Digital Issues Group.
The observations made on the study trip and the conclusions reached have helped me to shape the Librarys digital archiving action plan for the coming year. The main actions that will be included in the work plan that have been influenced by the information gathered on the Fellowship, are as follows
Collaboration
- Participate actively in the three work areas of the CDNL Digital Issues Group: to develop best practice guidelines for extending legal deposit to digital publications; develop a common approach by national libraries to persistent identification of online resources; and to determine a strategy for maintaining access to software at the international level and to deal with database driven online resources.
- Contribute to the deliberations of the Internet Archive project to develop harvesting software specific to the needs of national libraries, if it eventuates.
- Using the agreement reached between the Royal Library (Netherlands) and Elsevier Science as a model, explore the possibility of a similar agreement with selected Australian publishers.
- Keep a close monitoring brief on developments with the Library of Congress National Digital Information Infrastructure Preservation Program and take advantage of any developments or opportunities for collaboration of benefit to us.
Collecting government publications
- Discuss with the National Archives and NOIE the idea of developing best practice guidelines focused specifically on digital archiving issues for Commonwealth government agencies, and assess the relevance of the Canadian Information Management Maturity Model tool to this purpose.
- Work with NOIE and some individual agencies to explore cost-effective options for streamlining and broadening the collecting of Commonwealth government online publications.
Approaches to collecting and providing access
- Explore the automatic generation of descriptive metadata that can be used to search for and locate specific resources. (We will focus on Commonwealth government resources and will draw on the work being done in this area by the Danish Bibliographic Center and the National Library of New Zealand. We will also assess the Nordic Web Archive tool set when it is released for public use early next year.)
Implications for the wider APS
Safeguarding documentary heritage in digital form remains an expensive undertaking for national libraries. It is an evolving area of responsibility, in many respects still in its infancy. Many future costs, such as maintaining on-going access through preservation action, remain unknown. Until scaleable and sustainable technical systems and automated procedures are available to support collecting, preservation and access, national libraries will be limited in the scope of what they are able to achieve. This is currently the situation for the National Library of Australia.
This conclusion, as well as other findings from the study trip, has implications for the wider APS. The online publishing output of government agencies forms an important component of the national documentary heritage. Agencies must be active partners in ensuring it remains accessible to the public into the future.
Commonwealth government agencies having been operating within the whole- of -government policy framework established and monitored by the National Office of the Information Economy (NOIE) for some time now. They have also received extensive guidance from the National Archives on policy and procedures for managing web based information. However, from our experience with working with agencies it would appear that, on the whole, the specific issues associated with managing online resources for long-term public access (such as creating quality metadata, using a persistent naming system, managing versions of older documents, and having technical infrastructure in place to adequately store and manage resources for preservation purposes) are not well-understood or receiving adequate attention. This is not due to an unwillingness to address the issues, rather to uncertainty or confusion about what to do.
In order to foster a greater level of collaboration between the key stakeholders, and to raise awareness of practical steps that can be taken to implement sound information management practices, a seminar was held at the National Library in July this year. The seminar, organized jointly by the Library, National Archives and NOIE, was called Keeping government publications online and a brochure with the same title was published (http://www.nla.gov.au/guidelines/govpubs.html). The seminar was very well attended and stimulated a considerable amount of discussion. The Library concluded that it must build on the momentum of good will established by it.
To reinforce the message conveyed by the seminar, it would be beneficial for NOIE to highlight issues associated with on-going public access to government information in whole-of-government policy, and to include specific strategies in their work program to address these issues. To this end, the Library will approach NOIE to explore opportunities for research into cost-effective options for collecting and managing government publications and for developing exemplar projects. As well, we will discuss with the National Archives and NOIE the idea of developing best practice guidelines and standards for the management of information in online form, drawing on the work done in Canada through the Information Management Maturity Model. At the same time, the Library intends to work on a one-to-one basis with a couple of Commonwealth government agencies next year to explore whether it is possible to make our current procedures for collecting online government publications more streamlined. The outcome of this exercise could feed into development of best practice guidelines.
Without active participation by Commonwealth government agencies in ensuring important information they make available to the public in online form is protected for on-going access, a considerable amount is likely to be lost to future generations. The Library is aware that already it is not collecting as widely from the sector as it would like to do. However, it is not possible to increase the scope of our collecting without additional external funding, a greater level of collaboration with government agencies, and greater use of automated procedures that still need to be developed.
The desirable model for safeguarding government information in online form is one based on partnership. To carry out their roles effectively in the partnership, agencies that have not already done so would need to commit resources and expertise to the digital archiving aspects of their online publishing activities. This would involve not only staff resource but also investment in appropriate technical systems and skills.
The National Library is very willing to work with agencies to share our own experience and expertise in this area.
Conclusion
While the Fellowship study trip did not reveal solutions to the many issues that the Library is now confronting in carrying out its digital archiving responsibilities, it helped considerably in clarifying the course of action the Library must pursue over the next couple of years. It was also very useful in sharpening my understanding of the international context in which we are operating and I am much better informed about what is actually happening elsewhere in the world among national libraries.
The Fellowship was as also enormously rewarding professionally and personally. It was stimulating to meet so many dedicated and enthusiastic colleagues who were eager to engage in discussion on the issues I was concerned about. It was also very gratifying that the progress the Library has achieved was widely acknowledged and respected by other national libraries.
I would like to thank the National Library for supporting my application for the Fellowship and time away from work during the study trip. I would also like to thank the Australian Public Service Commission for the wonderful experience provided by the Fellowship and to offer the view that the Fellowships program provides motivated senior executives with an invaluable opportunity to enhance their contribution to public service in Australia. It is an opportunity I would recommend highly to colleagues.
Pam Gatenby
Assistant Director General
Collection Management
Division
National Library of Australia
9 December 2002
[1] The National Bibliographic Database (NBD) is a database describing over 35 million documentary items held in over 1,100 Australian libraries. The database is provided through the Kinetica service by the National Library of Australia and is accessible by Australian libraries and their users.
[2] The Internet Archive is a not-for-profit private organisation located in San Francisco, which has been doing regular gathers of the whole Internet since 1996. Their mission is to ensure important heritage information in online form is not lost to future generations and they wish to work with national libraries and other collecting repositories to do this. Earlier this year they issued an invitation to some national libraries to participate in a 3-year project to develop gathering software that is specifically tailored to the needs of national libraries.
[3] Legal deposit is a statutory requirement that exists in most countries that requires publishers to deposit at least one copy of their publications with designated repositories, one of which is usually the national library.
