Staff Papers

Digital Libraries: Constructing a Value in the Digital Age for Libraries and Clients

A paper presented by Roxanne Missingham, Information Services, at the Information Online 2001 Conference held in Sydney, Australia, 16-18 January 2001


"Truth" is to be understood to be a system of ordered procedures for the production, regulation, distribution, circulation and operation of statements.

Foucault p.74


Abstract:

In constructing library services in the digital age, libraries have moved across the basic paradigms of creation of collections (through acquiring digital resources - CD Roms, electronic journals and Internet materials); describing material in the collections (catalogues, web pages and portals) and creating online virtual reference desks. Digital libraries have emerged as three different bodies of theory and reality. To understand the value of digital library services, a number of models are reviewed. This paper investigates key concepts of measurement and value in digital libraries.

Value and evaluation are key concepts for libraries. In the digital age establishing the place of libraries in a sea of information, with search engines filling the role of virtual sharks, is critical to reviewing existing services, developing new services and establishing performance standards. Creating value and performance measures is topical and complex. At the ALIA 2000 biennial conference many papers presented views on value of libraries in a digital environment – including Brodie, Bryson, Mercer, Burgess & Lake, Dewe, Bundy, McAllister, King, Griffiths and others. All suggested that measurement of value was difficult, contextual and required a mix of qualitative and quantitative measures.

Traditional measurement of the value of libraries has revolved around two key methodologies- economic impact and cost benefit. As Cullen (p.22) notes the determination of a statistical basis to support any enquiries is complex "…an organisation may value its outputs report on all transactions e.g. circulation, statistics, reference enquiries, interlibrary loans, on-line searches etc. and set goals to increase these each year, and yet have little notion of whether these transactions are meeting the needs of it’s primary client groups."

Economic impact analysis have taken the perspective that libraries exist within the context of their organisations, providing services that essentially increase the efficiency of the organisation. An excellent example is that of Meg Pauls labour efficiency studies described in her manual for special libraries. The methodology consists of survey to determine how much time staff would spend searching for information if they did not have a library. This amount is then compared to the resources required for the library to provide the services. The net savings demonstrates a return on investment often of a significant proportion. Sassone has refined this model from the value of information based on time savings time salary, to a hedonic wage model Sj Pij ê MVj =(I+U)· Ci. The usefulness of such economic impact studies is limited by the fact that the resources available in organisations for research and libraries vary because of non-rational factors such as external economics and political organisational decisions.

Cost benefit studies have a chequered history in libraries. While it is relatively easy to determine the costs of outputs it is much more difficult to determine the benefits. At the height of its application some attempts were made to address tangible and intangible costs and benefits. The establishment of benchmarks or best practice for the libraries within sectors have been documented from process activities (Wilson & Pitman). For complex service resulting in an understanding of the value of library services benchmarks have not been developed. Underpinning this is a combination of problems of pricing information goods (Varian) and determining a dollar value for benefit independent of the cost of information goods.

Such a complex and unresolved history in determining the value of libraries in the print environment does not augur well for digital library evolution. Many articles are emerging on the problems of measurement of digital library services and use of digital library resources. This paper reviews the concept of value in library and information services, current digital service measures and the application on value measures in a three-tier library service.

What is Value?

For the cultural institutions including libraries value is a tricky concept. Is it the embodiment of truth? (Or perhaps beauty) Or is it the worth of an activity or function which warrants the continuation of funding for the institution.

Bryson suggests that value can be:

  • "an estimate of appraise of being worth a specified sum or amount,
  • a standard of value in an exchange
  • a measure of worth, importance or usefulness. The relative merit or status according to an estimated desirability of utility of a thing; or
  • the quality of a thing considered in respect of its ability to serve a specified purpose or cause or effect."

Based in economic theory this approach suggests that evaluation in terms of opportunity cost, imputed value and market costs can be made. Taking these definitions assessments can be produced, with some difficulty, in the print and the electronic world.

Alternative views exist on information value. Saracevic and Kantor have constructed taxonomy of library and information services using the principle that value can be attributed to the use of information, rather than to the intrinsic value of information. They suggest (p.539):

"We take it that value within some context describes a relation between an object or objects (be they tangible like products or intangible like information) and their worth, which may include their merit, benefit, impact, quality, utility/usefulness, desirability and/or cost. "

This approach is implemented using a questionnaire which evaluates users vocabulary to questions including "why did the user use the service" and "what did the user get out of the service" (p.553). Such an approach has a number of critical assumptions. The first is that the value expressed by users is in fact the value desired by the library, or the organisation in which the library is located. Secondly, it assumes that readers/users can characterise their use within the complex framework provided. Cultural values are not reflected in the taxonomy creating an unsatisfactory tool for cultural libraries such as the National or State libraries.

An alternative approach to conceptualising value in the performance discourse can be seen in the Knowledge Based Economy. The Department of Industry, Science and Resources (2000, p.150) suggests:

  • "Knowledge has three properties that are different from traditional goods and services which make it difficult to measure it using traditional; indicators:
  • Sellers do not (give up) the knowledge that they sell. Knowledge is automatically and permanently ‘vested’ in whoever acquires it.
  • Potential buyers have no use for additional units of Knowledge identical to what they already have.
  • Buyers cannot readily appraise the knowledge that they might acquire without actually acquiring it. (Carter 1996)

For these reasons it is difficult to price knowledge as we do normal goods and services".

Libraries in their role as knowledge access engineers (Campbell) and knowledge stores experience many similar difficulties. Valuation of the knowledge-based economy has reached a stage of established international indicators despite these limitations. The OECD has developed a set of indicators covering:

  • Production and technology
  • Human capital
  • Information and communication technology

In Australia the indicators have been further developed to encompass:

  • Indicators of structural change
  • Information and communication technology indicators
  • Human capital indicators

The ‘composite’ indicator developed by World Times and the International Bank Corporation, the Information Society Index, has provided a national comparative measure for countries in the OECD.

The approaches taken within the library sector are limited in application compared to knowledge based economy indicators. This is clearest in the lack of comparative qualitative, despite many sectoral projects such as ARL’s LibQUAL+/SERVQUAL for the academic sector (Cook) and the HALPR Index covering US public libraries (Hennen). The holistic contextual approach offered by knowledge based economy offers some potential as a model for library value research. The ability to use a new method to determine the value of libraries and their services is however significantly limited by the statistical information available on digital libraries. Qualitative and quantitative measures are explored in the next section to consider a basis for the selection of Information for the development of measures.

Digital Library statistics

Libraries have been able to collect quantitative and qualitative information since the Great Library of Alexandria. Information on:

Size of collection
  • Use of collection
  • Use of reference services
  • Cataloguing and acquisition

gives a basis to consider inputs and outputs. The quality of the service or access to the collection has been assessed through user satisfaction, assessment of ‘accurate answers’ and service quality (CAVL-CIRG p.7-9).

The fundamental model has been:

       

Efficiency

E.g. items delivered no. of reference enquiries

 

Desired

 

 

 

Organisation

 

Measured by

 

Effectiveness – e.g. timeliness

Defines

Outcomes

     
       

Quality, user benefit – e.g. user satisfaction

         
       

Cost of service

In the digital world the measurement of use for value assessment has two major dimensions. The first is the assessment of value by users through their use of collection items and the % of items, or reference information; they achieve, as opposed to that they wish to access. The gap between the users information needs and the information gained gives a negative measure of value, or more correctly value failure. Secondly, the value of the whole of the collection and services need to be assessed in relation to the desired outcomes of the library. Details of what search tools are most effective, which pathways to information are used and what impediments exist to access, are important information to improve library services.

Two scenarios follow which give indications of the complexity of collecting information on use in digital libraries. Imagine a user coming to the National Library to undertake family history research. They come to the library and search 5 databases on CD-ROM, networked in the building, to find details of their family ancestors. Three have no relevant information as the user selected wrong years (and perhaps the titles were misleading) and two provide dozens of records. Several print sources are not used which would have more quickly reached the information. The monthly reports will list all searches, not those that are successful and unsuccessful from the users point of view. The time spent searching and false drops are not recorded. In another scenario a reader in Broken Hill is undertaking family history research. The user locates the National Library’s web site, is unable to access the subscribed CD ROM product, sends an email which results in a quick search on one database to verify that it has relevant records and a referral to the local public library. Two simple transactions for which only the reference enquiry has timeliness recorded, as well as placing the name of the enquirer into a database for possible selection to participate in a user survey as additional measure.

Given the complexities of interpreting digital use what information is available to library staff in 2000? The ICOLC guidelines require database, digital journal and e-book providers to report:

  • Number of queries (searches)
  • Number of menu selections
  • Number of sessions (logins)
  • Number of turn-aways
  • Number of items examined (i.e. viewed, marked or selected, downloaded, emailed, printed) to the extent these can be recorded controlled by the server rather than the browser.
  1. Citations displayed
  2. Full text.

From experiences with electronic information suppliers this is an early wish list rather than an immediate deliverable. Currently most CD-ROM networked products, database, ejournal and ebook services can provide:

  • Number of queries (searches) broken down by component e.g. database or journal
  • Number of sessions.

So how does this information compare to that we have been used to collecting in the print environment? While there are many similarities between measures available for print and electronic libraries there are also significant differences in the scope and nature of digital statistics, as the following table indicates.

Measure of collection use

Print

Electronic

Efficiency

Collection use – number of items used (loans, inter library loans)

% of collection used

Collection – number of items in the collection used/accessed,

Information may be available to title level

Effectiveness

Collection retrieval:

  • Success rate in filling requests
  • Timeliness

Collection retrieval:

  • Timeliness – length of search times

Quality

  • Degree to which information need is met (qualitative)
  • User satisfaction surveys
  • Degree to which information need is met (qualitative)
  • User satisfaction surveys

Cost

Defined library costs, can be divided by items delivered for unit cost

Defined library costs, can be divided by items delivered for unit cost

Value

Benefit to user – right information

Access to relevant information

If the fundamental statistics are comparable, what then are the unique issues in digital library measurement? Firstly information choices are made on a basis which does not necessarily relate to traditional areas of measurement. The user chooses from the options of a library or an Internet search engine, on the basis of their perception of likely success. This may be based on past experiences with the library, relating of the speed, accuracy, sufficiency, quantity or quality of information located. It may also be based on a general view formed from the press hype on search engines. Secondly there may be a lack of comparable data, particularly for newer electronic services, a lack of context for the data, incomplete data usage, lack of comparable content (some electronic services are just tables of contents, others will not give access to the most recent 18 months of journal issues) and different cost issues. Luther’s White paper on electronic journal usage statistics outlines concerns about all of these issues. Some of the most significant concerns with digital library statistics relate to problems in the collection of statistics and limitations of the statistics.

Collection of statistical information

At present the information which libraries can collect on the use of digital collections comes from individual suppliers, publishers and vendors. Many libraries, including the National Library of Australia, subscribe to tens if not hundreds of electronic information services. In collecting statistics from the various sources, visits must be made either to separate web sites, or information obtained from email reports. The frequency for these services varies, but is most often available monthly. At the National Library we collect this information from a small set of publishers and vendors, because of the difficulty and time required to collect the statistics from each source. Luther suggests that some of the data is difficult to use to compare to pre digital services – for example in the print environment a users may have borrowed one conference proceedings but copied and used many papers. In the digital library each paper viewing, printing or downloading is recorded, producing multiple statistics for what is actually one access (read and print) limiting comparisons. Some vendors supply statistics for "searches", some for documents accessed and some for login sessions. No two vendors supply the same type of information making a composite view difficult. The fundamental difference between print and digital indicators here is the level of granularity of access. This has an impact of the quantity measure, making pre and post-electronic comparisons impossibly. Comparisons of value in the two environments using these statistics cannot then have a valid basis.

In addition to these complexities of the statistics themselves libraries need to consider the time required to collate and report on electronic journal use. If reports are sourced from every producer or vendor of all subscribed services the processing of this information takes considerable time. The tasks involved are read the reports, select the appropriate figure and then entering the data into a report. The combination of these tasks for several hundred electronic resources is a significant organisational cost. Determining usage when reports make take longer than a week to arrive (for example to Electric Library report has to be generated from a US server which cannot occur before around 8-10 days into the next month) means that other reporting schedules within a library may not be met.

Timely and effective reporting is an information need which goes beyond the listing of output reports from a wide range of services. For detailed analysis comparable reports are required, which can be fed automatically into a library wide, or consortia reporting system.

Limitations of statistics

Currently the statistics on use produced for electronic resources all contain number of sessions and number of searches. This information is however of limited use because from these statistics it is not possible to determine how efficient or how effective the searching has been, i.e. has the user found the right citations with accurate searching. Such information is critical if the library wishes to develop better searching skills in its users, by training, or better communication, or service alternatives, or development of better access mechanisms. The simplicity of the figures, which cannot, with IP authentication, generally be broken down by user groups or branch libraries, provides a real impediment to important information for the library. Additionally, adoption of practices which ensure the privacy of data relating to individual users, means that quantitative research through statistics supplied by producers is limited. Qualitative research is essential for the sort of analysis of user behaviour required for collection and service development.

For digital libraries the challenges in assessing the value of their library collections and services begins with the simple collection of statistics. To take the next step of combining the statistics to develop a value measure required an understanding of the goals of the user and the library in delivering a digital library services. For this it is useful to consider a model of library service.

3 tier library service

Library services come is various shapes and sizes, depending on the commitment of the library and the nature of the library clients. Libraries are not a one size fits all service. The 3 tier model has been applied in Australian research libraries such as the Australian War Memorial, State Library of New South Wales (Burgess and Lake) and State Library of Victoria (Mercer). It can also be applied to the National Library of Australia (Missingham).

The 3-tier service models is best articulated in Ferguson (p 305) who typifies the services as:

Onsite/Remote service matrix

 

Services available on premises

Services available to remote users

Staffing

First tier gateway services

Basic use and finding questions related to core information resources, supported software and hardware trouble reports

Asynchronous user aids, search engines etc.; web service interface, telephone etc., to Customer Support Center

Paraprofessionals, students

Second tier intermediate services

General research support and initial triage of complex software and hardware issues, referral to experts on call or by appointment

General e-mail reference and computer consulting; general support of research communities by assigned liaisons; referral to experts on call or by appointment

Paraprofessionals, computer consultants, librarians

Third tier expert and specialized services

Subject or resource experts by appointment or during office hours

Subject or resources experts by appointment or during office hours

Librarians and computer consultants with higher-level subject or instructional expertise

With this breakdown of services it is possible to consider value within each tier of service. Value from the Libraries and users perspective still consists of factors such as quality of service, timeliness, and degree to which the information needs is met and user satisfaction components. Digital library services qualitative measures are proposed as follows:

 

E-service

Measures of use

Measures of value

First tier gateway services

Readers guides/finding aids, catalogues, unmediated access to the collection

No of web pages, reference tools, e materials accessed

Benefit to users, impact on research

Second tier intermediate services

General research enquiries

No reference enquiries, email, face to face, web

Access to relevant quality resources/collections,

Third tier expert and specialized services

Expert reference services

Number and nature of enquiries

Access to relevant quality resources, building of information skills

The current measures of electronic use of collections and digital services can provide the basic dimensions for usage measurement, but are not yet sophisticated enough to provide a basis for value assessment. The most relevant approach to value assessment for virtual reference can be seen in the paper "Facets of quality for digital reference services" by VRD. This document addresses issues of levels of service quality, timeliness, accessibility, response policies, authoritative service, expertise, privacy and review mechanisms.

Value in the use of materials has a number of components, as with reference services. In the print world the articulation of the elements has been described in readily measured statistics, including user satisfaction, number of items used, timeliness and costs. The basis of measurement has been use, a quantitative surrogate for relevance and quality. Value is very difficult to assess value in digital libraries using such a measure. Print use has faced similar difficulties, setting a continuing challenge to libraries.

Conclusion

The concept of value is a complex one for libraries, knowledge activities and cultural institutions generally. Like truth, value is in the eye of the beholder, be it a user, library manager or stakeholder of the library. Three approaches to assessing value have been described in this paper, together with a description of possible interpretation in light of a 3-tier service model. With the development of digital libraries – both in terms of access to digital collections and digital reference measurement and assessment has become more complex. Despite this the simple value paradigm of Ranganathan, re-interpreted by Crawford and Gorman holds true:

Books (and electronic resources) are for use.
Books (and electronic resources) are for all; or, Every reader his book
Every book (and electronic resources) its reader.
Save the time of the reader.
A library is a growing organism.

One model not explored in this paper offers a complex approach to value assessment, the Balanced Scorecard. Its grouping of qualitative and quantitative measures from the perspectives of users, stakeholders, finance, process improvement and learning and growth forms the basis of a new approach. Information on its application is covered in Wilson and Pitman. The Balanced Scorecard has been applied at the National Library of Australia in 2000.

In 2001 the decisions made by users to access collections and services in libraries depend on many factors. Libraries cannot be considered in isolation. The Internet is a new competitor, as the worlds largest library. However it is without the fundamental principles of collection development, a targeted audience and support services such as reference tools and reference librarians. This choice is offering alternative approaches to value for users. We need to understand the values of users, because if our users do not value us who will? The challenges is not just how to measure the current value and relevance of our services to users, but how we can make libraries relevant to the emerging group of researchers, developing their knowledge of libraries as well as their information skills.

Foucault suggests that information is power and the language of power is used to reshape our cultural understandings. Libraries need to take part in the debate on the value of knowledge of their collections/services to create a patch of sanity in the discourse of information communication and technology.

Bibliography

Association of Research Libraries, New Measures: Developing Statistics and Performance Measures to Describe Networked Information Services and Resource for ARL Libraries: Discussion Prospectus. Washington, D.C., Association of Research Libraries, 2000

Bishop, Ann Peterson, Logins and bailouts: Measuring Access, Use, and Success in Digital Libraries, 1998, Journal of electronic publishing, vol 4 no 2

Brodie, Maxine, The impact of change on research libraries: the State Library of New South Wales, in Proceedings of the ALIA2000 biennial conference, 2000, http://www.alia.org.au/conferences/alia2000/proceedings/maxine.brodie.html

Bryson, Jo, Value and Performance in the IT society, in Proceedings of the ALIA2000 biennial conference, , 2000, http://www.alia.org.au/conferences/alia2000/proceedings/Jo.Bryson1.html

Burgess, Kerrie and Lake, Therese, Value, Performance and Relevance, in Proceedings of the ALIA2000 biennial conference, 2000, http://www.alia.org.au/conferences/alia2000/proceedings/burgesslake.html

Cook, Colleen, LibQUAL+: one instrument in the new measures toolbox, 2000, ARL newsletter 212, http://www.arl.org/newsltr/212/libqual.html

Cotter, Rosemary, Cunnington, David, Fisch, Eva and Paton, Barbara, Final report: CAVAL Reference Interest Group Working Party on Performance Measures for Reference Services, http://www.caval.edu.au/crigwppm.htm

Crawford, Walt and Gorman, Michael, Future libraries: dreams, madness & reality, 1995, Chicago, American Library Association

Cullen, Rowena, Does performance measurement improve organisational effectiveness? A postmodern analysis, 1999, Performance measurement and metrics, sample issues, August p 9-30

Department of Industry, Science and Resources, Knowledge-based activities: selected indicators, 2000, Canberra, Dept. of Industry, Science and Resources

Department of Industry, Science and Resources, Measuring the Knowledge-based economy: How does Australian compare? Vol 1 main report, 1999 Canberra, Dept. of Industry, Science and Resources

D-Lib Working Group on Digital Library Metrics, http://www.dlib.org/metrics/public/index.html

Foucault, Michael, The Foucault reader edited Paul Rabinow, 1984, London, Penguin

Ferguson, Chris, ‘Shaking the conceptual foundations,’ Too: Integrating research and technology support for the next generation of information service, College & Research Libraries, July 2000 p 300-311

Hennen, Thomas J., Great American public libraries: HALPR ratings, round two, 1999, American libraries, vol 30 no 8 p 64-78

International Coalition of Library Consortia (ICOLC). Guidelines for Statistical Measures of Usage of Web-Based Indexed, Abstracted, and Full Text Resources. 1998, http://www.library.yale.edu/consortia/webstats.html.

Lancaster, F. W., Are evaluation criteria applied to "traditional" libraries equally applicable to digital libraries?, http://edfu.lis.uiuc.edu/allerton/95/sl/lancaster.html

Luther, Judy, White Paper on Electronic Journal Usage Statistics, Washington, DC, CLIR, 2000 http://www.clir.org/pubs/reports/pub94/contents.html

McAllister, Moyra, The Bottom Line: Performance Measurement in a Corporate Library, in Proceedings of the ALIA2000 biennial conference, 2000, http://www.alia.org.au/conferences/alia2000/proceedings/moyra.mcallister.html

Mercer, Prue, Changing our future: issues in leadership and management skills and the information profession, in Proceedings of the ALIA2000 biennial conference, 2000, http://www.alia.org.au/conferences/alia2000/proceedings/prue.mercer.html

Missingham, Roxanne, Virtual services for virtual readers: reference reborn in the E-library, in Proceedings of the ALIA2000 biennial conference, 2000, http://www.alia.org.au/conferences/alia2000/proceedings/roxanne.missingham.html

Paul, Meg, Rylatt, Diana, Atkins, Clare, The winning package : ACLIS special libraries advocacy project, 1996, Canberra :ACLIS

Poll, Roswitha, The House that Jack Built: the consequences of measuring, 1999 Performance measurement and metrics, sample issues, August p 9-30

Saracevic, T. and Kantor, P. B. Studying the value of library and information services. Part I. Establishing a theoretical framework, 1997, Journal of the American Society for Information Science, 48 (6), p.527-42

Saracevic, T. and Kantor, P. B. Studying the value of library and information services. Part II. Methodology and taxonomy, 1997, Journal of the American Society for Information Science, 48 (6), p.543-63

Sassone, Peter G, Cxost-benefit methodology for office systems, 1987, ACM transactioons on office information systems, vol 5 no 3, p 273-289

Thawley, John (ed), The Value of library and information services: papers presented at a seminar held in Melbourne, September 19 1990, 1992, Melbourne, CSIRO, Information Services Branch, Library Network Services

Varian, Hal R. Economics and search, 1999, SIGIR Forum, Fall

Varian, Hal R. Pricing information goods, 1995, Ann Arbour, Mi, Department of Economics, University of Michigan

VRD Facets of quality for digital reference services, version 4, 2000 http://askeric.org:6666/vrd/research.html

Wilson, Anne and Pitman, Leeanne, Best practice handbook for Australian university libraries, 2000, Canberra Dept. of Education, Training and Youth Affairs

Wilson, Anne, Pitman, Leeanne and Trahn, Isabella, Guidelines for the application of best practice in Australian university libraries: intranational and international benchmarks, 2000, Canberra Dept. of Education, Training and Youth Affairs

Back to the top