Staff Papers

Website Usability as a Key Performance Indicator Monitoring and evaluating online services to improve customers’ experiences … PictureAustralia

www.pictureaustralia.org

A paper by Debbie Campbell, National Library of Australia for the Performance Measures for Government Online Conference 12 September 2001


Abstract

pictureaustralia.org is a collaborative effort coordinated by the National Library of Australia to provide pictures online. The National Library undertook an extensive usability test before launching the PictureAustralia web site and is in the process of undertaking a major review of the site to improve customers’ experiences. The site was initially tested against a range of scenarios representative of the various target user groups.

Introduction

PictureAustralia went live in September 2000, with seven participants: the State Library of New South Wales, the State Library of Tasmania, the State Library of Victoria, the Australian War Memorial, the National Archives of Australia, the Fryer Library of the University of Queensland, and the National Library of Australia which also hosts the service.

The launch in September 2000 came at the end of a cycle which took place over the preceding two years. It included the development of a demonstrator service, three months acceptance testing, and a special usability test before going live. The demonstrator was shown at a conference showcase, and although it contained only 10,000 images, the concept of providing a single Web entry point to the distributed online pictorial collections of several cultural agencies simultaneously was very well received by the service provider community.

The acceptance testing phase was conducted by the inaugural participants, and focused on Web performance issues – whether the display of images was slow because they were ‘bandwidth-hungry’, how the software platform behind PictureAustralia coped with a significant load of simultaneous Web searches, and whether there were major issues relating to data or image quality display.

The special usability test was required to identify ‘showstoppers’, any impediment which may have affected the clients’ experience and given them a bad impression of the Web site in its early life. For this, a more independent assessment was required, and the National Library commissioned a local company, User Insite, to conduct an heuristic evaluation of the PictureAustralia service.

Heuristic evaluation

An heuristic evaluation is defined as “is a usability engineering method for finding the usability problems in a user interface design so that they can be attended to as part of an iterative design process. Heuristic evaluation involves having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the "heuristics").1 As the PictureAustralia development team and the participants in the service had all worked closely with the demonstrator to establish the service concept, pre-conceived notions may have crept into the final stages of development and testing. In addition, the library community itself has long-established ideas about how information systems should support discovery and access. The development team tried very hard to introduce other opinions about design and navigation from the outset, for example, by using local graphic design companies to provide the look-and-feel of the Web interface and the service logo.2

The heuristic evaluation was a second important step in this independent process. Its purpose was to: “alert the PictureAustralia team to any difficulties of the website from a user perspective.” It was conducted in three parts – firstly, a sociocultural analysis; secondly, a Quester analysis of possible major audience groups; and thirdly, some modest benchmarking against other similar services.

Sociocultural analysis

The sociocultural analysis examined how the Web site was constructed and whether it supported a range of user approaches to it. The analysis revealed several unexpected (to the development team) ideas as well as confirmation of some choices:

  • the choice of the Ned Kelly logo provided an opportunity to impart knowledge of a significant piece of Australia’s history as well as an artistic component. New images of, and extra details about, the work of Sir Sidney Nolan were subsequently linked into PictureAustralia;
  • the image gallery on the home page may be scrutinised in terms of its arrangement or juxtaposition of related images (a European and male-centric vision of Australia’s history), when the development team had merely chosen the images to represent a cross-section of all images in PictureAustralia. The image gallery is periodically changed;
  • the home page design endorses the visual pun which is the name of the service;
  • the name of the service creates an expectation of being all-encompassing. The PictureAustralia development team had already identified this as a possible risk, and the vision for the service always included approaching any agency managing digital images, not just libraries. However, there are areas which will take longer to address, such as the provision of more images related to indigenous culture. Recently, the team has started to negotiate for the provision of images from both indigenous agencies and overseas cultural institutions which hold significant works relating to Australia. Although PictureAustralia already provides access to half a million images, the National Library’s target is one million.
  • engagement and continued interaction of the client in the service would not be sufficient through periodic online surveys. So the idea of proposing new trails was born. Trails are sub-sets of the images in PictureAustralia, pre-selected according to a particular theme. They require an analysis of the metadata describing the pictures, in order to ensure no false matches or too many results. The ability to suggest new trails has proved to be one of the strengths of the service – they are suggested by school classes doing projects, higher education institutions overseas conducting Australian Studies, and local historians interested in featuring their home towns.

Quester analysis

Quester scenarios, a trademarked device of User Insite, are conducted by proposing potential clients of the service, then stepping into their shoes and recording their navigational experiences. The scenarios ensure that all contents and functions are explored, rather than relying on or leading ‘live’ testers to find all pathways in an interactive service.

Eight Quester scenarios were applied, including some suggested by the National Library. As PictureAustralia is provided for use by all Australians, free of charge, the range of clients tested had to vary widely in age (from 6 to 86), sex, profession or hobby (from architect to historian to web surfer). The scenarios consequently identified a range of issues including the following.

  • the use of both legal and professional jargon on the Web site. Legal jargon was rife in the provisions for reproduction and copyright associated with both online re-use of images, as well as offline use of high quality copies. The development team has modified the wording of the copyright conditions and use page several times already, and will continue to do so to meet the obligations of the Digital Agenda Bill (2001) and other changes in legislation;
  • professional jargon is sometimes exposed when describing the functions of information systems – words like ‘thumbnail’, ‘metadata’, ‘boolean’, and ‘truncation’ are not commonly used. Sometimes we continue to use the words so that people become familiar with them, but we try to keep these uses to a minimum because they are not well understood;
  • ordering of images is not easily achieved. This issue highlights one of the features which makes PictureAustralia unique. It is a hybrid service, with centralised searching and distributed access. Back-office functions, such as ordering a reproduction-quality copy of an image, are provided at the individual agencies’ sites, not centrally. Although some of these functions start online, they vary considerably in how they work. This is partly due to the fact that the function is part of the larger business of the agency. Order forms may have to be posted or faxed in, and there is a different scale of fees and turnaround times for each agency. This generates confusion for the client who hasn’t realised how PictureAustralia works behind the scenes. Where possible, uniformity and some navigational aids have been applied, but the imminent introduction of e-commerce solutions has put more practical interim solutions on hold.
  • disappointment with image descriptions is increasing. The metadata for the images in PictureAustralia have been generated automatically from existing or legacy descriptive practices, as one way of making it easy for agencies to participate in the service. Some collections of images have been inherited without any descriptive metadata. However, there is usually a trade-off between supplying more digitised images and spending more time on their descriptions. Online exhibitions, which tend to feature a handful of images, are likely to contain a lot more contextual information as well. But this is only possible because a decision has been made to provide this level of detail, which is usually not possible with much larger collections.

Benchmarking

Comparing a Web service to other similar Web services, or like with like, in terms of navigation, structure and content can be difficult. PictureAustralia needed to find other services which ‘accommodate different levels of familiarity and expertise with the Internet itself and with searching and research skills’.3 This is an area where the strength of the collaboration between the participating agencies becomes evident. The expertise of the State Library of Tasmania in the use of metadata searching by a whole populace (Tasmania)4, and the State Library of Victoria in measuring search patterns on systems containing similar content5 has been called on several times over the life of the service.

The PictureAustralia development team also assessed other sites during the early phases of the service’s development, and acknowledged those sites providing inspiration.6

As skill levels change over time, and the use of the Web becomes even more ubiquitous with the introduction of the Australian government’s online portals, it becomes essential to periodically re-apply the process of evaluation. Any Web service which wants to continue to be relevant, current, and accurate must incur a process of continuous enhancement and measurement.

Other performance measures

The initial heuristic evaluation was important because of its focus on the clients of the service. However, PictureAustralia has received several unexpected accolades since launch from other sources as well. The first was winning the Arts category of the Financial Review's Australian Internet Awards 2000 in November 2000. PictureAustralia was chosen because it is an "excellent portal for Australian cultural material of a pictorial nature", it "seamlessly integrates multiple multimedia databases into an elegant visual experience for the customer", and its "use of metadata is excellent"7.

The second accolade was the recent emulation of the service concept by the National Library of Canada.8 They were able to fast-track the introduction of their service, Images Canada, because they were able to learn from the development experience of PictureAustralia.

Closer to home, the evaluation of PictureAustralia was cited by the Australian National Audit Office as an example of best practice in evaluating online delivery in the preliminary stages of a service.9 In December 2000, a Victorian state government Arts Portfolio Leadership Award was given to PictureAustralia for “..provid[ing] remote access to some of Australia's most significant heritage collections.” 10

It would be very tempting to rest on the laurels of these accolades, but they are from peers, other service providers who understand the time required to provide high quality Web-based services and use collaboration to achieve them. The PictureAustralia participants in the service realised that it is very possible for peers to commend a service which clients are not interested in using, or will not revisit. So a second review, a major one, was recently conducted. This review encompassed the images provided by the newer participants in the service as well: the Australian Heritage Commission, the Nolan Gallery, the Lake Macquarie City Library, Museum Victoria, and the National Library of New Zealand.

A second evaluation

The preliminary evaluation resulted in an action plan which addressed the major showstoppers in the service prior to going live. However, it was essential to conduct another evaluation with ‘live’ users once the service had been available to the public for an extended period. The second evaluation took place in June 2001.

It was conducted using three mechanisms – Web site statistics, usability testing, and a satisfaction survey.

Web site statistics

The National Library uses the Web Trends software package for measuring the use of all of its online services. Some limitations were experienced with this software, especially for services which send clients to other sites as a search result. This was particularly the case for PictureAustralia, where it is difficult to establish user locality, duration of stay, and referrals to the participating agencies. It is offset by two other mechanisms – use of statistics gathered by the platform behind PictureAustralia11, and comparison using the other two mechanisms.

The statistics gathered by the platform showed that ‘aborigines’ (and variations on this term) is one of the most frequently searched topics. They also showed that the trails are often used. Many of the more commonly used searches reflected topical issues in the media. This data can be used to establish new trails, thereby encouraging revisits.

Usability testing

Usability testing replaced the Quester scenarios in the first evaluation. Ten people were chosen at random, encompassing a broad spectrum of age, profession and hobby. There were no discernible differences in behaviour on the basis of sex.

One of the findings out of the usability testing was that if a service is designed to look like a search engine, then people expect it to behave like a search engine. Unfortunately, the behaviours have to be different, because the search results are actual resources, not links (which may be defined as ‘shallow’ resources). Therefore, attempting to find a lowest common denominator may be self-defeating. Perhaps Web services of the collecting institutions have to look like, and behave like, the services of collecting institutions and not commercial search engines even if this reduces the size of the target audience.

The National Library is considering ways of meeting some of the functionality provided by search engines. For example, although the PictureAustralia participants are still developing their e-commerce functionality, a more uniform equivalent to ‘shopping basket’ functionality is being tested.

Online survey

The online survey was designed by User Insite with modest input by the National Library. The results showed a 97% satisfaction rate with PictureAustralia. Bouquets included recognition that the integration of dispersed collections of images is efficient in saving time (both virtually and physically), exploiting a technology to make our collections more accessible nationally, and its ease of use.

The ability to find the service itself rated strongly. The National Library has always made a point of creating awareness of new Web services by submitting information about them to well-known search engines. This is supported by the creation of Australian Government Locator Service (AGLS) standard metadata behind the home page, for inclusion in other sites as links. Overall, 36% of those surveyed discovered the service virtually. Word-of-mouth discovery has also proved effective, because each of the participating agencies is provided with publicity materials, and by use of the media – newspapers, radio interviews, and television.

There were strong preferences for discovering particular sets of images as well: Australian places (54%), historical subjects (49%), heritage sites or buildings (41%) and even unusual images (34%). This is useful data for preparing trails, and also for approaching new agencies, particularly where there is a shortfall in coverage by the service overall.

But there were a lot of considered comments in the online survey as well, showing a strong interest in the service. Brickbats included:

  • minimalist quantity and quality (usually incorrectly labelled) of the metadata describing the images. The PictureAustralia administration team encourages the identification of incorrect metadata, and passes the information onto the agency in question;
  • lack of uniformity in back-office function pages, such as the ordering form. Enhancements to the ordering form are being considered. Because of the facility provided by the Web, there is now a greater expectation from clients that a suitable end point is reachable with minimal clicks. Frustration levels rise when a client reaches an analogue impediment (print form and post it) at the end of a digital process.
  • The quality of some of the images is poor. While the PictureAustralia team has set no standard to be met in this area, the library community does share information about digitisation practices and standards12. Even though a common standard is used for the thumbnail images, JPEG (Joint Photographic Experts Group) Format, a bad photograph cannot be digitised into a good one via a routine scanning process. In addition, there is sufficient evidence to show that as long as accessibility is increased by digitising more images, higher quality images are not necessary, and may also have an adverse affect on bandwidth.13

It is interesting to note that a cross-sectoral service such as PictureAustralia will often face these sorts of conundrums. While not necessarily ‘dumbing-down’ quality (the original rich descriptions and images are still available through the linking capability of PictureAustralia), the introduction of a lingua franca such as the Dublin Core shows inequalities in metadata creation practices, in digitisation practices etc. This can be offset to a degree by encouraging adoption of and adherence to minimal standards, but can’t be eradicated completely.

After the evaluation

The second evaluation again provided important information about the expectations of clients for the service. The PictureAustralia administration team prepared an action plan to address concerns and schedule enhancements. While it is not always possible to continue applying large numbers of resources – human, financial, time – to services, the National Library considers this work to be meeting its strategic directions for 2000-2002, in particular, “to ensure that Australians, wherever they live or work, have access to information resources in Australia and overseas.”14

Performance measurement in the public service

The National Library’s performance measurement is based on a Balanced Scorecard approach. As part of this approach, statistics relating to the use of Web services are generated by the Web Trends software. During the refinement of our measuring processes, we have moved away from counting the number of hits, to counting the number of user sessions. Counting a hit, especially in a service such as PictureAustralia, is entirely misleading. It does not equate to an action by the client, rather it counts the number of graphic files such as the logo, the buttons, and the thumbnail images loaded into each page separately.

Our goal has been to aim for the number of user sessions, currently averaged at 300,000 per year. Is this sufficient – should we be counting the number of accesses, i.e., the number of searches conducted by each user to gauge their levels of interest ? The smallest number per annum would be 300,000 – one search per user. Would greater success by measured by the fact that they are conducting more than one search ? It’s hard to know whether they would do this because they have found a relevant image or because they can’t. The National Library is still looking into this as part of its performance measurement.

As part of outputs measurement, comparing a Web service to other similar Web services, or like with like, is often recommended as a performance measure for the Web services of Australian cultural institutions. The Commonwealth Collecting Institutions Forum produced a paper titled “Performance Information for Collecting Institutions15. It defines a set of measures for Commonwealth collecting Institutions, including some web usage statistics. But it does not define by what is meant by Web usage – a count of hits, user sessions or accesses. Until this clarified, PictureAustralia cannot be appropriately compared to other similar services.

During 2001, the National Library is conducting a review of all of its Web-based services. An early review of statistical packages revealed that there is no single piece of software which meets all of our measurement requirements, which we don’t believe are particularly onerous. It is likely that a combination of measures will be more successful, and this approach is endorsed by the evaluation outlined above.

And again

The usefulness of these processes, despite the malleable statistics in the performance measurement area, has been proven each time. The National Library is already thinking about its third evaluation of PictureAustralia.

Contact

For further information about PictureAustralia contact the PictureAustralia Manager, Danielle Freeman on 6262 1641 or dfreeman@nla.gov.au

Additional links

  1. cc. Ten Usability Heuristics, by Jakob Nielsen
  2. PictureAustralia information links
  3. Sample screens from the Virginia Tech Imagebase
  4. The National Library’s Balanced Scorecard performance measurement
  5. Accessibility considerations

1. Ten Usability Heuristics, available in Jakob Nielsen's Online Writings on Heuristic Evaluation at <http://www.useit.com/papers/heuristic/heuristic_list.html>

Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

Match between system and the real world
The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

User control and freedom
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

Error prevention
Even better than good error messages is a careful design which prevents a problem from occurring in the first place.

Recognition rather than recall
Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

Aesthetic and minimalist design
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

2. Information pertaining to the architecture and design of PictureAustralia is available at <http://www.pictureaustralia.org/about.html>.

More technical specifications for participating in the service are available at <http://www.pictureaustralia.org/contribute/index.html>.

3. Two pages from the Virginia Tech Imagebase, which also uses Dublin Core metadata and thumbnail images in the results sets, at <http://imagebase.lib.vt.edu/search.php>

4. The National Library’s Balanced Scorecard performance measurement is explained in broad terms at <http://www.nla.gov.au/policy/bsc.html>

5. Accessibility considerations

The National Library recommends the use of BOBBY for checking initial accessibility of images, tables or image maps in any Web service – where accessibility means that a user with a browser that does not support graphics (such as a text-only browser used by the visually impaired) can successfully navigate the page. Bobby is a free service provided by CAST to help Web page authors identify and repair significant barriers to access by individuals with disabilities. http://www.cast.org/bobby/.

Footnotes

1 Jakob Nielsen's Online Writings on Heuristic Evaluation, accessed on 26th July 2001 at <http://www.useit.com/papers/heuristic/heuristic_evaluation.html>.

2 Swell Design developed the interface, Kathryn Wright Design developed the logo.

3 PictureAustralia – Preliminary Evaluation, User Insite Pty Ltd, August 2000

4 Via the Tasmania Online service, at <http://www.tas.gov.au/>

5 Via the Multimedia Catalogue, at <http://www.statelibrary.vic.gov.au/slv/mmcatalogue/>

6 Ideas for the PictureAustralia service were inspired by (all accessed on 26th July 2001):
AmericanStr@tegy <http://www.americanstrategy.org/>
Archaeology Data Service <http://ads.ahds.ac.uk>
Australian Libraries Gateway <http://www.nla.gov.au/libraries/>
Cornucopia <http://www.cornucopia.org.uk/>
Helix <http://www.helix.dmu.ac.uk/>
Virginia Tech Imagebase <http://imagebase.lib.vt.edu/>
24 Hour Museum <http://www.24hourmuseum.org.uk/>

7 The Financial Review, Australian Internet Awards 2000, <http://www.webawards.com.au/>, accessed on 26th July 2001

8 Images Canada, at < http://www.imagescanada.ca/>, accessed on 26th July 2001

9 Internet Delivery Decisions, A Government Program Manager’s Guide, Australian National Audit Office, Commonwealth of Australia 2001. Available at < http://www.anao.gov.au/> under Publications

10 < http://www.arts.vic.gov.au/publications/enews1500.htm>, section 8, accessed on 26th July 2001

11 Blue Angel Technologies, MetaStar Enterprise < http://www.blueangeltech.com/>, accessed on 26th July 2001

12 A good starting point is < http://www.nla.gov.au/libraries/digitisation/>, accessed 27th July 2001

13 Digitising records for improved accessibility, Paul Macpherson, November 2000; accessed at < http://www.naa.gov.au/Publications/Corporate_Publications/corporate_publications.html>, 27th July 2001

14 Directions for 2000-2002, http://www.nla.gov.au/library/directions.html, accessed 27th July 2001

15 7 August 2000, arising out of Safeguarding Our National Collections, Audit Report No.8, tabled 28/09/1998, Australian National Audit Office, http://www.anao.gov.au, accessed 27th July 2001

Back to the top