This paper was presented by Pam Gatenby at the IFLA International Newspaper Conference 2008 which was held at the National Library of Singapore.
Background to newspaper publishing in Australia
The history of newspapers in Australia reflects the history and development of the country, from its colonial origins to its current day membership of the global community. Australia’s history is short compared to most countries so it is not unreasonable for Australian libraries to aim to preserve a comprehensive record of newspaper publishing from the time of white settlement of the country to the current day. Early Australian newspapers are among the few remaining resources that provide contemporary accounts of how the colonies were governed and of key historic events that shaped the nation such as the first encounters with Indigenous Australians, land settlement and the discovery of gold. They reflect the concerns and circumstances of our ancestors and are heavily used in most Australian research libraries to support historic enquiry.
Australia’s first newspaper– the Sydney Gazette and New South Wales Advertiser - was published on 5 March 1803. It was a government gazette published by authority of the Governor of New South Wales. It distributed official announcements, shipping news, excerpts from foreign newspapers, and local social news. In each of the other Australian colonies, the first publication was also a government gazette. By the end of the 19th century several metropolitan, provincial and suburban newspapers were being published and weeklies were starting to appear. These played an important role in bridging the distance between city and country and in fostering Australian creative writing at a time when the book publishing industry was still in its infancy. The most famous weekly was the Bulletin, established in 1880, which nurtured a distinctive, radical national literature. (Unfortunately the Bulletin ceased publication this year due to declining sales.)
Many of today’s main newspaper titles can trace their origins to publications from the colonial era; for instance, Australia’s longest running title, the Sydney Morning Herald, was first published as the Sydney Herald in 1831. However, the history of newspaper publishing in Australia is marked by competition, mergers and “take overs” which many titles have not survived.
Australia now has the most concentrated print media ownership in the western world. The Australian Press Council in its 2006 report, the State of the News Print Media in Australia, notes that in 1923 there were 26 metropolitan dailies owned by 21 proprietors and by 1987 there were three major proprietors as well as a small number of independent publishers. Today, four companies own most of the newspapers in Australia with News Ltd (the biggest) controlling 68 per cent of the market.
The Press Council’s 2007 supplement to its 2006 report states that currently there are 52 daily English-language newspapers (two national, 13 metropolitan, 36 regional and one suburban) and eleven metropolitan Sunday newspapers published in Australia. In addition, hundreds of suburban and community newspapers are published at different intervals and many of these contain comprehensive local news and have substantial circulation. Besides the English-language press, there are more than one hundred newspapers in other languages, nine of them dailies.
As in many other countries, the main newspapers published in Australia now have web sites, many of them including multi-media and interactive features. Newspapers report that visits to their news related web sites are increasing at a fast rate.
The historic nature and characteristics of newspaper publishing in any country have implications for how libraries attempt to preserve their newspaper heritage. In Australia, we are taking a national, collaborative approach to addressing the issues through the Australian Newspaper Plan.
The Australian Newspaper Plan (ANPlan) was established in 1992 but was then called the National Plan for Australian Newspapers (or NPLAN). The name changed to ANPlan in 2006 but for the sake of simplicity I will refer to it as ANPLan during my talk regardless of the time-period I am referring to.
The idea for a national plan for newspapers came from the State Library of South Australia, one of the few Australian libraries to run their own microfilming units. The initiative was taken up and endorsed by the Consortium of Australian State libraries (CASL) which has also had a name change and is now known as NSLA, or the National and State Libraries of Australia. (The role of NSLA is to provide a consultative and advocacy forum for state and public library services in Australia and to develop common policies and programs to advance these services.)
The broad objective behind establishing a national plan was to coordinate activity at the national level in order to maximise the effectiveness of limited resources available to preserve access to the country’s newspapers.
The ANPlan was managed by the State Librray of South Australia until 2001 when, following a review of the future directions of the program, CASL invited the National Library to take over responsibility for managing it.
ANPlan is a national, collaborative undertaking based on shared objectives, clearly defined responsibilities, and practical action. The National Library and each Australian State and Territory library is a member of the consortium and the National Library of New Zealand participates with observer status.
Each partner library has responsible for collecting, preserving and providing access to each newspaper title published in their jurisdiction – more specifically, to ensure at least one hardcopy of every newspaper published in their jurisdiction is retained in their collection for as long as possible and that a surrogate copy of every title is made to facilitate long-term public access at the national level.
I’ll talk about these responsibilities in more detail.
In the area of collecting partners are required to:
- collect hardcopies of all newspapers from their area of responsibility as published; and
- identify, locate and collect missing titles and issues.
All member libraries collect current print newspaper titles under legal deposit provisions and have acquisition programs in place to fill important gaps in their collections. Repatriation of missing titles to the library with primary responsibility for their preservation is one way in which gaps can be filled. However, even though repatriation is an agreed strategy that underpins ANPLan objectives, it is unpopular with some partners and can be difficult to manage. Libraries and their users can be reluctant to give up materials from their collections if they are used and the resources required to de-accession material can be a deterrent.
A recent initiative to fill gaps in newspaper collections is a search and rescue campaign which is using specially designed publicity material to draw media and public attention to the search for particular missing titles. Each library will run their own campaign and focus on their priority missing titles but they will use the same publicity materials to give the campaign a national identity. The campaign started a couple of weeks ago and has already attracted considerable media interest and triggered a pleasing level of interest from the public.
The second area of responsibility for ANPLan partners is preservation. Partners are required to:
- retain as long as possible one hardcopy of every newspaper published in their jurisdiction;
- create or purchase an archival standard master reproduction and at least one working copy reproduction of every title; and
- provide appropriate housing and management of all copies of every title.
The preservation strategy followed by ANPlan involves microfilming to archival standards and providing appropriate housing and treatment of the original print titles and the different generations of microfilm, to optimise their life expectancy.
To assist libraries meet microfilming standards, in 1998 the National Library published Guidelines for Preservation Microfilming in Australia and New Zealand. The standards are also available on our web site in the policy document, Retention and preservation of Australian newspapers.
This document covers standards for preservation storage of print and microfilm versions of newspapers, as well as the microfilming standards that should be followed. (These are based on current versions of recognised international standards.)
To support the ANPlan preservation strategy the National Library provides members who require it with free Cold Storage of archival standard preservation masters, based on a Deed of Agreement. The Library also provides funding support to member libraries to microfilm titles in their collections. A submission process is used to allocate funds with priority given to “at risk” titles. Since 2002, we have provided around $1.2 million for this purpose. Additional funding comes from the member libraries’ own budgets and occasionally from government grants.
Australian libraries have been microfilming newspaper titles for almost 50 years and it has long been considered a relatively easy means of capturing a reliable copy that can be managed for the very long term and facilitate access. A couple of libraries have their own microfilming facilities and there are a small number of commercial agencies providing copying services to the others.
However, two recent developments have led a serious challenge to the previously pre- eminent preservation role of microfilming with regard to newspapers, namely;
- the development of increasingly sophisticated digital technology for capturing, organising and presenting content, and
- the increasing costs and difficulties of relying on microfilm as an adequate preservation and access path for newspapers - i.e. older film does not comply with current standards, it is expensive to store adequately, is unpopular with users, and the equipment needed for access is likely to become hard to maintain and acquire.
These developments, as well as the increasing availability of newspaper titles in online form, were the catalyst for ANPLan to organise a workshop to explore the implications of digital technologies for newspaper preservation. I’ll return to this topic later.
The last area of responsibility for ANPLan partners relates to access. Partners are required to:
- catalogue all print and microfilm holdings of newspapers into the Australian National Bibliographic Database (ANBD) on Libraries Australia, and
- provide easy access pathways to the content of each title.
At the heart of the ANPlan is the belief that libraries should make it easy for people to locate and obtain access to their cultural heritage, including newspapers. Australian libraries have a long tradition of collaboration in resource discovery with the most obvious manifestation being Libraries Australia.
Libraries Australia is an online resource discovery service managed by the National Library which provides access to the holdings of around 900 Australian libraries through the Australian National Bibliographic Database (ANBD), as well as access to a range of other Australian and overseas databases. The public can search Libraries Australia free of charge through an easy, Google-like interface, and they can then link to easy “getting” options via copying and document supply services, when full-text is not available online. Several formats of material, including newspapers, can be searched separately or as part of an integrated search across many material formats.
There are currently around 11,200 records for Australian newspapers recorded on Libraries Australia - around 4,600 of these are for microform versions and 500 for online versions. However, it is not know how many might be duplicate records.
By describing the various versions of newspaper holdings held by partner institutions and information about the extent of issues held, the Libraries Australia database serves as a national register of holdings – both print, microfilmed and digital. However, this is a complex area of bibliographic control and the extent to which the Libraries Australia database represents the newspaper situation in Australia is unknown, mainly because it is hard to estimate the number of titles ever published in the country. Also, in many cases, it is not known who made and holds master copies of microfilmed titles so this information is missing, holdings information on records is not always up to date, and bibliographic records can be difficult to decipher. Standardised approaches to describing the different formats of newspaper titles have not always been followed – for instance, sometimes different records have been created for the different versions but other times all information has been recorded in the record for the original print version.
To address the problem of inconsistent bibliographic control of the different versions of newspaper titles, partners recently developed guidelines with the aim of making it easier to understand records in Libraries Australia and to record information about the different generations of microfilm and intentions to do preservation microfilming.
Another way of improving access to newspapers is, of course, to digitise them. This is very much on the ANPLan agenda and I’ll return to the topic later.
To coordinate ANPlan activities and to provide a national focus, the National Library contributes a dedicated part-time position to managing the Program and to maintaining the public website. The website provides information about the program and its objectives, the standards and guidelines followed and new developments relating to newspaper preservation in Australia and overseas. A members’ space and discussion list are also maintained.
ANPlan members meet by teleconference twice a year and we aim to have one face to face meeting as well. The meetings provide the opportunity to discuss issues and share information and also serve to build commitment to the shared work plan.
The basis of collaboration is through a 5-Year Plan, which sets specific goals for the period, serves as a basis for internal and external funding and provides a framework for reporting progress. An annual report on progress against the Plan is provided to NSLA.
The specific goals of the current 2005-2010 Plan are to:
- acquire and preserve access to 51 missing titles that are considered nationally significant;
- microfilm 12 nationally significant “at risk” titles;
- re-film several titles to a quality which will support digitisation; and to
- address a number of particular concerns that include checking the condition of masters, replacing acetate masters, improving storage arrangements for masters and for original newspapers, and reviewing the extent to which Libraries Australia is serving as a national register of Australian newspapers.
The 5-Year Plan also includes several actions that arose from the workshop to explore the impact of digital technologies that I mentioned earlier.
So what has ANPlan achieved so far?
During the 16 years that the Australian Newspaper Plan has been in place, considerable progress has been made towards protecting and making accessible Australia’s newspaper heritage. The coordinated, collaborative approach offered by ANPlan has been a great stimulus to sustaining progress against milestones, to developing national strategies aimed at eliminating duplication of effort, and to sharing solutions to common problems.
However, it is very difficult to come up with reliable statistical estimates of progress at the national level against the core ANPLan objectives relating to collecting, access, and preservation. This is because some partner libraries do not have the relevant information available and it can be complex and resource intensive to compile. Nevertheless, some general statements can be made about progress made as well as the issues still to be addressed.
With regard to microfilming for instance, in June 2006 a national audit of progress with microfilming programs was undertaken by ANPlan in order to help shape planning in this area. It revealed that, while at the national level there is still some way to go in preserving our newspaper heritage, steady and impressive progress is being made, with four out of eight libraries indicating that they had filmed between 70 and 97% of their entire newspaper collection to preservation quality. Access to the country’s newspapers has also definitely improved over the last decade. All partner libraries now catalogue current print newspaper titles onto Libraries Australia and most have catalogued their complete collections online. Also, standards supporting the core ANPlan objectives have been agreed and are, by and large, being followed; we have a much better understanding of the state of newspapers at the national level; and strategies are in place to address issues.
The issues that remain to be addressed as a priority are reflected in the 5-Year work plan and some of these I have already mentioned. To summarise, the key ones are as follows.
- There are many gaps in the record of microform copies of newspaper holdings in the register on Libraries Australia.
- The location, ownership and condition of some preservation masters are unknown as some key titles have been filmed over the years by commercial bureaux or by publishers.
- Some partners have identified quality control concerns with older microfilm that renders it less than useful for preservation, access or digitisation purposes and some have large collections of older film still need to be quality assessed.
- All partners are aware of important titles that are missing from their collections.
As well as continuing to deal with a range of issues associated with pursuing microfilming as a preservation strategy over the years, ANPlan partners must now come to grips with the impact of digital technologies on newspaper preservation and access.
As I mentioned earlier, in June 2007 the National Library held a workshop to address the broad topic and to identify strategies for dealing with specific issues of more immediate concern to ANPlan.
A catalyst for the meeting was a discussion paper prepared by Colin Webb, then Director of the National Library’s Preservation Branch, titled Roles of digitisation and microfilming in Newspaper preservation. This paper explored questions such as:
- the preservation potential of digital copies;
- the timeframe for considering microfilm a viable preservation medium; and
- the timeframe for digital copies to become the preferred preservation medium.
The paper reached the following conclusions.
(i) Digital copies could serve perfectly well as preservation master copies of newspapers once some concerns are met – for instance, the cost of capturing a high level of fidelity, the cost of storing uncompressed files, and ability to commit to an appropriate digital preservation plan to meet long-term sustainability requirements.
(ii) Microfilm is potentially subject to a number of factors which could seriously threaten its continued viability as a preservation medium but it is unclear when this might eventuate. The factors include for instance, withdrawal of suitable film stock and microfilming bureaux services from the market, withdrawal of industry support for access technologies, withdrawal of suitable microfilm storage facilities, and user rejection of the format.
(iii) Australian libraries are likely to have a mix of microfilm and digital preservation approaches for the foreseeable future, depending on the situation (or “states of being”) of titles in their collections, their budgets and capacity to manage digital collections for the long-term.
The paper raised a number of particular issues for ANPlan members which were the focus of discussion at the workshop. Most of the issues translated into actions for the 5-Year Plan which, once carried out, will enable ANPlan partners to take more informed steps towards shaping the future preservation strategy. The key actions are:
- develop practical standards and guidelines for digital capture of hard-copy and microform copies of newspapers, and for management, storage and preservation of digital newspaper files;
- develop guidelines on how others can contribute content to the National Newspaper Digitisation Program (I’ll talk about this program in a minute) ;
- investigate the issues involved in collecting, preserving and providing access to online newspapers and pre-press electronic versions of newspapers;
- develop better understanding of the future viability of microfilming; and
- develop a costing framework for comparing newspaper microfilming and digitisation costs.
Good progress is being made with these actions - in fact the guidelines, except those for user contributions, are now available through the ANPlan website.
As with many other countries, Australia has commenced digitising its newspapers though a number of local projects as well as a major national undertaking, the Australian Newspapers Digitisation Program. This Program is managed by the National Library under the auspices of ANPlan and like ANPlan, is a collaborative undertaking involving the Australian state and territory libraries. Information about the Program is available from the website.
Planning for the Australian Newspapers Digitisation Program commenced in 2006. The Program is staffed with a mixture of permanent, fixed term and casual staff, with staff from our Information Technology area and two senior manager positions dedicated to it.
The aim of the program is to build a database containing newspaper content from the first Australian newspaper issue in 1803 through to the 1954, when copyright comes into effect. We have started with one major newspaper from each state and territory and intend to extend coverage in the future by the addition of regional newspapers. The aim is to develop one national access point for all digitised newspaper content.
During the first phase of the Newspaper Digitisation Program, which is now underway, we aim to create up to 4.4 million pages of digital newspaper content by June 2011. Titles included in this phase are The Sydney Gazette; The Sydney Morning Herald; The Maitland Mercury; The Argus; The Courier‐Mail; The Hobart Town Gazette; The Advertiser; and The West Australian. The Program is funded by the National Library wish the assistance of some external donations – for instance, we were very pleased to receive a grant of $1 million from the Vincent Fairfax Family Foundation to support the digitisation of The Sydney Morning Herald.
The actual digitisation of the newspaper titles is carried out under contract by two commercial companies. One company undertakes the initial stage of the digitisation which is to convert newspaper microfilm into digital page images. To date, over 800,000 page images have been created. The most complex stage, carried out by the second company, involves the conversion of the digital page images into text‐searchable files through the use of Optical Character Recognition (OCR) technology and other processes including the “zoning” of the newspaper articles. (The zoning process involves defining the border of each article and creating links between parts of an article that may be separated).
While progress with establishing the newspaper digitisation program has been slow and at times frustrating, significant progress has been made over the last year. Particular challenges we have encountered include microfilm quality, OCR accuracy, zoning and categorisation of text, and quality checking procedures. However, management and workflow procedures, including quality assessment, have been established and specifications for OCR requirements have now been finalised following several months of iterative testing and refinement using sample data.
To provide access to the full text of digitised newspaper articles, a search and delivery system is being developed at the National Library. The search service will be offered free of charge to the public. Development of the database search and delivery interface is well advanced and we hope to release it to the public in late 2008. The interface includes advanced search features such as relevance ranking, clustering of result sets by date span, geographic coverage, article category and size, and title of newspaper. In addition, related resources such as pictures and published works retrieved from other Library discovery services are presented.
Through the Australian Newspaper Plan Australian libraries are working together to preserve and provide on‐going public access to their country’s newspaper heritage. While many issues remain to be tackled, significant progress has been made with many “at risk” titles saved and a much better understanding of the state of control of the country’s newspapers established. By pursuing agreed standards and strategies over several years and working together to address issues, Australian libraries are now well placed to take on the challenges that digital technologies present to libraries with the responsibility for saving their newspaper heritage.