Paper written by Colin Webb, Manager, Information Preservation and presented by Margaret Dent at the ALIA Rare Books and Special Materials SIG meeting in Sydney, 3 October 1996.
John Alexander Ferguson noted, in the introduction to volume 3 of his Bibliography of Australia, that "one strange feature of this period is the extraordinary number of 'ghost' books, that is, books to which a reference has been made in some other publication but which cannot be found today in any library or collection."
I wish I could present a claim to you that by some mysterious alchemy we had made these rarest of the rare re-appear, but I cannot. Instead, I do want to tell you about a slightly less ambitious, but more achievable, project that should help us look after and improve access to some of the genuinely rare, but extant, materials documenting nineteenth century Australia.
I need to make it clear from the outset that I am drawing heavily on two reports by the project coordinator, Ross Coleman of the University of Sydney Library, who is currently overseas. I have substantially modified Ross's papers, so I suppose I am responsible for any comments you might find contentious. I also want to thank Margaret Dent, who has kindly agreed to present this paper on my behalf.
In talking about the Ferguson project it would be easy to get into a lot of technical detail - at the present time, this feels like the most interesting and challenging part of the project, because it is what we are immersed in - but papers full of technical detail can be eye-glazingly boring, and there is probably virtue in talking about the concepts rather than the detail. It is unfair to suggest this, but I'm sure that Alan Ventress, who is also involved in the project, would be willing to discuss the details if you have any questions.
The way I want to approach this is to give you some background, and describe the project, and then talk about where we are up to and our experiences so far.
The Ferguson in the project title refers to the same Ferguson's Bibliography of Australia that I just mentioned, of course - the most comprehensive listing of titles published in Australia prior to 1900. It is important to remember that we are talking about a concept, a list of publications, rather than a single collection. This fact provides one of the fundamental reasons for the project, and many of our challenges.
Preservation is a concern with a lot of this material. Much of it is in surprisingly good condition, but much of it is poor, mainly because of use. A lot of it is rare, existing in only one or a few copies. The main response to the preservation problem over the years has been to do some repair work, worry about how the material is housed, and restrict access.
In recent years - let's say the past 25 years anyway - there has been growing interest in what we call reformatting - probably just a fancy name for copying - as a way of addressing both preservation and accessibility problems. Reformatting doesn't deal with either of them perfectly - it hardly solves the access problem, as anyone who stares at a microfilm reader or a computer screen all day will testify; and it doesn't amount to a complete preservation strategy, either - but reformatting to a high standard, taking pains to make the copy as user-friendly as possible, does offer some improvements in access to material that otherwise might have been withheld from use; and it does protect fragile or rare originals from the loving over-attention of users.
The Ferguson materials have long looked like a choice candidate for reformatting, and over the years there have been a number of proposals to microfilm all or part of them as a way of building an accessible and comprehensive virtual collection out of the bits and pieces scattered over many institutions. These proposals have always failed because the expected low demand for microfilm did not seem to justify the size and cost of such a project.
So, what happened to make a difference? The answer is complex, but it has a lot to do with the possibilities offered by digital imaging - capturing digital copies that can be stored, manipulated, and delivered using computers and related equipment. This promise was enough to encourage people from a number of institutions to try to design a feasible proposal for a limited, developmental - even experimental - project that could attract funding. This project, which in our loose way we call the Ferguson Project, but which is officially called the Australian Cooperative Digitisation Project 1840-1845, is the result. Its purpose is to provide large-scale digital reformatting of serials and fiction published in the period.
I now want to tell you a little about the project - who is involved, what we are trying to achieve, and how we are trying to achieve it, with some of the benefits that it should bring to managing, preserving and providing access to this rare 19th century material.
Some words from the project submission to the Australian Research Council may help give you a feeling for the vision behind it:
This project will create a unique research infrastructure in Australian studies through the digital conversion of Australian serials and fiction of the seminal period 1840-45. It will ensure access to, and preservation of, this material by a process of microfilming, scanning and networking. The first project of its kind in Australia, it will introduce international best practice, act as a technical benchmark, and provide a new level of resource sharing.
The success of the submission was a recognition that digital collections of otherwise rare source material could be important research infrastructure for academic use.
The project is a truly collaborative effort. There are four core partners - the University of Sydney Library, the State Library of NSW, the National Library of Australia, and Monash University Library, each of whom is making a large contribution in terms of time, expertise, and cash. Each of the project partners brings particular experience and interests to the project, and all will benefit by acquiring technical and management skills. Common concerns for all participants include selecting content, preparing material, design, technical production and conversion, user focus issues and detailed management of the project.
The project is substantially funded by the Australian Research Council, which provided a research infrastructure grant for 1996, in response to a submission which included statements of support from the four partners, two other State libraries, four other University libraries, and from bodies such as the National Preservation Office, the National Centre for Australian Studies, the Macquarie Dictionary, and a multimedia firm. The project has academic and technical advisers from nine institutions, including Cornell and Yale University libraries in the US. Collectively, this represents an extensive pool of support and expertise committed to making the venture a success.
It is worth emphasising that as an ARC infrastructure project, the needs of the scholarly community in terms of content and useability are given a very high priority - getting the user focus right is fundamental.
What should be included in the project was refined through several discussions with a focus group of scholars in nineteenth century Australian studies. It was important to define a fairly narrow time period to be covered, so that a realistic proposal could be developed. There was early consensus that the target period should be 1840-1845, a time which saw the emergence of an Australian colonial culture and identity, marked by an upsurge in local publication. The scholarly reference group were also emphatic on the need to do this as a block exercise - to do all titles in the period rather than be selective.
The decision to concentrate on serials followed testing of a small sample of books from the Mitchell Library, which highlighted a number of physical problems. The focus group recognised serial literature as an important untapped primary source, which had also been identified as a priority area for action by the National Library and the Council of Australian State Libraries.
The Ferguson Bibliography lists seventy-five serial titles as having commenced in the period 1840-45. (Only six of these continued publication after 1855, the general cut-off date for the project.)
A small component of published fiction was added to the project, building on the interest of the National Centre for Australian Studies at Monash in developing fully-searchable text of nineteenth century fiction. Only four works of fiction were identified from the period.
We know from experience that any reformatting project attracts compromises, because it is almost impossible to foresee all the dilemmas that will arise. This makes it particularly important to build up a sound conceptual framework to guide project managers towards the client-focused objectives of the project - if you don't have this, it is just too easy to end up a long way from the aims you set out to serve. I want to talk about that conceptual framework now.
The technical concept is based upon what we and others have described as a "hybrid" approach to reformatting - a process of microfilming and digitisation to provide long-term preservation and digital access at the same time. It is an attractive approach that recognises the needs of present users while trying to ensure the material can be maintained for future use.
I have already mentioned the benefits of digitisation in providing simultaneous, almost instant access, anywhere in the world, to images that can be easily searched, copied, printed, zoomed, manipulated - all the things that networks and software let us do with digital data.
The importance of microfilm in the hybrid approach is that it provides a chemically and physically stable storage format, with an anticipated life of around 500 years. It is relatively technology-independent, not requiring computers or special software for access, but is able to be read with the assistance of a lens - something that digital images, with their total dependence on changing technologies, cannot match. Microfilm can be used as a source for duplicating throughout its life. It does not need repeated intervention or refreshing for maintenance, and its storage requirements are well understood. Basically, it is a proven technology with which we have a lot of experience, with established standards and some reliable vendors.
Given the decision that we wanted the benefits of both microfilming and digitisation, there was the question of what should be done first.
Put simply there are two models of the hybrid approach. Cornell University Library have developed a model which is to scan first and then produce microfilm from the digital images. This has the advantage of allowing the digital image to be manipulated in order to get the best possible microfilm image. On the other hand, Yale University Library have reversed the approach, filming first, and then scanning from the film. These complementary models have generated important information on standards, comparative costs, and techniques. Anne Kenney and Paul Conway, the coordinators of these projects respectively, are technical advisers to our Ferguson project.
We have chosen to follow the Yale model, and film first. It is an approach that suits the rare and fragile material we are dealing with, and it allows us to take advantage of the extensive experience we have had with preservation microfilming.
A number of aspects illustrate the strength of this approach and the options it can open up.
For example:
Concern about the physical integrity of the original materials was a motivating factor for the project partners; it was also raised early on as a key issue by the academics advising the project. Access was to be enhanced, but the original material was also to be protected. It is reasonably common for non-preservation oriented projects to virtually destroy the originals in the interests of high speed copying. Projects in the US where brittle originals were discarded after re-formatting, have been criticised as "the latest form of book-burning", where the physical objects become "sacrificial offerings in the name of textual preservation".
These concerns regarding the significance of original materials, highlight a distinction between "replacement" and "surrogacy" in reformatting. Some material may need to be reformatted to produce a replacement copy - one that will take the place of the original - a legitimate preservation step if it is simply not feasible to maintain an unstable original forever. But in our project we are looking at conversion not as a replacement, but as a form of surrogacy, to deflect casual use away from the originals and towards an easily accessible copy.
The digitisation stage is meant to be a relatively high speed process where digital bitmapped images of each page will be scanned from a duplicate negative microfilm at fairly high resolution (approximately 400 dots per inch). We will produce something like 100,000 uncompressed TIFF digital files, with compressed versions to be also created for networked access.
Like the microfilming, we expect this part of the project to be carried out by commercial vendors, to set technical and quality specifications. Rather than setting up in-house facilities, we wanted to follow this path for such an investigative project, because of the investment required in specialised equipment. It is also in the interests of collecting institutions to have a viable and high standard commercial sector able to provide the production requirements of projects like this one.
Work practices and technical standards are not as well established for digital reformatting as they are for microfilming, so specifications needed to be developed and agreed on early in the project. As I will describe in a moment, that remains a challenge for us.
But it is not just a matter of good microfilming and scanning. How easily the images can be retrieved, read and printed, is critical. I have already said that this is a major focus of the research infrastructure grant, and it is time to turn back to this aspect of the project.
It is important to understand that in all cases other than the four fiction titles, we are making digital pictures of the pages, not searchable text. We have always recognised that it would be good to be able to do both. The only impediment has been the state of Optical Character Recognition (or OCR) technology, which has problems with marginally legible material - it is unreliable enough to require intensive proof reading, estimated to multiply the cost of the project by about eight times. Without full text searching, we have to provide other logical ways into the material.
What we are building is a kind of structure of digital data that can be turned into recognisable images. Structures like this need architectural attention, and there are at least three levels of architecture we have to attend to - how the information is put together within the file (what we call the document control structure); the structure of the database, or the collection of files and how they are indexed; and the structure of the tools that will make the images available - the delivery system. Without these, we may as well not bother scanning, as microfilm will be far more user-friendly than poorly structured digital data.
Our database will be set up to allow searching down to the page level - in other words, you will be able to call up a serial title, and any issue within it, and choose a page number to look at. You will also be able to call up all the issues of titles within a date range.
The delivery system for providing access to the images uses the World Web Web, using a powerful file server at the National Library. It is possible that we will also be able to provide CD-ROM versions on demand, although at this stage we are focusing on Internet access.
The vision for the project was meant to be fairly open-ended. We aim to provide an electronic collection or corpus of nineteenth century source material produced to defined standards so that researchers can access, search, and download material as they see fit for their particular purposes. It should also be possible to enhance the information by later OCR treatment or other manipulation that would support its research use. This is particularly appropriate for serial literature where different sections (such as prose, letters, poetry, news, articles, advertisements) may be used in many different ways . It also recognises the needs of different types of users such as historians, linguists, literary scholars.
All in all, it is quite an ambitious project. The philosophical basis is classic access and preservation. To provide scholars with access, and with opportunities to investigate and even enhance primary source material irrespective of their location. And to provide future scholars and the custodians of these resources with sound and stable means of preservation, not for the next five or ten years, but for many hundreds of years.
We believe the hybrid approach offers all these possibilities.
The funding for the project was announced in late 1995, and we began work in earnest early this year. Where are we now, and what experiences have we had?
I suppose the first thing to say is that the project has had all the challenges we envisaged at the start, and a whole lot more besides, and we are still at a testing stage.
Microfilming and scanning the four works of fiction were completed early in the piece so that they could be worked on separately while the technical issues for the journals and newspapers could be addressed. Work on optical character recognition to produce searchable ASCII text is proceeding, despite some difficulties with the software.
Mounting these fiction files will require a different database structure from the rest of the project because we plan to make both the ASCII form of the page and the image available for viewing at the same time, so that users can find the text they want and see how the page looked.
In common with most experimental digitisation projects every step, every aspect, seems to have taken longer than we expected. Despite the excellent background we have in cooperative microfilming projects, we have found it difficult to deal with the distance between Canberra and Sydney when there are complex decisions to be made. Much of the work of these months has been tied up in checking holdings and selecting "best copies" for use in the project. Just identifying holdings, and gaps in holdings, at each site has been a major task, even with the range of bibliographic tools available. It was difficult to establish criteria for deciding which copy to film.
All of this has been complicated by the very things that make rare serials such a delight to work with - annotations that may or may not be significant; erratic numbering of issues and pagination; frequent and subtle changes of title. In many cases, what the NLA holds matches the State Library's holdings - almost. Except that we have issues 20 to 35, they have issues 1 to 25; our are in good condition for the first 10 issues, theirs for the last 15; ours include annotations that look like they might have been made by John Dunmore Lang, on pages where the printing ink is faint, whereas theirs have no annotations, but they do have a handwritten index tipped into the front; ours are falling out of their bindings, while theirs are tightly bound and would be hard to film. And so on for many of the 75 titles.
For all that, we did manage to come up with a definitive list of what should be filmed - or perhaps it was two definitive lists?
The other major focus of this part of the project has been developing technical specifications. They have been needed for the microfilming, for the digitisation, and for the document, database and delivery structures that I mentioned earlier. And they needed to be comprehensive and reliable before calling tenders and letting contracts could go ahead. We set up a Technical Advisory Group to help us work through the whole process conceptually to arrive at these specifications. And we have recently realised that we don't know enough to be confident that a specification for the whole project will work.
For our broad database architecture and document control structure, we have decided to build on the National Library's growing expertise in putting together Web server databases of images, with the help of documentation from the Library of Congress and Cornell University Library. We will specify the kind of file structure in which images must be delivered by the scanning contractor so that they can be easily loaded into the database. This kind of structure must be able to handle and describe various kinds of pagination (such as covers, fly leaves, title pages, tables of contents, missing pages, inserts etc) as well as including information or metadata on technical specifications used and other relevant matters such as source. I will probably be shot by the people involved for saying this, but all of this looks like the easy part of the project.
There are two other technical issues that have made us realise that we need to do more testing. We are troubled by what we might call the image quality / image delivery dilemma, and the image quality / production output dilemma.
The critical technical issue in developing microfilming and scanning specifications has been the file size of the data making up the images. We believe it is possible to produce high quality images of even poor originals, with annotations, and with much of the feel of the original, even from large format pages - but only at the cost of huge file sizes that are expensive and impossibly slow to retrieve across the Internet, and that will exceed the space on most PCs if someone tries to download them. In other words, we would hardly meet our accessibility objectives anyway.
File size issues are related to issues of image quality, and data compression. We need to find or develop an optimal scanning specification that will provide most images at the best quality with the minimum file size. This means playing around with image depth (whether we scan bi-tonal or with shades of grey), and with resolution (the number of dots per inch).
Data compression offers some hope of getting useable file sizes without losing image quality, but again it comes with a range of less than perfect options. All of them would require users to download some form of viewer from the Project website, and some of them would involve very significant additional costs to pay for the compression itself. Unfortunately, what looks like the most promising compression option (fractal compression) would add about $1 per image to the cost - or $100,000 to the project.
The second dilemma I mentioned was the image quality / production output one. Scanning from very challenging originals, or from very challenging microfilm of very challenging originals, leaves almost no margin for error in scan settings - too much and it has picked up the background, making the image illegible; too little and it has missed the text. This means that every image may need to be individually assessed and "massaged". This is a long way from the kind of production outputs that we envisaged for the project.
Because of these difficulties, we decided that we needed to do two things we hadn't planned when we started:
Firstly, we have decided to introduce a longer testing phase into the project, working in conjunction with a company which has some experience in the area. This should help us iron out some of the unknowns in reconciling image quality, file sizes and network delivery before we proceed to call for quotes for the whole project; and
Secondly, we also decided that we should put off digitising the large format newspapers till later in the project, and investigate other means of delivery such as CD-ROM for them.
We have now entered this test phase with a local vendor. It is due to be completed in November. The vendor will work with us on five titles we have chosen to represent the challenges of the larger project. Three will be filmed and scanned from the print originals - these are The True Catholic: or Tasmanian evangelical miscellany (Jan 1843 - Jun 1843); The Colonial Observer (7 Oct 1841 - 26 Dec 1844) and The Colonial Literary Journal ( Jun 1844 - Mar 1845). Two other titles - The Colonial Magazine and Commercial Journal (Jan 1840 - Dec 1843) and The Star (2 Mar 1844 - 23 May 1845) - will be scanned from existing microfilm.
In doing this work as a test phase, we are requiring the vendor to also work with us to tighten up our specification for the main project, and to help us refine and test the work flow procedures we need to make it all work smoothly and reliably.
We are hoping to have some images available from the test phase by the end of this year, and to go into production phase for the rest of the project early in 1997, with the project to be completed by mid 97.
When we entered the project we talked about it being a benchmarking exercise in this country. We were probably not aware of the technological limitations that still seem to constrain us. Eventually most of these will be overcome, although in some cases it may take technological developments that will happen after we have finished.
This project will add to the research infrastructure of the nation, and will make a significant improvement in both the access to, and the preservation of, these important 19th century materials. All of these we set out to do. In the process, however, we are also finding that we will have to push back the horizons a little more than we expected.
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |