Australian Newspapers Digitisation Program - The Scanning Process

Australian Newspapers Digitisation Program - The Scanning Process
7 May 2010

The Newspaper pages you see in TROVE, are scanned from microfilm, and with the exception of replacing Target pages with automatically generated images (i.e. pages inserted during the microfilming process to indicate missing pages, missing issues, damaged pages, etc).  The pages are presented as they are found on the original microfilm (i.e. 'warts and all').  The Australian Newspaper Digitisation Program is a mass digitisation effort, aimed at providing the largest number of newspaper pages as efficiently and effectively to the public, and does not aim to (nor are we resourced to) find and replace missing or damaged pages.

An example target page

The microfilm which is scanned is predominantly borrowed from State and Territory libraries.  The lending libraries create an inventory as well as package the reels to be sent to the Scanning Contractor.  The scanning contractor then loads the microfilm reels onto a scanner, which captures the microfilm reels as a 'strip'.  Each image on the strip is then cropped and saved as an individual image.

Scanner

A scanner used by one of our contractors. Visible in the background is the software they use to split and crop the images captured from film

Our contractors produce two images - one Greyscale and one Bitonal file for each newspaper page.  As the Greyscale file is a more realistic capture of the newspaper page, it is the image that is used for delivery.  The bitonal file is an image made up of black and white pixels (no shades of grey), and is the file used in the OCR process (to be discussed in more detail in a later post).  These images are sent to the National Library on data tape, which is loaded into our system and the images are automatically ingested into the Newspaper Content Management system (NCM).  During the ingest process, a number of automated checks are made on the images to ensure data integrity and adherence to our technical requirements. Once ingested, derivatives are generated from the pages for Quality Assurance (QA).  Some of the issues we look for in the QA process include checking to ensure that they are not presented upside down, have not been cropped too closely so that text is removed, and to ensure that supplements are presented as whole pages (i.e. have not been captured on top of other pages).  We QA 100% of all scanned images.  At the current rate of production by our contractors, this means that we QA around 80,000 images every week!

The image shows where the text on the bottom left hand side of the page has been cropped - this would be rejected in our QA process

If the scanned job does not meet our specifications, it is rejected, and the Scanning Contractor must re-supply the entire reel.  If accepted, the images are saved to our Permanent Storage system and a copy is written to tape, which will later be sent to one of our OCR contractors. In my next post, I'll describe the next step in the process - the OCR component.