Australian Newspapers Digitisation Program - The OCR process

Australian Newspapers Digitisation Program - The OCR process
28 May 2010

In my last post I discussed the scanning process, and outlined how our Scanning Contractors digitise  microfilm.  The next step in the process to make content available is to take these digital images and get them converted to the content you see in TROVE. I left the last post with the digital images of various newspaper titles having been written to tape.  These tapes are then grouped in batches which hold around 500,000 pages and sent to one of our OCR Contractors.  We have a panel of four OCR contractors working to produce c ontent for the Program, and the processing branches of these contractors are all based in India. The digital content we send is accompanied by an Official Order, which outlines our specifications for work and the conditions of acceptance of content. The specifications cover two main areas - the technical requirements for content to be ingested, and the specifcications for content analysis. The technical requirements define how METS and ALTO XML files must be structured, and what each file must contain (for more detail see this project web page).  I don't intend to go into a technical description of these files, but it's enough to know that these files are created by our OCR contractors to hold details such as Issue dates and page numbers, the re-keyed and automatically OCRed text, location of each article on the page, and the article category that each article belongs to. These files need to be structured in such a way to allow our Newspapers Content Management system to recognise the article in relation to it's location on the page, the Issue to which that page belongs, and the title to which the Issue belongs. For processing, our OCR contractors load the images we send, and for each article, examine the content and provide an article category.  The article categories we use are News, Faily notices, Detailed lists, results, guides and Advertising.  They also 'zone' the article on the page.  This zoning provides information of where on the page the article can be found, and if necessary, link the article parts (for example, were an article may start on page two, but is continued on page four).  At this stage, Illustrations are also identified and categorised. Once this stage is complete, the title, subtitle, author and abstract (the first for lines of News articles) is re-keyed (manually). This human process has been chosen to help provide confidence that the content (which will be indexed and searchable) is as accurate as possible.  The remainder of text on any article is then atomatically OCRed. The OCR Contractors then use FTP (File transfer over the web), to deliver the content (the METS and ALTO files) back to us.  At this stage, the articles are loaded to the search and delivery interface, though they are not availble to the public you may have come across these articles in your searches.  They have the message "The newspaper issue containing this article has been digitised. It cannot yet be viewed. It is being quality assured to check it conforms to our specifications. It will be available in the next few weeks.").

article_not_yet_available_to_public

Example of an article that is yet to be QA-ed and made available to the public

Each of our four OCR contractors delivers between 25,000 and 30,000 newspaper pages a week, and, adding the 80,000-odd Scanned pages, this means that our Quality Assurance staff work through around 180,000 newspaper pages per week! If the files are structurally correct, they get ingested into NCM.  This is an automatic process.  Once ingested, the QA Staff can then start the Quality Assurance process for OCRed content.  During this stage, it is a  sample of the articles which is checked.  Based on that review, the content is either accepted or rejected.  If rejected, the OCR contractors must correct the errors and re-supply the file.  If there are multiple problems, the OCR content may have to be resupplied (and re-Quality Assured) a number of times, which can introduce delays in the articles being made availble via the serach and delivery interface (and is why we are not able to guarantee dates by which these articles will be available).  We work closely with our OCR contractors through each step of the process to try and minimise the risk of this occuring. Once accepted, the digital images and OCR content are re-formatted for the web, and made available in the search and delivery interface, where users can search, discover and text correct articles. I hope you've enjoyed this overview of the process, and trust that the content we're processing is useful for you.