Harvesting Radio National

Harvesting Radio National
New records provide a rich record of Australia’s cultural and political history
17 April 2014

It’s not all blogs, Twitter, and jelly snakes here in the Trove Support Team. A lot of our time is spent helping organisations make their collections discoverable through Trove. We love working with new content, and we’re particularly excited by our latest addition – more than 200,000 records from 54 ABC Radio National programs.

Search now in the Music, sound and video zone of Trove and you’ll likely come across one of the new records – there’s everything from Hindsight to Future Tense, RN Breakfast to Late Night Live. Some of the programs date back to 1997.

Screen capture of Trove results for PM

But it gets better. If there are transcripts online we’ve indexed them as well to make the content as easy as possible to find. Some programs, like AM, PM and The World Today, will have records for every segment broadcast over ten years or more. That’s a pretty remarkable record of Australia’s recent political, social and cultural history.

How did it all happen? In 2010, we began working with the ABC to bring records for Four Corners into Trove so there was a presence for the program’s 50th anniversary. Following on from this success, we worked with the ABC to investigate what we could do with Radio National content. In June 2012 records for two Radio National programs, the Philosopher’s Zone and the Science Show, were added to Trove. New records for these two shows have been regularly added since. In early 2013 we included an additional 12 programs, and by the beginning of 2014 we’d harvested the remainder of the currently airing Radio National shows.

We use  RSS and podcast feeds to find out when new segments have been posted to the Radio National site. Our harvester checks these within hours of a show going to air, and captures metadata from the webpage for inclusion in Trove. If a transcript is available this is also indexed. The combination of program metadata and fulltext transcripts helps you easily find the shows and topics that are of most interest to you.

But what about all the programs broadcast before we set up our harvests? In order to bring all this important historical content into Trove we set up a computer script to create a sitemap from the show’s archive. Harvests of these shows are run once a month to capture any updates that may have been made to the shows, such as new transcripts or content corrections.

Our work on ABC RN archives also encouraged us to look again at the way we were capturing Four Corners records. We’ve now implemented a new process to keep our Four Corners content up-to-date.

That’s the simple version anyway. In order to actually extract the metadata from the ABC’s web pages we sometimes had to create special processing rules to deal with differences in the way the data was displayed. Harvesting can be tricky and painstaking work. But that’s what we do. We find ways of getting your content into Trove.

Further information

We've posted some useful search tips and a full list of harvested Radio National programs to the Trove Forum.


This post appears in our "Trove" blogs. To subscribe to future blogposts in this blog add http://www.nla.gov.au/blogs/trove to your RSS reader. To subscribe to all National Library blogposts, use http://www.nla.gov.au/blogs.

Congratulations NLA. This development is in the tradition of what great Libraries have done for millennia.
Australia’s collections online