The news in a word
Finding patterns in Australian current affairs through Radio National data
1 May 2014
Trove has pretty much taken over my life. After a day spent managing the Trove Support Team, I head home and start building things using Trove data. Sad, but true.
So when a new collection is added to Trove, particularly one as rich as the 200,000 records we're now harvesting from Radio National, I begin to wonder what I could do with all that lovely metadata.
Trove is more than a website or a portal, it's a platform. As I've argued elsewhere, portals are for visiting, platforms are for building on. I can create things with Trove, because we push all the metadata we harvest back out to the world through an API (Application Programming Interface).
APIs allow computers to talk to other computers in a language they both understand. So by connecting up to the API, developers can easily build new applications, tools, interfaces and services that display data from Trove.
Researchers can also use the API to harvest large datasets for offline analysis. As Yvonne Perkins has shown in a recent series of blog posts, even non-coders can can take advantage of the API to streamline their research practices.
I like to experiment. The API lets me play, and through playing I think about new ways of using and presenting resources. Last weekend, for example, a little late night inspiration led to Eyes on the past – an interface to Trove's digitised newspapers that doesn't use search boxes or words, but faces and... eyes!
So perhaps you can imagine how excited I was to have access to all those Radio National records. They include just about every segment broadcast on AM, PM and The World Today since 1999. Anyone interested in Australia's recent social, political or cultural history would surely love to dive into the data and explore.
But where to start? The first thing I did was to harvest the records into my own database. Yes, some programming was required, but it's not too hard, and I've shared my code for anyone to use or adapt (see also my generic harvester code). It took about 90 minutes to download more than 200,000 records.
But what to build? I wanted something simple that would give the API-curious a taste of what's possible. I finally decided to extract and display a single word per month from each of the current affairs programs. The word was selected using a statistical measure called TF-IDF (Term Frequency - Inverse Document Frequency) – it's a way of measuring the significance of a word within a collection. Could a single word tell us something meaningful about what was going on in the country?
Generating all the TF-IDF values was a long process and I wasn't sure it would produce anything useful. So I was interested to discover that one of the first words highlighted, from October 2003, was 'Cormo'. Cormo? What was Cormo? I searched Trove and found out that October 2003 was when the Cormo Express, a live sheep export vessel, was desperately seeking somewhere to unload its cargo of 58,000 sheep. Yes, of course! We all remember the event, and that one word brought it back. It seemed like my experiment might actually work.
So I left my laptop to chug its way through the TF-IDF calculations. When it was finished, I built a web interface to display the results for each program side by side. You can explore the patterns that emerged through In a word: Currents in Australian affairs, 2003–2013.
But you can never really guess what people will be interested in. As I was exploring ways of cleaning up the data, I created a list of the ABC journalists involved in each story and added up their number of contributions. Not surprisingly, Mark Colvin, the long-time host of PM, came out on top, and I shared the results with him on Twitter. Mark tweeted the list himself, generating a flurry of interest and comment.
As is often the case, this little experiment also revealed some issues with the data, which we're now working to improve. I can already see what my next project will be – an interactive scoreboard of ABC radio journalists!
This post appears in our "Trove" blogs. To subscribe to future blogposts in this blog add http://www.nla.gov.au/blogs/trove to your RSS reader. To subscribe to all National Library blogposts, use http://www.nla.gov.au/blogs.