Keating redesigned - and the lengths to which web archive curators (sometimes) go!

Excitement was tempered slightly by a 404 page not found error

Former Prime Minister Paul Keating left office in March 1996 at around the time the original project group for what would become the PANDORA Archive started work. But did Keating have a prime ministerial website that was lost in the coincidence of such timing? Well maybe, at least there is some evidence that there was at least a departmental page; but in that 'pre-dawn' time for web archiving it was too early to be captured for the PANDORA Archive, or indeed for that matter by the Internet Archive which also began in 1996. (By the way, check out the neat ‘timeline’ of web archives on the newly re-designed International Internet Preservation Consortium website .) Evidence from the long since defunct Office of Government Information and Advertising (OGIA) Australian Government Home Page – remember that one? – suggests that the ‘Prime Minister’s Home Page’ was established in June 1995 as a page originally hosted – sound the trumpets and strike the viol – yes indeed, on the NLA’s own website! Sadly, I can find no trace of this site. However, it probably looked similar to the earliest website for John Howard who succeeded Keating as Prime Minister in 1996.

Splash page from Keating website 1999

The splash page from the Hon. Paul Keating's website as archived in June 1999

By the beginning of 1998 Keating had established his post-Prime Ministerial website, essentially to disseminate and ensure access to his speeches or, in his own words, “to help encourage discussion about some of the large public policy issues facing Australia”. This site, www.keating.org.au was selected for PANDORA (and permission granted to archive it) in May 1998. Obtaining permission is always an achievement for the web curator – and let me also suggest, a credit to the foresight of creators and publishers – so, a promising start.

Main page from Keating website in 1999

The main page of the Hon. Paul Keating website as archived in 1999

However, fundamentally there are two constraints that confound the successful collection of a website for the purpose of archiving it. One is obtaining the necessary permissions (see my previous post on jeff.com.au) and the other – technical barriers. In the case of Keating’s personal site it was the technical constraints that, until this year, posed the great challenge for completing the collection of the site for archiving. Indeed it would be more than a year after receiving the permission before we could add the first version of the site to the PANDORA Archive. Now, a warning for those who are not web archivists – some operational detail follows. In fact what is suggested by what follows about ‘hand crafted’ web harvesting may frighten some experienced web archivists! In appearance Keating’s website appeared quite basic and simple – a not infrequent taunt for the web archiving curator. However there was one small (and by that I mean large) problem; and that was that the speeches, the primary content of the site, had to be accessed by means of the reader selecting the speech from a drop-down pick and click box. In most cases this is a show-stopper for harvest robots since the robot needs a friendly ‘HREF’ link within the page (or some intelligible script) that it can parse and follow. The robot cannot make queries such as selecting from a drop-down box and clicking the ‘go’ button. In many cases such as this the curator will accept defeat and move on. However some sites are considered worth a bit more effort to see if some solution to the seemingly intractable can be found. In this case it was following a clue that lead to a ‘list’ file – not linked from the website but browsable on the target server – that presented a page which contained a link to all the files for the speeches in the database from which the content was being delivered. This allowed me to use this page as a starting file to collect the content – but that was not the end of the story. It was also necessary to re-write all the links in the referencing page to link to the harvested content (essentially a procedure of matching up file names) and replacing the drop-down box on the search page with the content of the page that listed and linked to all the speeches (see the screen shot below). This, it goes without saying, involved a lot of work.

Search results page from Keating site in PANDORA Archive 1999

The search results page used in the PANDORA Archive version of the Hon Paul Keating website so as to provide functional access to the speeches.

A similar ‘fix’ was done in May 2000 after the site acquired a new splash page promoting Keating’s book Engagement.

Splash page for Keating website May 2000

The new splash page promoting Keating's book 'Engagement' as captured for the PANDORA Archive in May 2000

Move forward a full decade and the site was still largely unchanged at least in the general manner in which it delivered content – selecting from drop-down pick and click boxes and searching. Still, having left it a for a decade it was time to take a deep breath and try another harvest to collect the content added to the site over the 10 years since last archiving harvest. With more content (speeches) on the site there were now six separate pages each with a pick box to select the speeches. Unfortunately this time no convenient ‘link’ page listing all the files could be found. However the value=ID script in the pages was able to be extracted and I created a file of links to add to the harvester to collect all the speeches. Again the pages with the drop-down pick box had to be edited to insert basic HREF links that matched the harvested content. I noted in my records at the time that this work took about two hours.

Keating homepage as archived in 2010

The main page for the Hon Paul Keating website as archived in March 2010.

Search results page from 2010 archived version of the Keating website

An example of one of the re-created search results page from the Hon. Paul Keating website archived in March 2010.

Given this amount of effort you can understand my eager anticipation when I checked the site earlier this year to find a note on the homepage saying the The Honourable Paul Keating website was, finally, in the process of being redesigned. And my excitement when the new site came online with wonderful, glorious, robot-friendly links to the speeches. This excitement was tempered slightly by the fact that most of the links to PDF versions of the speeches result in a 404 page not found error (these error pages incidentally are captured and included in the archived copy and are from the original site and not an indication of archiving failure). As I write this, these 404 errors are still occurring on the live site – but there is plenty of HTML content to keep us all happy and I trust the webmaster will rectify this in due course. Is such effort heroic - or foolish? Well I would like to incline to the former, though the disingenuous may  suggest that the effort was wasted given that eventually, albeit more than 14 years after the site first appeared, it became possible to harvest the complete site for archival purposes and that by waiting we would ultimately have collected the content. But then, you never know, and digital material with its inherent capacity for being 'un-published' with as much facility as it may be published means you have to seize the opportunity while it is there. It may seem strange to some that we archive content that is still available on the live site – until you understand that it ever remains in the shadow of the Damoclean delete button (if I may be permitted to quote myself) with the ever present prospect of disappearing without trace. A print item, no matter how short and short-lived the publication run, being a tangible artefact, always has the prospect of serendipitous or disinterested survival – helped not the least by legal deposit requirements – but not so the digital publication. Web archiving is fundamentally about taking the initiative. I haven’t dwelled upon the appropriateness (or otherwise) of the minor changes to the site that were required in order to get it into the Archive in a functioning state. In substance the changes are minor – no content, no words were harmed in the process and relatively little of the design was tampered with. An 'un-touched' version of the harvest is retained for preservation purposes and detailed notes of the 'fix' were kept in the archiving workflow system – which have informed this post. Moreover, in deciding on this sort of initiative and action we do gain the additional benefit of seeing how the Keating site changed (however subtly) and how Keating presented himself through the redesign of his site’s homepage over a period approaching a decade and a half. This is a fascinating value that web archives add to web content that I intend to discuss in a forthcoming post looking at our complete and revealing collection of various iterations of the first, fully formed and fledged prime ministerial website, that of John Howard. Finally, while the impetus grows and compels towards the heady intoxication of collecting ‘big data’, sometimes, fashionable or not, the imperative to collect small content seems worthy and worth the effort, time, and attention to timeliness.