In any journey, there's a destination but half the 'fun' is getting there. This article chronicles our journey towards a common way of packaging and exchanging digital content in a future Australian data commons – a national corpus of research resources that can be shared and re-used. Whatever packaging format is used, it has to handle complex content models and work across multiple submission and dissemination scenarios. It has to do this in a way that maintains a history of the chain of custody of objects over time. At the start of our journey we chose METS extended by PREMIS to do this. We learnt a lot during the first two stages that we want to share with those travelling to a similar destination.
In December 2007 the National Library of Australia registered an Australian METS Profile  with the Library of Congress. This profile describes the rules and requirements for using the Metadata Exchange and Transmission Standard (METS)  to support the collection of and access to content in Australian digital repositories. A diagrammatic representation of the profile is given in Annex 1. Early in 2008 a sub-profile, the Australian METS Journal Profile , was also registered. This sub-profile inherits the generic profile and contains specific rules and requirements for journal publications.
The aim of this article is to describe our journey towards a generic Australian METS profile that can be used across multiple domains and usage scenarios. It also describes how the main profile and the sub-profile work together and what additional profiling work is planned by the National Library of Australia and its partners to address the needs of the Australian repository community and (hopefully) of the international community as well.
When libraries first began to engage with digital content, the main issue was how to catalogue it for discovery purposes. It soon became clear that if digital content wasn't brought into collections for safekeeping, it might be lost. Just storing it was not enough; it needed to be preserved for future access. Data was subject to corruption, formats to obsolescence and repositories to neglect or closure. Digital objects could also be extremely complex, made up of multiple components, versions and files. Relationships between these needed to be recorded to ensure that the files making up an object could be deployed by delivery systems for the right purpose and delivered in the right order.
In 2002, the Consultative Committee for Space Data Systems published a Reference Model for an Open Archival Information System (OAIS)  that defined the functions needing to be supported by an archive responsible for preserving content on behalf of a designated community. OAIS introduced the concepts of a Submission Information Package (SIP) and a Dissemination Information Package (DIP) as ways of bringing content into or delivering it from an archive, and outlined the kind of information needing to be included: not just files or pointers to them, but also the reference, provenance, context and fixity information needed for preservation management purposes.
In 2003, the Australian Government funded four projects to develop systemic infrastructure for the Australian repository environment. The Meta Access Management System (MAMS) Project  looked at federated software for authentication, security and access. The Australian Research Repositories Online to the World (ARROW) Project  looked at repository software solutions. The Australian Digital Theses Program Expansion (ADT)  looked at enhancing capability to collect Australian digital theses. The Australian Partnership for Sustainable Repositories (APSR)  focused on accessing and sustaining digital collections in a federated data commons.
APSR complements the work of the other projects by conducting outreach and educational programs. It also undertakes collaborative development of systems and tools to promote interoperability between repositories in ways that ensure their sustainability . Outputs of the four projects are now feeding into development of an Australian National Data Service that will "transform the disparate collections of research data around Australia into a cohesive corpus of research resources" .
Scenarios demonstrating the need for 'clever' and 'long-lived' collections to support this vision were clearly articulated by researchers at two conferences held by APSR during 2007 - for example , , .
In 2006, APSR commissioned the PREMIS Requirements Statement (PRESTA) Project  to identify or develop an open, extensible and standard way of packaging digital objects that could be used as a generic, repository-independent metadata submission and exchange format by Australian repositories. The project partners were the Australian National University, National Library of Australia, University of Queensland and University of Sydney. The impetus for this work came out of a pilot project to manage multidisciplinary data collected during fieldwork from remote locations in Northern Australia .
PREMIS (PREservation Metadata Implementation Strategies)  provides a means of encoding the data elements needed for preservation management purposes using the OAIS model as a framework. The data elements are expressed in the form of four XML schemas – one for objects, one for rights, one for events and one for agents. These can be combined in a PREMIS container as a stand-alone representation of an object. However, PREMIS is mainly concerned with preservation metadata. It does not deal with the description of the object and its structure for discovery purposes or the recording of technical metadata specific to the file format.
At this time, METS was already in use as a packaging format for digital objects. While METS does not natively support the encoding of preservation metadata, it is extensible by plugging in various other schemas such as MODS (Metadata Object Description Schema)  for resource description, MIX (Metadata for Images in XML)  for still image technical metadata and PREMIS for preservation metadata. In particular, it has wrapper elements in its administrative metadata section for technical metadata, rights metadata and provenance metadata encoded in other schemas.
Building with LEGO bricks is a fitting analogy for building a METS document. METS allows descriptive and administrative metadata elements from one schema to be conjoined with elements from another schema and to be referenced by elements that describe the structure of the object and its component parts. By combining METS and PREMIS data elements in a single METS document, an object can be packaged for submission or dissemination in a way that is completely self-describing.
The capacity for METS to bring together different schemas makes it a very powerful packaging format, but also difficult to implement in a standard way. While a number of repository systems now support METS as an interchange format, how it is deployed is starting to vary widely without an agreed set of rules. Having many different implementations of a single standard equates to having no standard at all.
A profile is a way of defining best practices for implementing a standard or set of standards within a given community by specifying the requirements that conformant implementations must satisfy. A good profile addresses a range of usage scenarios and content models to ensure interoperability within that community and provides both entry level and extended options for conformance.
If a single standard implementation were agreed upon, then objects could be more easily transferred from one repository to another. Discovery and access to content residing in multiple repositories could ideally be made seamless from the user's perspective by sharing representations of objects between systems. Agencies implementing a service-oriented architecture could use METS as an interface object. Software and systems developed to support the submission and dissemination of objects could be shared amongst institutions.
The aim of the PRESTA project was to develop a profile over METS that would support the most complex of the required usage scenarios – the transfer of an object from one repository to another, using PREMIS as a METS extension schema to encode the necessary preservation metadata. The PRESTA draft profile  was the result.
Based on this work, the Australian National University and the University of Queensland were able to implement demonstrators for exchanging content between a DSpace repository  and a Fez-Fedora repository . This proved that METS extended by PREMIS could be used to support the transfer scenario but also exposed the need to test the profile against a range of data content models and submission and dissemination scenarios.
In 2007, a second APSR project with the same partners – the Australian METS Profile Development Project  – undertook to test the PRESTA profile against actual implementations and then to register it with the Library of Congress. (The Library of Congress maintains the METS standard as well as a register of approved profiles .)
The National Library of Australia came to the project with experience in the digitisation of still image and audio collections, the archiving of web sites and the creation and publication of born digital content. The Library had also been exploring the infrastructure needed to support legal deposit of digital content (not yet a legislative reality in Australia). The university partners brought experience with the implementation of institutional repositories to the project and had started to explore the need for dynamic interactions between repositories and other services. Digital objects under their care ranged from simple medical images to massive scientific data sets with complex data management and access issues.
Data content models
An important step on the first stage of our journey was to recognise that, when packaged up to be self-describing, despite their singular characteristics, objects of all types can be represented through the same packaging format and submitted or disseminated through the same protocol. In the second APSR project we embraced complexity by looking at how rules for a range of digital content models could be encompassed by a generic, repository-independent profile. The whole journey towards a solution is documented in the final project report , but the end result was the three layered model illustrated in Figure 1.
Figure 1 - Three-layered model
At the top level a generic profile describes the rules for generating a self-describing document. It addresses both submission and dissemination workflows and specifies the mandatory elements and attributes, extension schemas and controlled vocabularies applicable to all content models.
At the second level, content model-specific sub-profiles inherit the generic profile and clarify requirements for specific content models based on test implementations. The first of these sub-profiles – the Australian Journals METS Profile – has now been registered.
Australian content models are defined in a <div> TYPE attribute vocabulary  that is maintained locally so that it can be extended as new content models are developed and existing models are tested by new implementations. Work is now progressing on content models for digital publications stored on physical media, mixed media and audio. This is highlighting the need for a class hierarchy in order to enable common structures to be re-used.
At the third level is the local registration of implementation profiles as part of a national registry of repositories. These will inherit the generic profile and relevant content model-specific sub-profiles. They will specify exception extension schemas and controlled vocabularies, refine inherited requirements and detail technical behaviours.
The infrastructure for registering implementation profiles locally is currently being explored through another APSR project, the ORCA Registry Project , which is testing ISO/CD 2146  as a framework standard for the development of a suite of registry products to serve the Australian repository environment.
The team identified four primary submission workflows needing to be supported to bring digital content into Australian repositories in ways that would ensure preservation and future access:
- Digitisation workflows initiated by staff or contractors involved in creating and uploading master copies and derivative works of physical format materials.
- Deposit workflows initiated by the creator of content in order to submit a born digital work to a repository.
- Capture workflows initiated by a repository in order to bring published content under archival control.
- Publication workflows initiated by a publishing system, as a by-product of publishing digital content, to ensure that a copy is transferred to a repository.
Figure 2 shows how each of these workflows can be abstracted as far as the receiving repository is concerned. Regardless of the different external business processes needing to be supported, the end result in each case is a submission process using the same packaging format, in this case a METS document conforming to the Australian METS Profile. However, additional formats might also need to be supported in order to interoperate with other communities.
Figure 2 - Submission and Dissemination Workflows
The external workflow system packages up the content in a standard format and passes it to the ingest workflow system, which performs various actions such as adding persistent identifiers, generating derivatives and doing virus checking as part of the process of taking the object into the repository. As the content is processed, it may remain packaged in the submission format or changed as needs dictate. For example, a Submission Information Package conformant to the Australian METS Profile may be converted to the Fedora 'flavour' of METS  for ingest into a Fedora repository.
Use of a standard interface to separate submission from ingest workflows is a key requirement in the development of an Australian data commons. Submission may take place as part of a larger and more complex workflow and be a by-product of that workflow, not its primary purpose. As Salo indicates , only a handful of content creators will become actively engaged by the vision of sharing data. For most the act of deposit will need to be integrated with other project activities and result in a positive personal outcome.
The team also identified four primary dissemination workflows:
- Delivery workflows initiated by the repository's own delivery system in order to provide access to content under its control.
- Requests for representations of an object for use by another system as part of seamless federated discovery and delivery workflows.
- Requests for representations of an object for modification by an authorised service or user.
- The transfer of content from one repository to another (the original PRESTA workflow).
Figure 2 shows how, as with submission and ingest, each of these workflows can be abstracted as far as a receiving external workflow system is concerned. Even when the delivery system is a repository service providing access to content under its own control, there are benefits in terms of a service-oriented architecture in disseminating that content in a standard format. In addition, a repository that is not the primary means of publishing content may need to take on that role over time. Simply delivering a list of files is not enough. The repository's delivery system needs to understand the underlying content model and render the object in usable ways.
For example, when a digital journal goes 'out of print,' the repository holding a copy for safekeeping should be able to reproduce the table of contents and navigation capabilities supported by the original website. This requires knowledge of how articles are grouped in each issue and the relative position of the issue in the hierarchy of the journal. The Australian METS Journal Profile specifies rules for the Submission Information Package that ensure the repository can generate a Dissemination Information Package with the required information.
Differences in submission and dissemination packages
In almost all cases a Submission Information Package will contain different information from a Dissemination Information Package. In the Australian METS Profile a Dissemination Information Package must have a globally unique persistent identifier (Handles have been selected as the preferred persistent identifier system in the Australian context ). A Submission Information Package need not have one unless required by the context of the submission workflow: for example, when the submission system is updating an existing object; or has already been given (or can predict) the identifier.
A Submission Information Package may not have any provenance information beyond the context provided by the submission process itself, although information about the creation event should be included if it is a digitised object. In a scenario where content is being transferred from one repository to another, a Dissemination Information Package must have information about the ingest event as well as any transformation and integrity verification events that the object has undergone while in the repository.
Both kinds of packages can reference content that has not yet been ingested. Not all objects described by METS documents need to be digital. In a dissemination workflow, the object or some of its components may not yet have been digitised or ingested, yet descriptive metadata and information about host-component relationships may have already been described to facilitate discovery, navigation and requesting.
The primary object represented by a METS document in a submission workflow might be different from the primary object in a dissemination workflow. For example, the journal issue is the primary object in a publication submission workflow but in dissemination scenarios it might equally be a page, an article or an illustration.
To cater for multiple dissemination scenarios, the Australian METS Profile requires each component of an object to be fully self-describing and able to be represented by its own METS document. This means that each component needs its own persistent identifier. By allowing for multiple dissemination scenarios, the profile does not restrict a delivery system to one way of doing things.
A persistent identifier will usually resolve by default to a delivery system for the referenced object. By contrast, in a machine-machine transaction a service will need a way of requesting a METS representation of the object for use in another system. A discovery system might only want thumbnail images for display in result sets. A delivery system might only want derivatives and not master files or a full preservation history. There may be a requirement for a representation of an object as it was at a given date or time.
OpenURL  is currently being used to solve the 'appropriate copy' problem in other spaces . The controlled vocabularies developed for the METS <fileGrp> USE attribute  and the METS <structMap><div> TYPE attribute  could also be deployed effectively within OpenURL to request a METS document describing, for instance, only files of a given use or only components of a given type.
Things learnt along the way
METS can be used as a means of transmitting a representation of an object (physical or digital or partially digital) from one system to another. It can:
- Fully describe the object and its components.
- Encode the metadata needed to aid its preservation and future access.
- Represent the physical and/or logical structure of highly complex objects.
- Represent collections of objects, even where these objects are not stored in the same repository.
- Support a range of submission and dissemination scenarios.
- Deliver representations of an object appropriate to the scenario by using a protocol such as OpenURL to request the required parameters.
METS could also be used as an Archival Information Package within a repository, although this is not a scenario that we addressed in the profile. Figure 2 shows that how a repository stores content is its 'own business'.
Even when a repository system supports METS natively, this will not necessarily address requirements for interoperability. The repository may need to support multiple ways of packaging an object for submission and dissemination, and it may need to disseminate an object in a packaging format different from the one used for submission. It may need to support more than one METS profile to engage with other communities, and it may need to interoperate with other packaging formats, of which there are many.
The proliferation of packaging formats [35-41] does raise the question of whether METS is the most appropriate 'flavour' or whether there are better solutions. Certainly, how METS sets up the relationships between descriptive metadata, administrative metadata, file metadata and structural metadata is not as elegant as it could be. Redundancies and duplication of information are introduced by the use of extension schemas that are trying to solve overlapping problems without the context of an overarching information model.
However, it does the job, and the information it encodes is the information needed to support the submission and dissemination scenarios described above. In particular, in combination with PREMIS it addresses the requirement for 'clever' and 'long-lived' collections. While some preservation metadata can be re-created from the object at any time, provenance information needs to be collected as a by-product of processes that create, validate or transform an object or the chain of custody will be broken.
The content modelling work that we did to develop the Australian METS Profile and that will continue in the development of sub-profiles is work that would have been necessary regardless of the chosen packaging format. As part of the Australian METS Profile development project we spent a considerable amount of time developing the three-tiered model for the profile in such a way that the generic complexities were addressed in the top level profile, leaving the complexities around defining shareable content models to the second level. This model is independent of the packaging format and can reference new encoding standards if required.
Lessons learnt while developing the profile can also be transferred or applied to other packaging formats. For instance, the Australian METS Profile requires each component of a complex object in a METS document to be fully self-describing. Testing the profile against real implementations helped us to recognise that ingest workflow systems should be able to locate and process information about each component solely via the packaging structure without having to unpack and interpret the metadata encoded in extension schemas.
Submission and dissemination services
Regardless of the packaging format there still needs to be an agreement on the type of service needed to support submission and dissemination scenarios. This was explored under the APSR Repository Interoperability Framework (RIFF) suite of projects  comprising a Submission Service , Dissemination Service  and several Workflow projects [45-48].
The Submission Service aimed to address the disparate requirements for ingestion, submission and exchange of repository objects through a common service. The Dissemination Service examined a standard means of rendering objects across different repository platforms. The Workflow projects acted as use cases to inform the development of these services as well as the development of the Australian METS Profile. The use of these services is best described via an example.
The Journal Workflow project focussed on the use case of preserving access to an on-line journal created via the Public Knowledge Project (PKP) Open Journal System (OJS) application . The Submission Service takes packaged content (OJS Native XML), performs pre-ingestion processing over it (transform OJS Native XML into a METS package) and submits it to a repository. This workflow is customisable via the ability to develop and configure localised workflow steps within the service. The METS package is unpacked by the receiving repository and stored in whatever form the repository requires.
In future, the OJS application itself is likely to support the export of content as a METS package.
The Dissemination Service is available to repositories and makes use of the Digital Repository Interface (DRI) XML  as the standard for representing the repository objects. In this way any repository able to generate DRI-compliant markup can store their objects natively but through the Dissemination Service have them rendered in a common way. Under the Journal Workflow, a journal stored in DSpace and Fedora native formats (vastly different) can be given the same look and feel.
While in the APSR implementation DRI was used as the common dissemination format schema, it would also be possible to achieve similar results using only METS. DRI exposes not only an object but information about user interface properties of the containing repository that a processing application could turn into a web page.
Using METS rather than DRI would have the advantage of being relevant in many other contexts than object rendering from a repository. In a repository publishing environment, for example, a METS package wrapped by DRI could be appropriate, whereas a service requiring only a repository object or part thereof could just make use of the METS package alone.
While the Submission and Dissemination services have been successfully tested internally, they are not yet being used in production. The future Australian National Data Service may have use cases to which these services can be applied and thus further developed.
The Simple Web-service Offering Repository Deposit (SWORD) project  led by UKOLN published its Deposit API not long after the Submission Service project had concluded. SWORD has been developed as a profile of the ATOM Publishing Protocol and is agnostic to workflow or content packaging format. Combining METS with SWORD in submission workflows is a direction we are currently exploring.
Pausing to catch our breath
METS is a good fit for reaching our destination. It is, however, one of a long line of standards developed to meet emerging needs. Standards will continue to be developed to meet changes in technology and the dynamic nature of the digital universe. What history has shown us (continuing the journey analogy) is that standards power us along for a while and then break down at the side of the road, while a new standard is taken up and used to move us all a little further down the road towards interoperability. However, as a preservation community what is important is that the integrity of our data and metadata is not affected when we stop using one standard and adopt another.
As part of the international METS community we still have a lot of thinking to do. The community itself is currently reviewing the profile process to move towards best practices, rather than registering solutions developed for specific implementations. Perhaps in an ideal world there would be no need for an Australian METS Profile. In the meantime we are hopeful that chronicling the journey we have taken may be of assistance to those travelling to a similar destination.
We have probably made decisions that are controversial in trying to push the boundaries. In particular, by making the profile work for hybrid collections (physical and digital objects) we have taken the word digital out of 'digital repository'. In our quest towards a packaging standard for the Australian data commons, we have tried to synthesise requirements and to do this via a profiling model that has meaning for everyone. Our journey is by no means over and we will continue to chronicle it on the Australian METS Profile website .
This journey would not have happened without Chris Blackall and Leo Monus, who paved the way in Western Arnhem Land through the Bidwern Project; Bronwyn Lee, Gerard Clifton and Somaya Langley, who prepared the PRESTA Draft Profile; Douglas Elford and Nicholas Del Pozo who (with Bronwyn and Gerard) helped us with the Australian METS Profile development and (with Snezana Mihajlovic and Matthew Walker) are now taking this work forward; Colin Webb, Adrian Burton and Warwick Cathro (our sponsors); and the RIFF project partners.
1. Australian METS Profile 1.0. Library of Congress.
2. METS: Metadata Encoding and Transmission Standard. Library of Congress..
3. Australian METS Journal Profile 1.0. Library of Congress.
4. Consultative Committee for Space Data Systems. Reference Model for an Open Archival Information System (OAIS). CCSDS 650.0-B-1. Blue Book. January 2002.
5. Meta Access Management System (MAMS). Macquarie University.
9. Adrian Burton, Chris Blackall and Scott Yeadon, "The metadata coalface for digital repositories", VALA 2008, Libraries/Changing Spaces, Virtual Places, Melbourne 5-7 Feb 2008.
10. ANDS Working Group. Towards the Australian Data Commons: A Proposal for an Australian National Data Service. Australian Government. Department of Education, Science and Training. October 2007, p.5.
11. Kaarin Anstey. "Data Use & Re-use at the Centre for Mental Health Research". Paper given at Long-lived Collections: The Future of Australia's Research Data. Australian Partnership for Sustainable Repositories (APSR). National Library of Australia, 11 September, 2007. PDF.
12. Ashley Buckle. "The Australasian Repository for Diffraction Images". Presentation given at Clever Collections: A National Showcase of Technical Innovations for Digital Collections. Australian Partnership for Sustainable Repositories (APSR). University of Melbourne, 29 November, 2007. Podcast. Day 2. 9.00 Session.
13. Steven Hayes. "Field Helper – How to Submit to a Repository Without Knowing What a Repository Is ...". Paper given at Clever Collections: A National Showcase of Technical Innovations for Digital Collections. Australian Partnership for Sustainable Repositories (APSR). University of Melbourne, 29 November, 2007. Podcast. Day 2. Lecture Room 2. 11.00 Session.
14. PRESTA: PREMIS Requirements Statement. Australian Partnership for Sustainable Repositories (APSR).
16. PREMIS (PREservation Metadata: Implementation Strategies) Working Group. PREMIS Data Dictionary, version 1.0. OCLC.
17. MODS: Metadata Object Description Schema. Library of Congress.
18. MIX: NISO Metadata for Images in XML Schema. Library of Congress.
19. Bronwyn Lee, Gerard Clifton and Somaya Langley, PREMIS Requirement Statement Project Report. Australian Partnership for Sustainable Repositories (APSR). July 2006.
21. FEZ: Fedora-based Repository Management System. Australian Partnership for Sustainable Repositories (APSR).
22. METS Profile Development. Australian Partnership for Sustainable Repositories (APSR).
23. METS: Metadata Encoding and Transmission Standard. METS Profiles. Library of Congress.
24. Judith Pearce, David Pearson, Scott Yeadon and Megan Williams. Report of the METS Profile Development Project. Australian Partnership for Sustainable Repositories (APSR). November 2007.
25. Australian METS Profile <div> TYPE Attribute Vocabulary. National Library of Australia..
26. Online Research Collections Australia (ORCA). ORCA Registry. Australian Partnership for Sustainable Repositories (APSR).
27. ISO/CD 2146. Information and Documentation – Registry Services for Libraries and Related Organizations. See also: ISO 2146 Project. National Library of Australia.
28. Encoding Fedora Objects in METS (Fedora Extension). Fedora Release 2.1.1.
29. Dorothea Salo. "What You're Up Against". Paper given at Getting the Most Out of Your Institutional Repository: Gathering Content and Building Use. NISO : Beltsville, MD. 3 December, 2007. PDF.
30. PILIN (Persistent Identifier Linking Infrastructure) Project. Project Closure Report. 20 December 2007.
31. ANSI/NISO Z39.88 -2004. The OpenURL Framework for Context-Sensitive Services. NISO: National Information Standards Organization.
32. Examples include: FRED (Federated Repositories for Education). Appropriate Copy Toolkit. 3 August 2007.; Jeroan Bekaert, Lyudmila Balakireva, Patrick Hochstenbach and Herbert Van de Sompel. "Using MPEG-21 DIP and NISO OpenURL for the Dynamic Dissemination of Complex Digital Objects in the Los Alamos National Laboratory Digital Library". D-Lib Magazine. Volume 10, Number 2, February 2004..
33. The controlled vocabulary for the <file Grp> USE attributes consists of the following terms: co-master, derivative, derivative master, finding aid, master, original, preview, print, related metadata, structural map and transcript. For definitions see the profile.
34. See note 23.
35. IMS Global Learning Consortium Content Packaging Specification (IMS-CPS).
36. Sharable Content Object Reference Model (SCORM) 2004 3rd Edition Documentation Suite. Advanced Distributed Learning.
37. Nestor Peccia, Gianpiero Di Girolamo, Michael Dyck, Towards a CCSDS XML Packaging Approach in the ESA Data Disposition System.
38. Cover pages. Technology reports: MPEG-21 Part 2: Digital Item Declaration Language (DIDL). OASIS.
39. WARC File Format (working draft proposed to ISO TC46/SC4).
41. The RAMLET project: Developing a reference model for resource aggregation for learning, education, and training. IEEE LTSC CMI Working Group 2005.
42. RIFF Project.
43. Submission Service. Australian Partnership for Sustainable Repositories (APSR).
52. Australian METS Profile. National Library of Australia.