Disaster recovery for ISO ILL locations

1. Objective

To outline the networking protocol for ISO ILL sites interoperating with the Libraries Australia Document Delivery gateway.

Four areas will be covered: testing, system downtime, location data and suspensions.  By being aware of the protocol recommended for ISO ILL partners, we can reduce risk, minimise damage and expedite possible solutions.  Specific solutions will not be given, as each situation is unique and requires a different solution.

2. Testing live ISO ILL sites

ISO ILL interoperability introduces the possibility of configuration changes made in one system adversely affecting another system. Because of the complexities of  ISO ILL interoperability, it is important that ISO ILL partners co-test configuration changes with Libraries Australia Document Delivery.  As an ISO ILL partner you should:

  • be aware of the potential impact of changes made to your system on other systems.  This includes minor configuration changes through to system patches and upgrades
  • advise Libraries Australia Document Delivery administration staff of configuration changes, patches or upgrades before they are put into Production so that testing can occur
  • investigate the cause of Production problems which may have been generated by changes to your system

The National Library of Australia provides a complete Libraries Australia Document Delivery test system for testing with ISO ILL sites.

2a. Understanding system differences

Vendors are constantly developing and refining their software to fix bugs and problems identified by their users as well as developing new functionality to improve their product.

ISO ILL partners need to be aware of the limitations of their and other systems, and the impact this may have on the expected lifecycle of a request.  Some systems may also include functionality that is not available in other products - this can make the workflows of another organisation more opaque to the outsider.

Configuration changes are made constantly as systems are upgraded, workflows are adapted to suit individual locations and in-house developments are implemented.  Partners should be aware that even minor changes can cause problems, either for the ISO ILL partner sites or in transaction processing via the Libraries Australia Document Delivery gateway.  Changes in the Production environment should be tested as fully as possible to reduce the risk of unforeseen problems.

2b. Importance of testing

Problems can be roughly divided into those affecting the software and those affecting workflows.  The impact of such problems can be substantial in terms of financial loss, the administrative time spent troubleshooting problems (both by the vendor and local administrators) and the reputation of an organisation.  Another intangible cost of interruption or disruption of ILL services is the cost to the end user.

Testing changes in a controlled environment before promoting them to Production will minimise the risk of problems in the Production environment, allow for a more measured investigation of potential issues and hopefully provide end users with a high quality, reliable service.

During testing of post-production configuration changes, it is impossible to  test all possible scenarios  given the demands on resources.  The nature of the change should determine the extent of  pre-Production testing.

3. Disaster recovery/system downtime

Occasionally ISO ILL libraries experience problems with their ILL system or with the server on which their system runs. Similar problems occur from time to time on the Libraries Australia Document Delivery gateway, through which ISO ILL libraries interoperate.

While some problems are minor, others have a more profound impact, not only on the library experiencing the problem, but on other libraries interoperating with that library. Examples include: system crashes, system downtime, lost data, corrupted data and loss of functionality.

What are the effects of critical system or server problems? What collective approach can we take to disaster recovery planning in order to minimise the damage and expedite solutions?

3a. Communication during system downtime

Effective communication is paramount when a system issue occurs.

If an ISO ILL library experiences a problem it is important that the library notify others as soon as possible in order to minimise the damage and expedite possible solutions.

Notify Libraries Australia Document Delivery administration at the National Library of Australia important will ensure that  Libraries Australia Document Delivery staff can take appropriate action such as:

  • suspending the affected location on the Libraries Australia Document Delivery gateway so no ILL requests can reach that location
  • advising other ISO ILL libraries of the suspension so they can suspend the affected library on their system
  • help with data recovery (if required) based on ISO ILL messages recorded on the Libraries Australia Document Delivery gateway

If the problem occurs on the Libraries Australia Document Delivery gateway, Libraries Australia Document Delivery staff will send an email to the relevant e-lists(LibrariesAustraliaDocDel-l and ISOlibs-l) advising users of the problem. If Libraries Australia Document Delivery staff cannot post to these e-lists (e.g. due to internet/email problems), they will contact another ISO ILL partner and ask them to post to these e-lists on behalf of Libraries Australia Document Delivery.

3b. Roles during system downtime

a. Role of vendor

It is not the role of ISO ILL software vendors to inform other ISO ILL libraries of system or server problems experienced by their customers.

b. Role of affected library

The library experiencing the system or server problem should communicate the problem to other libraries interoperating with them via the Libraries Australia Document Delivery gateway. Some problems are of a sensitive nature, therefore publicising details of the cause of the problem is at the discretion of the library concerned.

When communicating the problem to Libraries Australia Document Delivery or other libraries it is useful to:

  • advise when the problem first occurred;
  • detail the nature of the problem e.g. system or server problem;
  • advise how the problem is manifested e.g. lost data or creation of duplicate requests;
  • estimate the extent of the problem e.g. number of libraries affected;
  • be as specific as possible;
  • advise what steps have been taken to resolve the problem;
  • advise when the problem has been resolved.

c. Role of Libraries Australia Document Delivery

In the event of ISO ILL partner libraries experiencing system or server problems, Libraries Australia Document Delivery staff can help with:

  • communication;
  • problem resolution (e.g. extracting information from the Libraries Australia Document Delivery gateway and providing it to a library whose system has ‘lost’ that data).

Libraries Australia Document Delivery staff will advise ISO ILL partners in the event of data lost on the Libraries Australia Document Delivery gateway due to gateway system or server problems.

4. Location data

Location Data refers to the information about Libraries Australia Document Delivery members needed for successful requesting and stored on the Libraries Australia Document Delivery system.  Before a location begins interoperating with the Libraries Australia Document Delivery gateway, a full data load of all locations within the Libraries Australia Document Delivery gateway ( must be extracted from Libraries Australia Document Delivery and set up on the ISO ILL system.  Location data includes Libraries Australia Document Delivery and Te Puna locations as well as all ISO ILL locations. This location data needs to be kept up to date both on the gateway and on all ISO ILL partner systems.

A location must be set up on both the requesting and responding systems in order to successfully exchange requests. If a standard Libraries Australia Document Delivery library sends a request to an ISO ILL site, unless this Libraries Australia Document Delivery location has been properly set up on the ISO ILL site this request will not reach the ISO ILL system.

The member base of the Libraries Australia Document Delivery service is constantly changing as new members join and existing members leave the service.  ISO ILL sites must continue to maintain location data to ensure interoperability with all Libraries Australia Document Delivery members.

4a. Maintaining location data

ISO ILL sites need to keep the initial location data information up-to-date by adding, changing or removing details on their ISO ILL system.  The way to maintain location data is:

  • Libraries Australia Document Delivery administration sends notification to the ISOlibs-l of all new sites which join Libraries Australia Document Delivery, either as a standard Libraries Australia Document Delivery member or as an ISO ILL site.  This notification includes all the details required for ISO ILL administrators to create the new locations on their ISO ILL systems.  ISO ILL sites must create all new locations on their ISO ILL systems and send notification to Libraries Australia Document Delivery administration once this has been completed.Libraries Australia Document Delivery administration also sends notification to the ISOlibs-l of all sites which withdraw from Libraries Australia Document Delivery.

5. Suspensions and ISO ILL sites

Occasionally libraries cannot accept ILL requests and need to temporarily stop requests from being sent to their location. Suspensions may be needed because of staff absences, building moves, system upgrades etc.  Within the Libraries Australia Document Delivery gateway environment Requesters may still add a suspended location to a potential rota of suppliers and when a request reaches a suspended location this request ‘skips’ over it and moves onto then ext location.

Maintaining suspension information is an ongoing system related task that contributes to successful interoperability.  When ISO ILL sites neglect to suspend a location, it means that requests can be sent to an inactive location, affecting the speed with which the end user receives the item.

5a. Notification of suspended locations

a. Role of the ISO ILL site

ISO ILL locations should notify the Libraries Australia Document Delivery administrator when they are suspending their location so that the Libraries Australia Document Delivery administrator can suspend them on the Libraries Australia Document Delivery gateway. If an ISO ILL location is not suspended on the Libraries Australia Document Delivery gateway, requests will still reach their location even when suspended on their own library system.

The ISO ILL administrator for each ISO ILL site should suspend inactive libraries on their ISO ILL system.  If an ISO ILL site has not suspended a location on their system, when the ISO ILL site then sends a request to a location already suspended on the gateway, the gateway applies a status of 'suspended' on the responder side of the request. However the requester side never receives this message. For the requester, this request remains active on their ISO ILL system and does not expire.

b. Role of Libraries Australia Document Delivery

When an ISO ILL site notifies Libraries Australia Document Delivery administration of their suspension, this suspension status will be applied to the ISO ILL location on the Libraries Australia Document Delivery gateway:

  • This suspension status will be factored in to requesting activity by all standard Libraries Australia Document Delivery members during the dates specified.  However it does not flow through  to ISO ILL systems.  and each ISO ILL system administrator will need to manually apply the suspension status on their ISO ILL system
  • Current suspensions appear in the LADD Members & Suspension Directory.  This dynamic page is maintained by Libraries Australia Document Delivery administration and displays all suspended locations from the Libraries Australia Document Delivery gateway