Networked Services Project
Request for Tender for NLA Networked
Services
Chapter 5 - Technical Requirements
5.1 Introduction
5.2 Technical Architecture: Guiding Principles
5.3 Technical Architecture: Specific Requirements
5.4 Technical Evaluation - General Principles
5.5 Technical Evaluation - Specific Issues
5.6 Technical Environment - General Capabilities Sought
5.7 Technical Environment - Specific Capabilities Sought
5.8 Background Information for Sizing: Current Activity Profile
5.9 Standards Conformance
5.10 Questions For Tenderers
5.1 Introduction
The purpose of this chapter is to ensure that Tenderers have an understanding of the overall framework or environment within which the Project capabilities (Chapter 4) are to be delivered, and the key business characteristics and constraints that apply to this framework.
Specifically, this chapter provides guidance to the Tenderers in terms of technical architecture and related critical factors as the Library perceives them. The architecture and the critical factors are outlined below.
5.2 Technical Architecture: Guiding Principles
The critical factors that the Library has determined for the NSP technical architecture are:
| a) | Sizing: adequate sizing for the Library's aggregate business functions at their required levels of performance | |
| b) | Scalability: an ability to accommodate future growth in business volumes and complexities | |
| c) | Flexibility: an ability to accommodate: | |
| i. | trade-offs among conflicting performance and service level requirements | |
| ii. | portability as necessitated by business needs and functional requirements. | |
The above three factors are to be used by the Library as the guiding principles for selecting an NSP technical architecture.
5.3 Technical Architecture: Specific Requirements
| a) | High availability and robustness of the overall system, including rapid ability to recover recent data in case of any disruptions or problems. |
| b) | High performance for Library's business activities, maintaining that performance level while executing concurrent tasks required to support all systems. |
| c) | An auditable yet flexible security mechanism for accessing various parts of NSP environment. |
| d) | Sound System Management facilities that can control (a) to (c) above. |
| e) | Compliance with de facto and de jure standards, wherever possible, at all layers of software and data, to meet the need for creating a modular and flexible open system solution. |
5.4 Technical Evaluation - General Principles
The evaluation principles for the NSP technical environment will be based on the guiding architectural principles as outlined in Section 5.2, ie sizing, scalability and flexibility of the tendered solution.
As described in Chapter 4, the NSP will have a complex matrix of workload requirements. The total workload will comprise a number of distinct types or categories, eg. the online category and the background category, with the online category having the search component and an update component.
Each of these workload categories will operate on large data sets, and have relatively high levels of user traffic, data transfer and transaction volumes.
Many of these workload categories will often need to run concurrently to meet deadlines and other business requirements. This could potentially result in competing demands on systems resources, leading to potential degradation in service levels and/or performance.
In the above context, Tenderers will be expected to present satisfactory treatment of such complexities and inter-dependence, preferably with alternative scenarios involving trade-offs on different sets of requirements.
The balance between these technical considerations and the functional fit with the Library's business requirements will determine the final score of a solution.
5.5 Technical Evaluation - Specific Issues
5.5.1 Sizing and Scalability
Tenderers are asked to ensure that the tendered sizing accommodates all the key performance indicators eg. online response levels, data management and housekeeping throughput, activity logging throughput etc, together with all the concurrency requirements as mentioned above.
Following are some of the specific issues that the Tenderers are required to address.
| a) | Database volume and user traffic: how the hardware/software configuration offered will address the database sizes and numbers of active users specified by the Library |
| b) | Transaction volumes: transaction volumes are deliberately not specified, as a transaction in the new environment will be defined and packaged very differently from that in the current environment. However, the transaction mix that each proposed Configuration is expected to support needs to be detailed by the Tenderer |
| c) | Online response characteristics: the expected online response time characteristics need to be detailed. The effects of update transactions on the online search environment need to be described. Alternative scenarios need to be provided regarding the optimal mix of search and update environments, eg impact of real-time updates and indexing vis a vis batched updates and/or batched indexing |
| d) | Activity logging: the levels and nature of activity logging available in the proposed system, and their respective impacts on online performance |
| e) | Background activities: estimated elapsed times needed, and potential impact on online performance |
| f) | Maintenance: the time window required for data and system administration (eg data reorganisation/update, index build, processing of logs) and housekeeping, and the implications of these on system availability |
| g) | Indexes: whether the Library's business transactions can be met with the Tenderer's standard sets of indexes as deployed at currently installed sites, or whether additional indexes are required. If the latter, then the costs are to be provided in terms of estimated elapsed time, system unavailability, online response times and resource usage |
| h) | Resource-intensive requirements: areas that are known to the Tenderers to be resource-intensive in their proposed solutions (eg any specific types of searches, aspects of activity logging, modes of updating), should be identified in the tendered responses. |
5.5.2 Methods for Validating Sizing and Scalability
The Tenderer should note:
| a) | That the Library prefers that the above information be provided in tabular forms, showing inter-relationships among Sections 5.5.1(a) to Section 5.5.1(h), to assist in evaluating the trade-offs and options involved |
| b) | The Library will strongly prefer solutions that have demonstrated evidence regarding sizing information, such as existing production sites, available benchmarks etc. The onus will be on the Tenderer to provide adequate sizing and scalability information in their responses and/or mechanisms for validating their sizing and scalability estimates |
| c) | That the Library would prefer the Tenderer to nominate actual sizing examples rather than to use broad extrapolation in arriving at estimated capacity. |
5.5.3 Potential Areas for Trade-offs
Through its past experience in automated online systems, the Library has identified some specific issues that are likely to bear on overall system performance and can therefore call for trade-off considerations. Tenderers are required to address the issues below in their responses, and present analyses of alternative approaches and options where appropriate.
a) Simple and complex searches
One of the Library's most important requirements is a high performance online searching environment.
Online response times can vary widely from simple searches, eg direct searches on a full key such as ISBN or other identifiers, through complex searches, eg Boolean searches from multiple fields involving multiple operators. A few very complex search transactions can potentially slow down the average response times of the simple searches.
One of the ways the Library has mitigated this conflict in the past was by pre-indexing frequently occurring phrases and word combinations. Tenderers should examine and present alternative approaches in this area of performance management.
b) Update mechanisms
In addition to requiring a high performance online searching environment, the Library also requires rapid incorporation of new/update information into the online search environment.
An online real-time update/indexing mechanism can significantly impact the efficiencies of online search responses. Tenderers should examine this issue and may wish to propose alternative scenarios of update management, such as:
i. batching the update transactions and applying them as a background task a few times a day as opposed to instantaneous real-time updates
ii. instantaneous updates and background indexing
iii. any other combination the Tenderer considers beneficial.
c) Transaction management and activity logging
Comprehensive transaction management ie recording of full audit trails and roll forward information can prove to have significant overheads on an online search environment. Tenderers should state logging options and the associated effects on overall system performance.
d) Extent of background tasks and housekeeping activities
The Library's current Bibliographic Utility ABN has a significant batch/background component, greater than half the ABN online component in terms of CPU usage and about a fifth of the online component in terms of device activities (see Table 5.1, rows 1, 2, 4 and 7). Of the background tasks, slightly more than half comprises ABN new data load and data updates, and creation of products for other libraries, while the remainder comprises administrative functions such as backup and housekeeping, processing transaction logs, index builds and data re-organisation (see Table 5.1).
Current Service Level Agreements involve only 12 hours of online ABN services, with the background tasks being run in scheduled after-hours shifts. In the new Networked Services environment the online availability is expected to be significantly longer. Tenderers must specify how the above activities are to be handled, whether a batch processing window will be required and must state the likely impact of running background work concurrently with online sessions.
5.6 Technical Environment - General Capabilities Sought
General capabilities sought include high online performance levels, high throughput rates for background/batch loads, and an architecture that supports concurrent processing of the above two as much as possible and offers extended hours of online availability.
5.7 Technical Environment - Specific Capabilities Sought
5.7.1 Online Response Times
The Library aims for a target performance level for online searches at least equivalent to the current environment (as in Table 5.2), and better if possible. Specifically, the Library would prefer that:
| a) | The response times for 95% of search transactions do not exceed 2 seconds at the user work station. These search transactions are to include direct searches (eg full key searches such as ISBN, ISSN), Simple Boolean Searches (ie Search 'ands' keywords in one search field) and Complex Boolean Searches (ie searches involving multiple fields and operators) | |
| b) | The response times for 95% of Online Display transactions do not exceed one second at the user work station | |
| c) | The response times for 95% of Document Delivery Request transactions do not exceed 2 seconds at the user work station (see Section 5.10.1) | |
| d) | The response times for 99% of Update Transactions do not exceed 5 seconds at the user work station. These Update Transactions are to include Simple Update transactions (ie less than five attributes are created or changed) and Complex Update transactions (ie more than or equal to five attributes are created or changed). | |
| To that end, the Library would like NSP to incorporate the following broad principles: | ||
| i. | to maximise the use of electronic memory eg cache, disk cache, buffers etc, and minimise calls to mechanical devices (see Appendix B.3.6). | |
| ii. | to deploy a balanced systems configuration (ie among CPU, memory and device components), minimising potential bottlenecks. | |
5.7.2 Background Processing
Greater flexibility in running background processing than is feasible in the current environment (see Table 5.3).
Specifically, the Library would prefer that background processing be able to run concurrent with online sessions, without significant impact on the online performance. These background processes are to include MARC and other data load, indexing, extraction of database records, product creation, and report production.
5.7.3 Backup and Housekeeping
Fast efficient backup and restore capabilities that can perform backups frequently enough to allow a high level of ability to recover data. Specifically, the Library would like the Tendered backup solution to:
| a) | Comprise a proven suite of platforms, software and media that have established track records of performing reliably with large amounts data equivalent to the Networked Services data in question |
| b) | Be able to recover data at least as reliably and precisely as in the current environment, and better if possible |
| c) | Have a range of options such as the ability to take full or incremental backups, and backups on the basis of physical or logical images (ie devices and files respectively), and offer equivalent facilities for restore operations |
| d) | Have a fast enough throughput to allow sufficient system availability |
| e) | Be capable of network backup of multiple LAN connected servers so as to allow a better coordinated backup operation. |
5.7.4 Activity Logging
Activity logging would need to address both day-to-day management requirements such as problem tracking and resolution and change management, as well as longer term needs such as trends analysis for business purposes.
5.7.5 Availability
Online system availability to be longer than in the present system, ie greater than twelve hours, without affecting the overall throughput of the necessary background processing (see above).
To that end, the Library would require for the NSP environment:
| a) | hardware reliability: low-fault hardware components, preferably with some degrees of fault tolerance or 'hot-plugging' abilities in order to minimise the impact on system availability |
| b) | software reliability: proven stable software, that has acquired maturity through use at large sites |
| c) | data recoverability: good backup and restore capabilities as in Section 5.7.3 |
| d) | maintenance: efficient maintenance mechanisms that have minimal requirements for a quiesced system. |
5.7.6 Storage Capacity - Online, Offline, Archival
| a) | A storage efficient system, with minimal space overhead for storing indices, and minimal overhead of temporary work area during processing |
| b) | Provision for online and offline storage that is readily accessible, and offline storage of archival status data |
| c) | Backup storage media that suit large data entities as well as small amounts of miscellaneous data |
| d) | Adequate redundancy built into the online storage design to ensure a high level of availability. Faulty devices should be able to be isolated and replaced without impacting online sessions (eg hot-pluggable devices). |
5.7.7 Processing Architecture and Capacities
The Library aims for an NSP processing architecture that would have the following attributes:
| a) | A processing architecture that achieves a balance between modularity - ie independent platforms for various functional components - and ease of management |
| b) | A processing architecture that would make ample use of electronic storage at all levels - eg large memory, storage cache, disk cache etc - to yield good online performance and to prevent the I/O subsystem from becoming a bottleneck |
| c) | The architecture needs to be scalable for growth, and open to accommodate new technical components as they emerge |
| d) | If the Tenderers recommend/provide specific brands of equipment, the Library would like to ensure the ability for future migration to other platforms should that prove beneficial to the Library. |
5.7.8 System Management
In order to coordinate and optimise the technical factors mentioned in Sections 5.3 to 5.5, NSP requires an overall System Management facility that would effectively control and balance the range of variables in question.
Some of the specific needs are outlined below, aligned with the key technical requirements expressed in Section 5.3, ie availability, performance and security.
5.7.8.1 Availability Management
Availability was defined in Section 5.7.5 in terms of reliability, recoverability and low-disruption maintenance facilities.
These translate into the following technical functions:
| a) | Process management: automated facilities to start tasks, monitor completion of tasks, and initiate events as appropriate, eg. issue alerts, display messages or initiate further tasks | |
| b) | Error management: automated facilities to detect equipment and software errors and initiate action | |
| c) | Preventive maintenance for device and data: utilities to check device and media status and issue remedial actions (eg. detection and treatment of disk and tape surface problems), and utilities for bulk data manipulation eg. re-blocking and re-formatting of data, and de-fragmenting of file space | |
| d) | Low-disruption software maintenance: automated utilities for software configuration management, version control, error tracking and change management, to ensure: | |
| i. | a stable software environment with lower incidence of error | |
| ii. | an auditable environment with greater management information such as problem history and update patterns. | |
5.7.8.2 Performance Management
Automated facilities to produce performance and resource usage statistics, for both day-to-day management and for long-range analyses for capacity projections and other business planning.
Specifically, tools are needed to:
| a) | Identify hardware and software bottlenecks and sources of delay | |
| b) | Measure usage levels of system resources such as | |
| i. | CPU | |
| ii. | memory/cache | |
| iii. | devices and data paths | |
| iv. | software resources eg. buffers, threads, processes | |
| c) | Measure response/throughput of business transactions | |
| d) | Itemise, where possible, resource usage by business entities such as service and transaction types, user-ids and/or Customer Libraries | |
| e) | Produce analyses in a flexible way, to address routine as well as ad hoc queries/requirements. | |
5.7.8.3 Security Management
As mentioned in Section 5.3, the Library will need an auditable yet flexible security mechanism that will allow effective access to data and facilities of the NSP environment. The specific functions sought are:
| a) | A security system that allows a unified means, as much as feasible, of access control across all NSP resources ie. data, processes, equipment and System Management facilities |
| b) | Abilities to implement installation-specific modifications/customisation |
| c) | A security administration mechanism that can be delegated across different parts of the Library functions, in accordance with business requirements |
| d) | An ability to ensure integrity in the security implementation mechanisms, especially in case of distributed security administration arrangements as in (c) above. |
5.8 Background Information for Sizing: Current Activity Profile
To assist Tenderers in formulating some of the responses sought above, the Library is providing the following information (Tables 5.1, 5.2, 5.3) on the current system characteristics. These are broad estimates of some of the key factors mentioned earlier such as the relative proportions of online and background activities, relative proportions of basic NBD search and advanced search, ratios between inquiry, cataloguing and interlibrary loan commands etc.
The figures have been presented in percentages as opposed to the actual numbers, as this was regarded as a better indicator, since the new environment is likely to be based on a completely different technology and architecture from that of the current ABN/Ozline environment. The actual figures are included in Appendix B as reference material and may assist in the sizing process.
Table 5.1 ABN: Sample of Relative Monthly Resource Usage by Applications/ Activities (Expressed as percentage, with ABN Online Base as 100%)
August 1996 |
November 1996 |
|||||||
Elapsed Time(hours) |
CPU |
Total I/O (disk & tape) (mill.) |
Print Lines(x 1000) |
Elapsed Time |
CPU(hours) |
Total I/O (disk & tape) |
Print Lines(x 1000) |
|
| 1. ABN online Base Component | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| 2. Supersearch (Advanced Search) online | 99% | 7.5% | 6.7% | na | 99% | 5.8% | 5.3% | na |
| 3. Ozline (hosted citation databases) | 0.7% | 0.8% | na. | 0.6% | 1% | na. | ||
| 4. ABN batch (*) | 36% | 13% | 4958% | 30% | 10.3% | 2782% | ||
| 5. Supersearch batch & admin | 9.6% | 5% | 3.7% | 506% | 12.9% | 3.5% | 3.6% | 348% |
| 6. Ozline Data Creation (AMI & APAIS)(++) | 16.3% | 0.6% | 1.3% | 888% | 11.9% | 0.5% | 0.6% | 146% |
| 7. ABN Database Admin(**) | 39% | 37% | 12% | 269% | 28% | 9% | 152% | |
| About This Site | Copyright | Privacy | Accessibility | Site Map | Site Search | Content A-Z | Contact Us |