Management of application complexes in multitier clustered systems. (Technical forum).

Clustered systems are often used as a platform for running different services that compete for the same server resources. For e-commerce applications, these services are provided by different software components, such as load balancers, Web servers, application servers, and database servers. The software components are organized in a multitier architecture and cooperate to provide the e-commerce service. The two main challenges involved in the management of these services are life-cycle management and management of capacity-on-demand. Life-cycle management consists of deploying, monitoring, and maintaining the healthy operational state of the e-commerce service. This requires an understanding of the software dependencies, the management capabilities of the involved components, and their intertier relations. To date, most of these management tasks were performed manually (and hence were error-prone) and there was no holistic view of a service, which led to a significant increase in the total cost of ownership. Managing capacity-on-demand poses another challenge as the workload patterns of many services, especially commercial Internet services, display significant variance. The inability of current management systems to provide dynamic allocation of servers to such services leads to inefficient utilization of the cluster.

This paper describes the Raquarium project, which is aimed at meeting these challenges. Raquarium introduces the notion of "application complex," an entity that significantly simplifies the life-cycle management of services as well as the management of capacity-on-demand. It provides IBM customers with the ability to allocate resources to xSeries * servers in response to changes in load conditions and the allocation properties specified by the customer. Raquarium is integrated seamlessly with IBM Director, (1) the xSeries (2) systems management product. Provisioning supported by Raquarium in the initial release is restricted to configuration of preprimed servers, that is, those preloaded with operating systems (OSs) and applications. Future releases will take advantage of remote deployment capabilities to supply the OS and application images on "bare metal" servers, as well as configure network security using VLAN (virtual local area network).

During the last several years, a few commercial products have come to the market offering provisioning support. For example, Jareva Technologies, Inc. offers BladeForce Suite, OpForce Suite, and Elemental Server as software products that support deploying of OS and application software and configuring of networks using VLAN. (3) ThinkDynamics Inc. offers ThinkControl Suite, which automates allocating of server and network resources in fulfillment of service level objectives. (4) Terraspring, Inc. provides Control Center, a browser-based user interface that enables an operator to design server farms comprising server, network, and storage resources. (5) The design is captured as an FML (Terraspring's Farm Markup Language) document for later configuration of actualphysical resources. These products support multiple OS and hardware platforms.

The Raquarium project stems from the Oceano project (6) (the name "Raquarium" was inspired by the "rack"-mounted servers and "acquarium" as carrying ideas from Oceano). Raquarium's generalized application management framework and its integration with IBM Director was beyond the scope of Oceano, a proof-of-concept prototype.

Application complexes and their management

An application complex is composed of multiple tiers, where each tier contains servers that run a specific application component. A typical three-tier application complex may consist of a load balancer, a number of Web servers and a number of application servers. The application complex entity presents a view in which the inter-relations and dependencies, the deployment properties, and operational characteristics of the components are hidden. Raquarium provides the following life-cycle management functions for application complexes:

* Automatic deployment--configures the participating servers to work together, each with its specific application component and role.

* Performance monitoring and analysis--is based on the structural knowledge of the entire application complex.

* Automatic server allocation and configuration--provides capacity-on-demand management and hardware failover support.

The initial release of Raquarium provides no network isolation such that access between servers within a cluster is not restricted. This is not a problem as long as the entire cluster is owned by a single customer. The initial release also restricts provisioning to preinstalled applications. Future releases will provide OS and application deployment based on image repositories, and also application complex isolation based on VLANs in order to support multiple customers environments.

Raquarium is implemented as a management framework (a container-style implementation) where application complexes are introduced as "plugins" that provide the specific wisdom of their application complex type, by implementing a "Configuration Provider Interface." The Configuration Provider plugin defines the structure of the application complex, including the structure of the tiers, the application component in each tier, whether it is scalable by adding instances of the applications, and whether it is shareable with other application complexes. The plugin also identifies the properties that require the administrator to supply input values (e.g., threshold values for controlling events). It also collects and publishes monitoring data selected by the administrator via the management framework, generates status events, and issues requests to add or remove servers. The management framework (or simply "framework") provides the visualization GUI (graphical user interface) of application complexes and clustered servers, and manages the free server pool and the allocation of servers. Finally, it applies a power management policy on the free pool such that the servers will be powered off until they are needed.

As an example, consider a two-tier application complex in which the first tier consists of an IBM e-network dispatcher that acts as load balancer, and multiple Web servers comprising the second tier. When the plugin detects an excess load on the Web servers, it requests an additional Web server from the framework. If the framework determines that an available Web server can be selected (from the free pool), it then calls the plugin to perform the necessary configuration actions. These may include updating the network dispatcher IP (Internet Protocol) tables, provisioning the correct content on the Web server, and so on. The plugin would also be called to perform the necessary configurations if the same scenario is initiated by a user operation such as dragging a Web server icon from the free pool onto an application complex view.

Raquarium provides an SDK (software development kit) to assist in the development of application complex plugins that meet the Configuration Provider Interface. These plugins can be managed by the Raquarium framework to provide a consistent user experience for different types of application complexes. The plugins are loaded during initialization time and cannot be changed dynamically.

Integration of Raquarium into IBM Director

The Raquarium prototype is integrated into the IBM Director (Reference 1 provides an overview of the IBM Director architecture). IBM Director is an Enterprise Management System that provides management functions (e.g., performance monitoring, inventory queries, and process information) for individual Intel-based servers, as well as the ability to manage racks and server blade enclosures. The integration of Raquarium into IBM Director provides a new view of the multicomponent services and the ability to perform multiserver and multitier application complex configurations, using single drag-and-drop operations. In addition to this function, the administrator can use the same GUI to perform the standard management of any individual server that participates in the application complex.

Currently, Raquarium supports application complexes, called WebComplexes, that belong to the WebSphere * Edge Server domain. (7) The WebComplexes were added to IBM Director as a new generic managed object class. The Raquarium framework is integrated into the rack-manager component of the IBM Director. Figure 1 shows the association between the physical view of the servers (their location inside the rack enclosure) and the logical view (the application complex to which they belong). The tree panel on the left presents the logical tier structure of each of these application complexes. Specifically, it shows two application complexes: Support and Sales. Selecting the Support WebComplex on the left panel causes the participating servers to be highlighted on the right panel. The free servers (i.e., servers not allocated to any application complex) are presented according to their role under the free pool branch.

[FIGURE 1 OMITTED]