An architecture for the coordination of system management services.

In an enterprise, a large fraction of the budget spent on information technology (IT) is associated with providing system management services. The primary objective of system management services is the continual operation of servers, desktops, and laptops in the enterprise. Typical of the services provided by system management are deployment services (initial configuration, software distribution, and installation), support services (help-desk support and troubleshooting), preventive maintenance services (upgrades, backups, and virus scanning), and other administrative services (asset management, user management, and license management). Today these services are typically provided by the IT staff.

System management tasks are complex, error-prone, and training- and labor-intensive. In current practice, a system administrator invokes services manually through the facilities of one of more consoles. Typically, each console controls one service, and the system administrator is responsible for mentally mapping the desired actions into the specific service invocation syntax. Multistep service invocations are often stored as scripts whose sequence is fixed but whose parameters can be supplied at run time. Taken together, these scripts represent an enterprise-specific integration of system management components. These scripts must be constructed from scratch for each enterprise in low-level programming languages, and they constitute a significant management and maintenance task in themselves.

The increasing complexity and integration of enterprise IT is reflected in the increasing complexity of system management. In order to reduce the total cost of ownership associated with system management, enterprises have begun to standardize software applications and streamline IT processes used in their enterprise.

The next step in streamlining the IT infrastructure is to deliver the standard set of services and processes through a shared infrastructure with a consolidated help desk and IT staff, or to outsource the management of the IT infrastructure to other companies that specialize in IT outsourcing. We define a system management utility as delivering IT services to multiple customers over the Internet from a shared infrastructure. (In the rest of the paper, when there is no ambiguity, we use the term utility to mean system management utility.) The goal of such a utility is to deliver IT services at reduced cost through standardization, automation, and leveraging economies of scale.

Effective automation in service deployment, delivery, and problem determination requires a high degree of coordination among services and the objects managed by the services. We observe that the current approaches to system management are not designed with coordination and automation as requirements as follows:

1. Services maintain information in silos and do not automatically take advantage of information gathered by other services. This situation makes any multiservice use of system management services IT administrator-centric. To perform any system management task, the system administrator has to analyze the requirements, plan an approach, execute steps, and monitor success across services.

2. System management tools and services do not automatically customize themselves to the configurations being managed. As a result, the configuration, deployment, and coordination of system management services is labor-intensive.

3. System management services and applications are enterprise-centric: most services and applications treat trust and privacy issues from an enterprise perspective.

4. System management services are configured with certain built-in assumptions about the managed systems. They are not configured to adapt to a changing environment. For example, broken dependencies or unsatisfied prerequisites are the main reason when management procedures do not complete their execution normally.

In summary, the existing services and practices do not lend themselves easily to the utility computing model. Although there is a clear need for delivering system management services using the utility model, existing system management services are not sufficient for realizing this goal.

To overcome these difficulties, today's system management services could be redesigned and re-implemented, or new technologies could be developed that would allow the use of existing services while enabling automation by some other means. We take the latter approach here and describe a method and an architecture for developing an infrastructure that is highly automated and scalable, yet adaptive to each specific managed element in its unique context. With this approach, system administrators are relieved from the burden of performing repetitious tasks and are able to focus more on complex tasks such as solution architecture and planning.

Several key contributions are presented in this paper. We describe how to use a rule-based system to model policies, profiles, and dependencies of requirements. We also describe an architecture in which a repository of rule-based objects is used for automated configuration and sequencing of multiple services. This architecture also handles management tasks related to spontaneous changes in remote systems. The organization of the objects in the repository makes it possible to reason about actions to take and which services to deploy.

In the next section of this paper, we describe objectives and requirements of a system management utility and discuss the shortcomings of existing solutions to meet these objectives and requirements. We conclude the section by describing some of the salient points of our approach. In the third section, we introduce rules and discuss how they can be used to represent policies, profiles, and requirements. The architecture of our approach is discussed in the fourth section. We present some design issues after that, and in the sixth section we describe an outline of a prototype implementation of this approach. Other work in the literature related to this work is discussed in the seventh section. Finally we conclude with a summary and some directions for future work.

Delivering system management services as a utility

In this section we first discuss the objectives and requirements of a utility. We then discuss the shortcomings of current system management tools and services, and following that we describe intuitively the key ingredients needed to realize the objectives of the utility model.

Objectives and requirements of a utility. In the introductory section, we briefly described the objectives and principles used in managing systems. To offer system management services as a utility, some additional considerations must be taken into account. By its nature, a utility has to cater to a diverse set of customers, each customer having its own unique requirements. At the same time, to be successful as a viable business, the utility cannot assign expert system administrators individually to each customer. It must replicate its services economically and be able to customize them before deployment on customer systems. In particular, a system management utility has the following three key objectives:

1. Provide policy-based system management and service administration

2. Provision services to customers on demand and according to service level agreements (SLAs) with the customers

3. Leverage economies of scale

Policy-based system management means adjusting the service behavior according to some customizable policy. The policy may be customer-, user-, or machine-specific. It can also be a policy specified by the utility. Ala example of a policy is performing incremental backup on a daily basis and a full backup on a monthly basis. At the machine or user level, the policy might specify the exact time of the day when the backup is to be performed.

"On demand" service provisioning means being able to configure and deliver a service just in time when the need arises. "SLA-driven" means that the quality of service is maintained at a certain prescribed level. An example of an on demand service is performing a nonroutine backup just before upgrading the operating system of a machine. An example of an SLA-driven service is maintaining a particular machine configuration in a usable state 99 percent of the time during normal working hours over a specified period of time.

To satisfy the first two objectives just described, for each kind of system management service, a utility has to manage multiple types of system management services (i.e., similar service functionality offered by different brands, versions, etc.) and needs to be capable of configuring and deploying different variants of a particular service, depending on the situation at hand.

The third objective is driven mostly by pragmatic considerations. Leveraging economies of scale means being able to amortize costs efficiently over a large customer base. In other words, costs associated with system management services should grow sublinearly as a function of the number of customers, machines, and users.

Clearly, for a utility to be successful as a business concept, large-scale automation is required to achieve the three key objectives listed above. Large-scale automation in configuring and coordinating services is necessary to provide system management services from a utility.