Monday, July 21, 2014

How to set up a highly available Apache cluster using Heartbeat

A highly available cluster uses redundant servers to ensure maximum uptime. Redundant nodes mitigate risks related to single points of failure. Here's how you can set up a highly available Apache server cluster on CentOS.
Heartbeat provides cluster infrastructure services such as inter-cluster messaging, node memberships, IP allocation and migration, and starting and stopping of services. Heartbeat can be used to build almost any kind of highly available clusters for enterprise applications such as Apache, Samba, and Squid. Moreover, it can be coupled with load balancing software so that incoming requests are shared by all cluster nodes.
Our example cluster will consist of three servers that run Heartbeat. We'll test failover by taking down servers manually and checking whether the website they serve is still available. Here's our testing topology:
Topology The IP address against which the services are mapped needs to be reachable at all time. Normally Heartbeat would assign the designated IP address to a virtual network interface card (NIC) on the primary server for you. If the primary server goes down, the cluster will automatically shift the IP address to a virtual NIC on another of its available servers. When the primary server comes back online, it shifts the IP address back to the primary server again. This IP address is called "floating" because of its migratory properties.

Install packages on all servers

To set up the cluster, first install the prerequisites on each node using yum:
yum install PyXML cluster-glue cluster-glue-libs resource-agents
Next, download and install two Heartbeat RPM files that are not available in the official CentOS repository.
rpm -ivh heartbeat-*
Alternatively, you can add the EPEL repository to your sources and use yum for the installs.
Heartbeat will manage starting up and stopping Apache's httpd service, so stop Apache and disable it from being automatically started:
service httpd stop
chkconfig httpd off

Set up hostnames

Now set the server hostnames by editing /etc/sysconfig/network on each system and changing the HOSTNAME line:
The new hostname will activate at the next server boot-up. You can use the hostname command to immediately activate it without restarting the server:
You can verify that the hostname has been properly set by running uname -n on each server.

Configure Heartbeat

To configure Heartbeat, first copy its default configuration files from /usr to /etc/ha.d/:
cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/
cp /usr/share/doc/heartbeat-3.0.4/ /etc/ha.d/
cp /usr/share/doc/heartbeat-3.0.4/haresources /etc/ha.d/
You must then modify all three files on all of your cluster nodes to match your requirements.
The authkeys file contains the pre-shared password to be used by the cluster nodes while communicating with each other. Each Heartbeat message within the cluster contains the password, and nodes process only those messages that have the correct password. Heartbeat supports SHA1 and MD5 passwords. In authkeys, the following directives set the authentication method as SHA1 and define the password to be used:
auth 2
2 sha1 pre-shared-password
Save the file, then give it permissions of r-- with the command chmod 600 /etc/ha.d/authkeys.
Next, in, define timers, cluster nodes, messaging mechanisms, layer 4 ports, and other settings:
## logging ##
logfile        /var/log/ha-log
logfacility     local0hea

## timers ##
## All timers are set in seconds. Use 'ms' if you need to define time in milliseconds. ##

## heartbeat intervals ##
keepalive 2

## node is considered dead after this time ##
deadtime 15

## some servers take longer time to boot. this timer defines additional time to wait before confirming that a server is down ##
##  the recommended time for this timer is at least twice of the dead timer ##
initdead 120

## messaging parameters ##
udpport        694

bcast   eth0
## you can use multicasts or unicasts as well ##

## node definitions ##
## make sure that the hostnames match uname -n ##

Finally, the file haresources contains the hostname of the server that Heartbeat considers the primary node, as well as the floating IP address. It is vital that this file be identical across all servers. As long as the primary node is up, it serves all requests; Heartbeat stops the highly available service on all other nodes. When Heartbeat detects that that primary node is down, it automatically starts the service on the next available node in the cluster. When the primary node comes back online, Heartbeat sets it to take over again and serve all requests. Finally, this file contains the name of the script that is responsible for the highly available service: httpd in this case. Other possible values might be squid, smb, nmb, or postfix, mapping to the name of the service startup script typically located in the /etc/init.d/ directory.
In haresources, define to be the primary server, to be the floating IP address, and httpd to be the highly available service. You do not need to create any interface or manually assign the floating IP address to any interface – Heartbeat takes care of that for you: httpd
After the configuration files are ready on each of the servers, start the Heartbeat service and add it to system startup:
service heartebeat start
chkconfig heartbeat on
You can keep an eye on the Heartbeat log with the command tailf /var/log/ha-log.
Heartbeat can be used to for multiple services. For example, the following directive in haresources would make Heartbeat manage both Apache and Samba services: httpd smb nmb
However, unless you're also running a cluster resource manager (CRM) such as Pacemaker, I do not recommend using Heartbeat to provide mulitple services in a single cluster. Without Pacemaker, Heartbeat monitors cluster nodes in layer 3 using IP addresses. As long as an IP address is reachable, Heartbeat is oblivious to any crashes or difficulties that services may be facing on a server node.


Once Heartbeat is up and running, test it out. Create separate index.html files on all three servers so you can see which server is serving the page. Browse to or, if you have DNS set up, its domain name equivalent. The page should be loaded from, and you can check this by looking at the Apache log file in server1. Try refreshing the page and verify whether the page is being loaded from the same server each time.
If this goes well, test failover by stopping the Heartbeat service on The floating IP address should be migrated to server 2, and the page should be loaded from there. A quick look into server2 Apache log should confirm the fact. If you stop the service on server2 as well, the web pages will be loaded from, the only available node in the cluster. When you restart the services on server1 and server2, the floating IP address should migrate from the active node to server1, per the setup in haresources.
As you can see, it's easy to set up a highly available Apache cluster under CentOS using Heartbeat. While we used three servers, Heartbeat should work with more or fewer nodes as well. Heartbeat has no constraint on the number of nodes, so you can scale the setup as you need.

No comments:

Post a Comment