Tech Blog

In this post I’m going to take a look at a two-node CentOS 7 Apache active/passive cluster using mostly packages in the Red Hat cluster suite.  In case you don’t already know, CentOS is created by making use of the source packages provided by Red Hat Enterprise Linux (RHEL). 

You may sometimes see Red Hat referred to as the CentOS ‘upstream provider’.  In a previous post, I used a software component called GlusterFS for storage within a cluster.  Red Hat offers Red Hat Gluster Storage (formerly called Red Hat Storage Server) which is based on the GlusterFS system.  In this cluster, I’ll be using a different type of clustered storage called DRBDDRBD, or Distributed Replicating Block Device, is a package that keeps the block devices of 2 nodes in sync.  It is somewhat similar to a network RAID 1 system.  I’ll be adding that in part 2 of this series.

This series is not intended to go over every option and/or feature of clustering using the Red Hat cluster suite nor do I make any guarantees that this will work to suit your needs.  Instead, it is intended to get you started with the packages so hopefully you can build a cluster that works for you.  You will have to look at your environment and determine what is redundant and what is not and even what should be.  So think of this as an introduction to Red Hat clustering that takes a look at some possibilities.  In other words, use at your own risk and make it work for your particular environment.

A note for you Red Hat Enterprise Linux users.  Red Hat’s official stance on DRBD appears to be that using it as part of a cluster is a supported configuration, BUT you need to get a support contract with LINBIT, the maker of DRBD, if you require support.

So let's get started.  I think it would be beneficial to define some terms first:

Cluster – a group of computers or nodes that work together to accomplish some task.  An HA cluster as I will be setting up here is intended to provide a service or group of services with a minimum amount of downtime.

Split-Brain – I think the best way I can define this is to give you an example.  Let’s say we have a 3-node cluster and the network communication fails between one node and the rest of the nodes in the cluster.  Which node or nodes has the right to present the clustered resource(s)?  If one of the 2 systems that are still communicating on the network presents the cluster resources and so does the node that has become disconnected, we have a split-brain.  If changes are made to those resources and then all communications return to normal, which node or nodes have the correct information?  As you may imagine, this is not something we want to happen so we must take appropriate measures to prevent this.  So we can define split-brain as a condition that occurs when one side doesn’t know about the other so each try to take over the clustered resource(s).

Fencing – This is a process of removing or isolating a node or nodes of a cluster when something is wrong.  The fencing method is intended to keep split-brain scenarios from happening.  It is critical that your cluster takes this into consideration!

STONITH – Short for “Shoot The Other Node In the Head”, this is a mechanism for fencing or isolating a node from a cluster.  The intent is to keep a misbehaving node from interfering with the proper operation of the cluster and to help prevent split-brain.

Quorum - The minimum number of nodes required for the cluster to function properly.  It's primary purpose is to eliminate split-brain.  Normally this is considered to be half the total number of nodes plus one.

The primary components I’ll be using for this cluster are as follows:

Corosync – This is the cluster engine or messaging layer used for communication between nodes.

Pacemaker – This is the resource manager that will take care of starting/stopping services as well as the necessary logic.

pcs/pcsd – This is the command and daemon used to control and configure Corosync and Pacemaker.

DRBD – As previously mentioned, this will provide the clustered storage system.

CLVM – The clustered logical volume manager.

CentOS 7 - Minimal install followed by updating it via “# yum –y update”.

Before going any further, this is probably a good time for an important note.  This setup is specifically for RHEL/CentOS v7.  That is important to note as Red Hat made changes to their cluster suite between v6 and v7.  I won’t spend time going into detail, just know that some of the software packages and commands are different.

For this setup, we will be using two hosts in the cluster that will share a virtual IP address (192.168.1.200), an Apache httpd instance, and a partition to hold the Apache web data.  The systems should have static IP addresses and should have NetworkManager disabled.  The two systems I’m using are fully patched (as of this writing) CentOS 7 systems with firewalld and SELinux enabled as shown below:
node1.theharrishome.lan -> 192.168.1.180
node2.theharrishome.lan -> 192.168.1.181
vip.theharrishome.lan -> 192.168.1.200

To begin, let’s make sure we have accurate time on each system.  We’ll configure NTP to take care of that as follows:

# yum –y install ntp
# systemctl enable ntpd
# ntpdate pool.ntp.org
# systemctl start ntpd

Now let’s make sure we can resolve each machine from the other.  To do so, I modified my /etc/hosts file by adding the following lines:

192.168.1.180   node1 node1.theharrishome.lan
192.168.1.181   node2 node2.theharrishome.lan
192.168.1.200   vip vip.theharrishome.lan

Before moving forward, verify that each can ping the other by name.  

Let’s make a few firewall rules on each system.  The following 2 are for corosync:
# firewall-cmd --permanent --add-port=5404/udp
# firewall-cmd --permanent --add-port=5405/udp

The next one is for pacemaker:
# firewall-cmd --permanent --add-port=2224/tcp

And for Apache:
# firewall-cmd --permanent --add-port=80/tcp

Now let’s reload the firewall with our new settings:
# firewall-cmd –reload

And now to install some software on both nodes:
# yum –y install corosync pacemaker pcs httpd

You will notice several dependencies are also installed so now is a good time to grab a cup a joe J

We need to disable Apache (httpd) from starting automatically on both nodes as we want the cluster to handle that instead:
# systemctl disable httpd

Now let's make a simple html page to test with on both nodes.  The following command should take care of that:
# vi /var/www/html/index.html

Enter and save the following:

<html>
<body> Test on Node1</body>
</html>

Make sure and change the host name in the file for each respective node.

In order to monitor the health of your Apache instance and recover it if it fails, the resource agent used by Pacemaker assumes the server-status URL is available. On both nodes, enable the URL with:
# vi /etc/httpd/conf.d/status.conf

Enter and save the following:

<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>


Now we need to run corosync-keygen so that the communication between nodes is encrypted.  We only need to do this on one node as we will then copy the key to the other node.  The command to generate the key is as follows and note also that it may take a few minutes to complete:
# corosync-keygen
You will be asked to press keys on your keyboard to generate entropy.  Then wait . . .

Time to create the corosync config file of /etc/corosync/corosync.conf.  The installation comes with an example config file at /etc/corosync/corosync.conf.example that you could copy and modify but to get started, here is an example of the one I am using.  Make sure to change the entries as noted to match your environment.  You will need to copy this to each node along with the key generated above which should be at /etc/corosync/authkey.  The command I used was as follows:
# scp corosync.conf authkey This email address is being protected from spambots. You need JavaScript enabled to view it.:/etc/corosync/

Here is the contents of my /etc/corosync/corosync.conf:

logging {
        fileline: off
        to_logfile: yes
        logfile: /var/log/cluster/corosync.log
        to_stderr: no
        debug: off
        timestamp: on
        to_syslog: yes
        logger_subsys {
                subsys: QUORUM
                debug: off
        }
}

totem {
        version: 2
        token: 3000
        secauth: on
        crypto_cipher: aes256
        crypto_hash: sha256
        cluster_name: cluster1
        rrp_mode: active
        interface {
                ringnumber: 0
                # Change bindnetaddr and mcastaddr to match your environment.  See 'man corosyn
                bindnetaddr: 192.168.1.0
                mcastaddr: 226.94.1.1
                mcastport: 5405
        }
}

nodelist {
  node {
        # Change the following node entry to match your environment
        ring0_addr: node1.theharrishome.lan
        nodeid: 1
       }
  node {
        # Change the following node entry to match your environment
        ring0_addr: node2.theharrishome.lan
        nodeid: 2
       }
}

quorum {
        provider: corosync_votequorum
        expected_votes: 2
}

With the files copied, let’s start corosync on both nodes and then set it to start on boot:
# systemctl start corosync;systemctl enable corosync

Let’s do some checks to see what our new corosync status looks like.  On each node, run '# corosync-cfgtool –s' and see if you get something like the following:

No faults . . . sounds good so far.  Let’s check the object database with the '# corosync-cmapctl'  tool and grep for members to see what we can find:

Still looking good.  Let’s check the cluster’s quorum status:
# corosync-quorumtool

Still looking good so we’ll move on to Pacemaker.  Let’s start it on both nodes and then set it to start at boot:
# systemctl start pacemaker;systemctl enable pacemaker

Followed by pcs so we can configure Corosync and Pacemaker:
# systemctl start pcsd;systemctl enable pcsd

Now we need to change the password for the hacluster user account that was created when pcs was installed.  The standard passwd command can be used.  It is a good idea to set the hacluster password the same on both nodes.  So the following should do the trick:
# passwd hacluster

Now we need to authenticate pcs to both nodes using the hacluster user and the password we just set.  We do this from a single node:
# pcs cluster auth node1.theharrishome.lan node2.theharrishome.lan

At this point, we should have all we need for pcs such that any change we make via the pcs command will make the necessary changes on both nodes.  So let’s set up our cluster from a single node as follows:
# pcs cluster setup --name cluster1 node1.theharrishome.lan node2.theharrishome.lan --force

Note the –force at the end of the command.  That is necessary in order for pcs to overwrite the existing pacemaker config file.

Now let’s start the cluster, again from a single node:
# pcs cluster start --all

To check the status, issue the following:
# pcs status

This tells us that both nodes are online, the pcs daemon is talking to them both and corosync, pacemaker, and pcsd are all active and enabled.  It also tells us which node has quorum and that we have no stonith and stonith-enabled is NOT false.  More on what that all means in a bit but for now, let’s bask in the glow of our brand new cluster!

A couple of interesting notes here.  I’m not sure why, but on numerous occasions I end up with something such as follows under daemon status:

If you get that, just enable the services again (# systemctl enable corosync;systemctl enable pacemaker) and all should be well.

Another thing you may run into is something like this:

Generally a reboot on both nodes takes care of this.

We should now be able to successfully validate our cluster config with the crm_verify command.  If all is well, it won’t return anything which is good:
# crm_verify –L –V

Uh oh, problems!  Well, not really.  This is telling us that we do not yet have any STONITH resources.  Please note that it is CRITICAL that a production cluster with shared data NOT be ran without any STONITH devices.  I plan to set up shared data in part 2 of this series.  Since we have not yet got to that point, we can disable it for now with the following command:
# pcs property set stonith-enabled=false

Then run '# pcs status' again to verify all is well.

We need to add some resources so we actually have something to cluster.  There are six resource classes containing a bunch of different types of resource.  For our configuration, we will start by creating an IP address resources named VIP via the following command:
# pcs resource create VIP ocf:heartbeat:IPaddr2 ip=192.168.1.200 cidr_netmask=24 nic=eth0 op monitor interval=30s

Nothing will be returned if the command is successful.  So what in the world does that mean?  We are creating a clustered virtual IP of 192.168.1.200 with a class C netmask and setting up a monitoring operation to run every 30 seconds.

You should see the new resource if you run the ‘# pcs status’ command as shown:

Now take a look at the IP addresses on each node of the cluster by issuing the command ‘# ip addr’.  You should notice that one (node1 in my case) now has the virtual IP address of 192.168.1.200.  You should even be able to ping it.  Let’s go ahead and add Apache to the cluster as follows:
# pcs resource create Apache ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s

Again, issue ‘# pcs status’ to see your new resource:

We can see the status of our resources as well:
# pcs resource show

We want to ensure that both of these resources run on the same host.  In order to do so, we need to set up what is called a colocation constrait with INFINITY as a score.  This also means that if the VIP is not active, then Apache will not be able to run either.
# pcs constraint colocation add Apache with VIP INFINITY

If you then enter ‘# pcs status’ you won’t notice anyting different.  Instead, we need to issue the following command to show the constraints:
# pcs constraint show

Another adjustment we need to make is to set the VIP to always start BEFORE Apache.  The following command should take care of that:
# pcs constraint order set VIP Apache

And again, ‘# pcs constraint show’ should show our new constraint.

Since we will almost always deal with the VIP and Apache as a unit, let’s combine them into a group named WebSite:
# pcs resource group add WebSite VIP Apache

Here is a new screen shot of ‘# pcs status’:

And ‘# pcs constraint show’:

Now might be a good time to look at a few of the many tasks that can be accomplished by pcs.  I will just hit on some of them so it will be up to you to dig deeper.  Keep in mind that you can always ask for help by entering something like this:
# pcs –help
# pcs cluster –help
# pcs resource –help

Let's have a look at the test.html page we set up earlier.  When I enter the VIP IP address of 192.168.1.200 in a web browser I see the test page from node1.  Let’s move the WebSite resource to node2.theharrishome.lan.  Before the move, I see the following from ‘# pcs status’ (output shortened) showing me that the WebSite resource is on node1:

Then I issue the following command:
# pcs resource move WebSite node2.theharrishome.lan

And I check “# pcs status” again and I can see the WebSite resource has moved to node2:  Sure enough, now when I enter in the VIP address in a web browser, it now pulls up the test page on node1.

One thing to note however when running "pcs resource move" is that it adds a constraint to the resource so it won't move back.  We need to clear that constraint with the following command:
# pcs resource clear WebSite

What if we want to stop the resource?
# pcs resource disable WebSite

To start it again:
# pcs resource enable WebSite

Removing our resource group:
# pcs resource group remove WebSite

Remove a resource:
# pcs resource delete VIP

Remove our order constraint:
# pcs constraint order remove VIP Apache

Clean a resource
# pcs resource cleanup Apache
I find myself using this one quite a bit.  It basically tells the cluster to forget any history for the resource and instead to redetect its current state.  When you are setting up a cluster, you may occasionally get resource errors and many times this will resolve them.

Shew, we have come a long way.  We now have a functioning cluster!  We could use a utility such as rsync (one of my favorites) to keep the contents of the web directory in sync and actually that might work just fine for many situations.

Now that I hope you have a general idea how to set up a basic active/passive cluster using the Red Hat cluster tools, I'll let you in on a few secrets you might find interesting.  First, there is a very nice Java GUI called Linux Cluster Management Console (LCMC) located at http://lcmc.sourceforge.net/ that you can use to manage your cluster from any desktop that supports Java.  It can also be used to create a cluster from scratch.  If you do use it to set up your cluster, it will even install some of the needed tools and copy the files to each node in the cluster. 

The pcs daemon also has a nice web interface.  You can reach it on each node via https://{node name or ip}:2224.  Use the hacluster user and  password that you set up earlier.  You will need to add a node before you actually see any of the cluster info.

I didn’t mention these 2 GUIs until the end as I think it is important to understand the underlying commands necessary to pull all this together.  It should also be noted that there are many more options available within the config files and via pcs than I have touched on.  I would really recommend that you use this as a starting point and follow up by reading the documentation for a better understanding of all this.

In part 2, I’ll add storage to the cluster using DRBD so make sure and check back and thanks for reading!

- Kyle H.