Nginx High Availability Clustering with NGINX Plus

As part of NGINX Plus, we have included a script that creates a highly available, active/passive pair of NGINX Plus instances.

Overview:

It is often desirable to have IP addresses that are highly available so that if a machine that is hosting an IP address fails, the IP address can be hosted on another machine. This can be accomplished by creating a cluster and using a Virtual IP address (VIP) that can move between cluster members. There are different clustering solutions available for Linux, but the solution that is delivered with NGINX Plus uses Pacemaker and Corosync. This solution is limited in scope to creating a simple configuration consisting of two NGINX Plus instances and a single VIP. This configuration can be used as the basis for developing a more sophisticated solution by directly modifying the Pacemaker configuration.

The NGINX solution is delivered as a package named nginx-ha and includes a script that installs and configures Pacemaker and Corosync. Once deployed, you will have a cluster of NGINX Plus servers running as an active/passive pair with one shared VIP. If the active instance fails, the passive instance will become the active instance and it will raise the VIP and start listening for traffic.


Installing the nginx-ha package:

The first step in setting up an HA pair of NGINX Plus instances is to install the nginx-ha package on both instances. There are two versions of this package, one for generic Linux installations and the other for the NGINX Amazon AMI’s. The AMI’s are already configured to access this package and for generic Linux installations, the certificates and instructions used to install the nginx-plus package will also work for the nginx-ha package.

If you are using RHEL/CentOS or Amazon Linux you install nginx-ha with: yum install nginx-ha

If you are using Ubuntu you install nginx-ha with: apt-get install nginx-ha

---

Running the script:

Once you have installed the package on both NGINX instances, you need to run sudo nginx-ha-setup script on both instances, at the same time. The information that the script will prompt for depends on if it is being run on generic Linux or an Amazon AMI. For a generic Linux environment it will prompt for:

• The local IP you want to use for cluster communications
• The IP of the other instance
• The virtual IP to be made highly available
• Whether the two nodes are equal or if one node is preferred

Note for Red Hat users: You have to have the Red Hat High Availability Add-On for access to the ha-clustering repositories before running nginx-ha-setup.​

For an Amazon AMI environment, the script will prompt for:

• The Amazon Access Key ID
• The Amazon Secret Access Key
• The Amazon instance ID of the other instance
• The Host Name
• The Elastic IP to be made highly available

Working with the cluster:

Once the script has completed, you should have a functioning two node active/passive cluster. You can check the status by running the command: sudo crm status bynode

For example, with two nodes named nginxha100 and nginxha101, and with the VIP running on nginha100, the output from sudo crm status bynode would be something like:

============
Last updated: Wed Mar 19 02:46:49 2014
Last change: Wed Mar 19 02:46:42 2014 via cibadmin on nginxha101
Stack: openais
Current DC: nginxha101 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Node nginxha100: online
ha-ip (ocf::heartbeat:IPaddr2) Started
ha-nginx (ocf::nginx-ha:nginx-ha) Started
Node nginxha101: online​

If everything is working correctly, you should see the same output when running this command on both nodes. If you want to see the complete Pacemaker configuration you can us the command: sudo crm configure show



Notes:

The following are some issues to be aware of:

• The NGINX configuration files on both nodes must have the services defined that are being made highly available. Keeping the configuration files in sync is outside the scope of this article.

• If you need to manually start or stop Corosync and Pacemaker, you should start Corosync first and then Pacemaker and you should stop Pacemaker first and then Corosync. To start or stop Corosync and Pacemaker you can use the commands:

sudo /etc/init.d/corosync <start or stop>​

sudo /etc/init.d/pacemaker <start or stop>​

• When attempting to do a graceful shutdown of a machine, by for example running the command reboot without –force, Corosync may refuse to shut down because Pacemaker has not shutdown. Therefore, when trying to do a grace shutdown it is best to first manually stop Pacemaker.

• The default timeout used by Corosync to consider a node as down is 1 second. If you wish to change this, you must modify the Corosync configuration. The file to edit depends on the environment. For generic Linux, the file is /etc/corosync/corosync.conf. For Amazon AMI’s, the corosync.conf file is created at runtime, so you must edit /etc/nginx-ha/templates/corosync.conf.templ. In either case, add the following line to the totem section of the file:

token: <timeout in milliseconds>​

The corosync files are not automatically synchronized across nodes, so you must make the change on both nodes.​

Adding more virtual ip addresses:

The configuration created by nginx-ha-setup is very basic, and makes a single IP address highly available. It is often desirable to have more then one IP address highly available (note: when running in Amazon EC2, only one IP is supported). This can be done by modifying the Pacemaker configuration. Here is an example that adds two additional IP addresses, 192.168.100.2 and 192.168.100.3. The first thing we need to do is add a resource for both IP addresses. The nginx-ha-setup script gives the name ha-ip to the IP address it configures. We will name the new IP addresses ha-ip-2 and ha-ip-3. The following commands will create these new resources (Note that the Pacemaker configuration is automatically synchronized between the two nodes so these commands can be run on either node and only need to be run once):

sudo crm configure primitive ha-ip-2 ocf:heartbeat:IPaddr2 params ip=”192.168.100.2″ op monitor interval=”10s”​

sudo crm configure primitive ha-ip-3 ocf:heartbeat:IPaddr2 params ip=”192.168.100.3″ op monitor interval=”10s”​

Next, we must tell Pacemaker that we want these new IP addresses to be on the same machine as the existing IP address. To do this we must edit the existing resource, “ha-ip-with-nginx”. This will bring up vi to edit the resource.

To edit the resource use the following command:

sudo crm configure edit ha-ip-with-nginx​

Make the following change:

From: colocation ha-ip-with-nginx inf: ha-ip ha-nginx​

To: colocation ha-ip-with-nginx inf: ha-ip ha-ip-2 ha-ip-3 ha-nginx​

Once the change has been saved, the two new IP addresses should be raised on the active node. Running sudo crm status bynode should show the output similar to the following, assuming the same setup as the previous example:

============
Last updated: Wed Mar 19 17:22:05 2014
Last change: Wed Mar 19 17:21:45 2014 via cibadmin on nginxha100
Stack: openais
Current DC: nginxha100 – partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
4 Resources configured.
============

Node nginxha100: online
ha-nginx (ocf::nginx-ha:nginx-ha) Started
ha-ip (ocf::heartbeat:IPaddr2) Started
ha-ip-2 (ocf::heartbeat:IPaddr2) Started
ha-ip-3 (ocf::heartbeat:IPaddr2) Started
Node nginxha101: online​

More complex configurations:

It is possible to create more complex configurations using Corosync and Pacemaker, including active/active configurations. Please consult the Pacemaker documentation for more information: http://clusterlabs.org/doc/

​

The post High Availability Clustering with NGINX Plus appeared first on NGINX.

Continue reading...