Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`

It’s not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It’s just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set state to MASTER and priority to 101, and on the second set them to BACKUP and 75. Also, do yourself a favor and generate the passwords with uuidgen. I’ve added a password generated with uuidgen, but don’t use the one I’m pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`

It’s not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It’s just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set state to MASTER and priority to 101, and on the second set them to BACKUP and 75. Also, do yourself a favor and generate the passwords with uuidgen. I’ve added a password generated with uuidgen, but don’t use the one I’m pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``

```Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`

It’s not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It’s just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set state to MASTER and priority to 101, and on the second set them to BACKUP and 75. Also, do yourself a favor and generate the passwords with uuidgen. I’ve added a password generated with uuidgen, but don’t use the one I’m pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`Setting up a modern, highly available system without external dependancies is a tricky business. It’s even tricker to think through when doing it on a shoestring budget. However, it’s critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can’t afford for it to go down when other systems are offline, so it’s design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it’s not as hard as it sounds, and it’s free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, not load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there’s a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and also usually means spanning those VLANs. Networking groups don’t usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it’s possible, but I digress.

Once they are on the same segment, it’s time to setup keepalived. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We’re going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:

`

It’s not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It’s just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set state to MASTER and priority to 101, and on the second set them to BACKUP and 75. Also, do yourself a favor and generate the passwords with uuidgen. I’ve added a password generated with uuidgen, but don’t use the one I’m pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``

Once that's done, and the config files are in place on both hosts, fire up keepalived on the master. There will be some messages in /var/log/messages talking about keepalived coming online, setting up loopbacks and entering MASTER state. You should now be able to ping the machine externally, and SSH into the virtual IP address you specified above. Bringing the second instance of keepalived online should start a short conversation in the logs – but the MASTER should remain master.

If there's no conversation in the logs, check the virtual\_router\_id and auth_pass in the config files. If they match, and there's still no conversation, you may have to poke iptables, as you need to explicitly allow vrrp traffic in iptables. In `/etc/sysconfig/iptables`, add the following:

````Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
`Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
` 

It's not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It's just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set `state` to `MASTER` and `priority` to `101`, and on the second set them to `BACKUP` and `75`. Also, do yourself a favor and generate the passwords with `uuidgen`. I've added a password generated with uuidgen, but don't use the one I'm pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
`Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
` 

It's not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It's just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set `state` to `MASTER` and `priority` to `101`, and on the second set them to `BACKUP` and `75`. Also, do yourself a favor and generate the passwords with `uuidgen`. I've added a password generated with uuidgen, but don't use the one I'm pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

`` 
  
```Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
`Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
` 

It's not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It's just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set `state` to `MASTER` and `priority` to `101`, and on the second set them to `BACKUP` and `75`. Also, do yourself a favor and generate the passwords with `uuidgen`. I've added a password generated with uuidgen, but don't use the one I'm pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

``Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
`Setting up a modern, highly available system without external dependancies is a tricky business. It's even tricker to think through when doing it on a shoestring budget. However, it's critical for many operations. My employer, for example, keeps a set of tools available to administrators on a well-secured web server. This system is a front-end to complex Nagios deployment, user management tools, system maintenance tools. We can't afford for it to go down when other systems are offline, so it's design principals are much like that of a monitoring system: everything possible is internal to the system. Luckily, it's not as hard as it sounds, and it's free.

If you use CentOS or RHEL, you can follow this guide pretty much directly. Other flavors will have to adapt as needed to reflect local packaging standards, service administration and filesystem layouts. This guide is going to focus mostly on setting up keepalived to act as a failover mechanism, while also looking at replicated MySQL and failure detection for apache. Also, keep in mind this is about High Availability and failover, _not_ load balancing.

First things first: you are going to want to have two servers that are pretty much identical. It helps in the long run for sizing and replacements. They are going to need to be somewhat beefy, as they are going to host the web servers and the database servers that administrators are going to hammer when there's a system outage. There is nothing quite like melting the face off the server that the admins are using to put everything else back together. Also, put the servers in different data centers/different buildings, so when power goes out to one, the other is still up. The tricky bit is the servers are also going to need to be in the same Layer 2 broadcast domain, while being geographically separated. This usually means being on the same VLAN, and _also_ usually means spanning those VLANs. Networking groups don't usually like this. The alternative is MPLS and VPLS, which is expensive – at least if you use Cisco gear – and well beyond my knowledge. I understand it's possible, but I digress.

Once they are on the same segment, it's time to setup [keepalived][1]. Keepalived is usually used to act as a load balancer, and uses VRRP to pass the “router” address back and forth between themselves for redundancy. We're going to use VRRP to do most of our magic today. Good news: on most modern RedHat-based distributions, installing keepalived is as simple as:
  
` 

It's not part of the Advanced Platform Server Magick that RedHat sells at a premium price. It's just basic, vanilla, plain-Jane keepalived. Your config file is equally as simple, with two minor differences between the two hosts that will be sharing the service. On the first, set `state` to `MASTER` and `priority` to `101`, and on the second set them to `BACKUP` and `75`. Also, do yourself a favor and generate the passwords with `uuidgen`. I've added a password generated with uuidgen, but don't use the one I'm pasting in here. Really. Also, give yourself a real IP address, not 1.2.3.4. (Honestly, if you use a password copied off a blog post and the IP address 1.2.3.4… well, you deserve what you get).

`` 
  

Once that’s done, and the config files are in place on both hosts, fire up keepalived on the master. There will be some messages in /var/log/messages talking about keepalived coming online, setting up loopbacks and entering MASTER state. You should now be able to ping the machine externally, and SSH into the virtual IP address you specified above. Bringing the second instance of keepalived online should start a short conversation in the logs – but the MASTER should remain master.

If there’s no conversation in the logs, check the virtual_router_id and auth_pass in the config files. If they match, and there’s still no conversation, you may have to poke iptables, as you need to explicitly allow vrrp traffic in iptables. In /etc/sysconfig/iptables, add the following:

````

Once the hosts talk to each other, test failover. Turn off keepalived on the first host. It will take 2 or 3 seconds to fail over to the second host. At that point, you’ll be able to ping, but your SSH client will scream at you about a key mis-match. This is expected, and what you want to see. Success! The VIP has moved. I’ll remind you again, this is HA, not load balancing. When you bring keepalived back online, the IP will move back to the host you have set as MASTER in the config files.

At this point, you’ve done half of the hard work. As a breather, fire up apache on both boxes and fail keepalived back and forth between the hosts, and you’ll be able to see traffic going to one or the other. It’s pretty slick. In the next installment, I’ll talk about MySQL replication between the hosts.