Minimizing MXAE downtime

Managing multiples services running on multi servers could be hard work and definitely hate it. Even know we can do simple script or cronjob to restarting the services by schedule but it’s not preffered way to manage it. When do some research, I was crossed by Monit service monitoring and quite interesting to implement it to MXAE environment. Here is a banner published in monit website that can explain what Monit can do.

What Monit can do
“Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resource. You can use Monit to monitor files, directories and filesystems for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; Monit can ping a remote host and can check TCP/IP port connections and server protocols. Monit is controlled via an easy to use control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions and recovery status via customizable alert.” – Source http://mmonit.com/monit/

Setting up
Setting up Monit is easy and for this purpose of introduction I’m using:

• CentOS 5.5
• MXAE 2.0
• Latest Monit from RPMForge package
• SSH / Terminal console

As the package was not included in official CentOS, We had to download it from RPMForge instead. You can find a complete listing of the RPMforge package packages at http://packages.sw.be/.
Download the rpmforge-release package. Choose one of the two links below, selecting to match your host's architecture. If you are unsure of which one to use you can check your architecture with the command uname –i by using SSH or Console.

• i386 http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.i386.rpm
• x86_64 http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm

The preferred rpmforge-release package to retrieve and to install in order to enable that repository is one of the two listed above.

 
Next, Install DAG's GPG key :

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt

And verify the package you have downloaded

rpm -K rpmforge-release-0.5.2-2.el5.rf.*.rpm 

Install the package

rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm

That’s it, we just installed rpmforge and included their repository to yum. After everything went well, now we can check Monit package by executing command:

Yum search monit

We will get the search result like:

.
.

libwnck-devel.x86_64 : Libraries and headers for libwnck
lm_sensors.i386 : Hardware monitoring tools.
lm_sensors.x86_64 : Hardware monitoring tools.
logwatch.noarch : A log file analysis program
mdadm.x86_64 : mdadm controls Linux md devices (software RAID arrays)
mon.x86_64 : General-purpose resource monitoring system
monit.x86_64 : Process monitor and restart utility
monitor-edid.x86_64 : Tool for probing and parsing monitor EDID
mrtg.x86_64 : Multi Router Traffic Grapher
mtop.noarch : Tool to monitor a MySQL database
mytop.noarch : Top clone for MySQL

.
.

 
Ok, so we found it on bolded line. Now, we are going to install it by executing command:

yum install monit.x86_64

By executing this command, yum will download and install to the server accordingly. Press Y to confirm to download and leave it. Yum will help you to do everything to make sure it’s installed and running properly.
Once yum installed the package to the server, now we have to edit Monit config file to monitor MXAE services (apachectl, mysqld and proftpd). Start editing the file /etc/monit.conf and add the following configuration into it:
 
------------------START CONFIG-------------------
set daemon  120 # check services at 2-minute intervals
# Apache monitoring
check process apache with pidfile /usr/local/mxae/apache2/logs/httpd.pid
   group siteadm
   start program = "/etc/init.d/apachectl start"
   stop  program = "/etc/init.d/apachectl stop"  
   if 5 restarts within 5 cycles then timeout
   depends on apache_bin
   depends on apache_rc
 
 check file apache_bin with path /usr/local/mxae/apache2/bin/apachectl
   group root
   if failed checksum then unmonitor
   #if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor
 
 check file apache_rc with path /etc/init.d/apachectl
   group root
   if failed checksum then unmonitor
   #if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor
 
# FTP monitoring
check process proftpd with pidfile /usr/local/mxae/var/proftpd.pid
   start program = "/etc/init.d/proftpd start"
   stop program  = "/etc/init.d/proftpd stop"
   if failed port 21 protocol ftp then restart
   if 5 restarts within 5 cycles then timeout
 
 
# MySQL database monitoring
check process mysql with pidfile /usr/local/mxae/mysql/data/mdaweb.pid
   group mysql
   start program = "/etc/init.d/mysqld start"
   stop program = "/etc/init.d/mysqld stop"
   if failed host localhost port 3306 protocol mysql then restart
   if 5 restarts within 5 cycles then timeout
   depends on mysql_bin
   depends on mysql_rc
 
 check file mysql_bin with path /usr/local/mxae/mysql/bin/mysqld
   group mysql
   if failed checksum then unmonitor
   #if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor
 
 check file mysql_rc with path /etc/init.d/mysqld
   group mysql
   if failed checksum then unmonitor
   #if failed permission 755 then unmonitor
   if failed uid root then unmonitor
   if failed gid root then unmonitor
------------------END CONFIG-------------------
 
Once finish editing save it and start monit service by executing following command:

Service monit start

To stop the monit service:

Service monit stop

To restart:

Service monit restart

 
We done!
Now whenever process down, Monit will bring-up it back for you. To try out, you can try to kill the process and wait for few second and check that process whether it’s running.
There is a lot services can be monitor using Monit. Assuming you’re successfully configured it. Now you can sit and relax and visit complete examples that can be found at: http://mmonit.com/wiki/Monit/ConfigurationExamples