How to set up server monitoring system with Monit

http://xmodulo.com/server-monitoring-system-monit.html

Many Linux admins rely on a centralized remote monitoring system (e.g., Nagios or Cacti) to check the health of their network infrastructure. While centralized monitoring makes an admin's life easy when dealing with many hosts and devices, a dedicated monitoring box obviously becomes a single point of failure; if the monitoring box goes down or becomes unreachable for whatever reason (e.g., bad hardware or network outage), you will lose visibility on your entire infrastructure.
One way to add redundancy to your monitoring system is to install standalone monitoring software (as a fallback) at least on any critical/core servers on your network. In case a centralized monitor is down, you will still be able to maintain visibility on your core servers from their backup monitor.

What is Monit?

Monit is a cross-platform open-source tool for monitoring Unix/Linux systems (e.g., Linux, BSD, OSX, Solaris). Monit is extremely easy to install and reasonably lightweight (with only 500KB in size), and does not require any third-party programs, plugins or libraries. Yet, Monit lends itself to full-blown monitoring, capable of process status monitoring, filesystem change monitoring, email notification, customizable actions for core services, and so on. The combination of ease of setup, lightweight implementation and powerful features makes Monit an ideal candidate for a backup monitoring tool.
I have been using Monit for several years on multiple hosts, and I am very pleased how reliable it has been. Even as a full-blown monitoring system, Monit is very useful and powerful for any Linux admin. In this tutorial, let me demonstrate how to set up Monit on a local server (as a backup monitor) to monitor common services. With this setup, I will only scrach the surface of what Monit can do for us.

Installation of Monit on Linux

Most Linux distributions already include Monit in their repositories.
Debian, Ubuntu or Linux Mint:

$ sudo aptitude install monit

Fedora or CentOS/RHEL:
On CentOS/RHEL, you must enable either EPEL or Repoforge repository first.

# yum install monit

Monit comes with a very well documented configuration file with a lots of examples. The main configuration file is located in /etc/monit.conf in Fedora/CentOS/RHEL, or /etc/monit/monitrc in Debian/Ubuntu/Mint. Monit configuration has two parts: "Global" and "Services" sections.

Global Configuration: Web Status Page

Monit can use several mail servers for notifications, and/or an HTTP/HTTPS status page. Let's start with the web status page with the following requirements.

• Monit listens on port 1966.
• Access to the web status page is encrypted with SSL.
• Login requires monituser/romania as user/password.
• Login is permitted from localhost, myhost.mydomain.ro, and internal LAN (192.168.0.0/16) only.
• Monit stores an SSL certificate in a pem format.

For subsequent steps, I will use a Red Hat based system. Similar steps will be applicable on a Debian based system.
First, generate and store a self-signed certificate (monit.pem) in /var/cert.

# mkdir /var/certs
# cd /etc/pki/tls/certs
# ./make-dummy-cert monit.pem
# cp monit.pem /var/certs
# chmod 0400 /var/certs/monit.pem

Now put the following snippet in the Monit's main configuration file. You can start with an empty configuration file or make a copy of the original file.

1 2 3 4 5 6 7

set httpd port 1966 and      SSL ENABLE      PEMFILE  /var/certs/monit.pem      allow monituser:romania      allow localhost      allow 192.168.0.0/16      allow myhost.mydomain.ro

Global Configuration: Email Notification

Next, let's set up email notification in Monit. We need at least one active SMTP server which can send mails from the Monit host. Something like the following will do (adjust it for your case):

• Mail server hostname: smtp.monit.ro
• Sender email address used by monit (from): monit@monit.ro
• Who will receive mail from monit daemon: guletz@monit.ro
• SMTP port used by mail server: 587 (default is 25)

With the above information, email notification would be configured like this:

1 2 3 4 5 6 7 8 9 10 11 12

set mailserver  smtp.monit.ro port 587 set mail-format {  from: monit@monit.ro  subject: $SERVICE $EVENT at $DATE on $HOST  message: Monit $ACTION $SERVICE $EVENT at $DATE on $HOST : $DESCRIPTION.              Yours sincerely,           Monit      }   set alert guletz@monit.ro

As you can see, Monit offers several built-in variables ($DATE, $EVENT, $HOST, etc.), and you can customize your email message for your needs. If you want to send mails from the Monit host itself, you need a sendmail-compatible program (e.g., postfix or ssmtp) already installed.

Global Configuration: Monit Daemon

The next part is setting up monit daemon. We will set it up as follows.

• Performs the first check after 120 seconds.
• Checks services once every 3 minutes.
• Use syslog for logging.

Place the following snippet to achieve the above setting.

1 2 3

set daemon 120    with start delay 240 set logfile syslog facility log_daemon

We must also define "idfile", a unique ID used by monit demon, and "eventqueue", a path where mails sent by monit but undelivered due to SMTP/network errors. Verifiy that path (/var/monit) already exists. The following configuration will do.

1 2 3

set idfile /var/monit/id  set eventqueue      basedir /var/monit

Test Global Configuration

Now the "Global" section is finished. The Monit configuration file will look like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

#  Global Section   # status webpage and acl's set httpd port 1966 and      SSL ENABLE      PEMFILE  /var/certs/monit.pem      allow monituser:romania      allow localhost      allow 192.168.0.0/16      allow myhost.mydomain.ro   # mail-server set mailserver  smtp.monit.ro port 587 # email-format set mail-format {  from: monit@monit.ro  subject: $SERVICE $EVENT at $DATE on $HOST  message: Monit $ACTION $SERVICE $EVENT at $DATE on $HOST : $DESCRIPTION.              Yours sincerely,           Monit      }   set alert guletz@monit.ro   # delay checks set daemon 120    with start delay 240 set logfile syslog facility log_daemon   # idfile and mail queue path set idfile /var/monit/id  set eventqueue      basedir /var/monit

Now it is time to check what we have done. You can test an existing configuration file (/etc/monit.conf) by running:

# monit -t

Control file syntax OK

If Monit complains about any error, please review the configuration file again. Fortunately, error/warnings messages are informative. For example:

monit: Cannot stat the SSL server PEM file '/var/certs/monit.pem' -- No such file or directory
/etc/monit/monitrc:10: Warning: hostname did not resolve 'smtp.monit.ro'

Once you verify the syntax of configuration, start monit daemon, and wait 2 to 3 minutes:

# service monit start

If you are using systemd, run:

# systemctl start monit

Now open a browser window, and go to https://:1966. Replace & with your Monit hostname or IP address.
Note that if you have a self-signed SSL certificate, you will see a warning message in your browser.

After you have completed login, you must see the following page.

In the rest of the tutorial, let me show how we can monitor a local server and common services. You will see a lot of useful examples on the official wiki page. Most of them are copy-and-pastable!

Service Configuration: CPU/Memory Monitoring

Let start with monitoring a local server's CPU/memory usage. Copy the following snippet in the configuration file.

1 2 3 4 5 6 7

check system localhost     if loadavg (1min) > 10 then alert     if loadavg (5min) > 6 then alert     if memory usage > 75% then alert     if cpu usage (user) > 70% then alert     if cpu usage (system) > 60% then alert     if cpu usage (wait) > 75% then alert

You can easily interpret the above configuration. The above checks are performed on local host for every monitoring cycle (which is set to 120 seconds in the Global section). If any condition is met, monit daemon will send an alert with an email.
If certain properties do not need to be monitored for every cycle, you can use the following format. For example, this will monitor average load every other cycle (i.e., every 240 seconds).

1	if loadavg (1min) > 10 for 2 cycles then alert

Service Configuration: SSH Service Monitoring

Let's check if we have sshd binary installed in /usr/sbin/sshd:

1	check file sshd_bin with path /usr/sbin/sshd

We also want to check if the init script for sshd exist:

1	check file sshd_init with path /etc/init.d/sshd

Finally, we want to check if sshd daemon is up an running, and listens on port 22:

1 2 3 4 5

check process sshd with pidfile /var/run/sshd.pid    start program  "/etc/init.d/sshd start"    stop program  "/etc/init.d/sshd stop"    if failed port 22 protocol ssh then restart    if 5 restarts within 5 cycles then timeout

More specifically, we can interpret the above configuration as follows. We check if a process named sshd and a pidfile (/var/run/sshd.pid) exist. If either one does not exist, we restart sshd demon using init script. We check if a process listening on port 22 can speak SSH protocol. If not, we restart sshd daemon. If there are at least 5 restarts within the last 5 monitoring cycles (i.e., 5x120 seconds), sshd daemon is declared non-functional, and we do not try to check again.

Service Configuration: SMTP Service Monitoring

Now let's set up a check on a remote SMTP mail server (e.g., 192.168.111.102). Let's assume that the SMTP server is running SMTP, IMAP and SSH on its LAN interface.

1 2 3 4 5 6

check host MAIL with address 192.168.111.102    if failed icmp type echo within 10 cycles then alert    if failed port 25  protocol smtp then alert              else if recovered then exec "/scripts/mail-script"    if failed port 22  protocol ssh  then alert    if failed port 143 protocol imap then alert

We check if the remote host responds to ICMP. If we haven't received ICMP response within 10 cycles, we send out an alert. If testing for SMTP protocol on port 25 fails, we send out an alert. If testing succeeds again after a failed test, we run a script (/scripts/mail-script). If testing for SSH and IMAP protocols fail on port 22 and 143, respectively, we send out an alert.

Conclusion

In this tutorial, I demonstrate how to set up Monit on a local server. What I showed here is just the tip of the iceberg, as far as Monit's capabilities are concerned. Take your time and read the man page about Monit (a very good one). Monit can do a lot for any Linux admin with a very nice and easy to understand syntax. If you put together a centralized remote monitor and Monit to work for you, you will have a more reliable monitoring system. What is your thought on Monit?