Procedure:

Contacts: Create new contact definitions for your client.
define contact{
contact_name customer1
alias Customer One Admin
service_notification_period 24x7
host_notification_period 24x7
service_notification_options c,r
host_notification_options d,r
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email
email customer1@domain.tld
}

Groups: Create contact groups or you can add the new contact for you existing group, depending on the checks that you want to allow.
define contactgroup {
contactgroup_name Dedicated-Server1-Admins
alias Admins for Server 1
members customer1,hostingadmins
}

Hosts / Services: Use the new Contact Group with customers email and your main admin. Note that i used the existing Host Groups but you create new HostGroups if you want.
define host {
use generic-host
host_name Server1
alias Server1
address 10.0.0.2 // private or public ip
hostgroups DedicateServers
check_command check-host-alive
contact_groups Dedicated-Server1-Admins
check_period 24x7
max_check_attempts 10
notification_interval 480
notification_period 24x7
notification_options d,r
notifications_enabled 1
}
define service {
use generic-service
host_name Server1
service_description HTTP
is_volatile 0
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Dedicated-Server1-Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_http
notifications_enabled 1
}

In my case, I created a new group and add our admin contacts and customers, then update the contact groups for hosts and services. You can also create a new definitions for hosts, contacts, groups, and services with different names for the clients if you don’t want to edit your existing definitions.

Htaccess: Lastly, you need to add htaccess user to your htpasswd file (htpasswd.users). Username should match the name on your Contact. In this sample it is customer1. [Update] If you’ve implemented “Digest Authentication” you need to update your digest file instead of the htpasswd.

Don’t forget to restart you Nagios.

At this time of writing, this module is still on a development version and there’s no guarantee that the installation guide will work out-of-the-box with your system.  And this post will mainly cover my own installation process on our Nagios monitoring server running on Debian and Nagios version 3.0, and Drupal version 6.x sites on web servers running CentOS 5.x.

Installation:
My installation is based on the included README file and with some adjustments to my liking.

Install the Drupal Module:

• Download the Nagios module from Drupal project page.
• Install the module to your Drupal site just like the other modules.  Download tarball, extract to modules directory ex: sites/all/modules/, go to admin->build->modules and enable the module.
• Configure your Nagios module and set the site’s UniqueID and Cron duration.

  UniqueID is your site identifier to be used by the Nagios (check_drupal) to authorize the service check and for security purposes. The author also suggests the use of MD5 or SHA1 string. Refer to README for more info on this parameter.

  Cron Duration – you need to supply the interval of your cron job that checks for Drupal updates. This value should match with your cron settings, ex: daily or every 3 hours..etc.

Configure Nagios checks:

• Copy the plugin file (check_drupal) found on the nagios-plugin directory of the module, to your Nagios plugins directory where the other Nagios check commands are located – in my case it’s on /usr/local/nagios/libexec/ (CentOS).

  If your Nagios installation is on a different machine than your Drupal server, you need to copy the check_drupal file in there.  You can also put it on the same server with Drupal sites and use NRPE instead.

  On my CentOS machine i received an error on check_drupal regarding the location of basename file – it’s on /bin/basename.  You can edit the check_drupal file directly to adjust the path to basename.
  ./check_drupal: line 14: /usr/bin/basename: No such file or directory.

• Add command, host, hostgroup, and service definition:

  Command (commands.cfg): I made small modification on the given commands from the README file to match my setup.
  define command{
command_name check_drupal
command_line $USER1$/check_drupal -H $ARG1$ -U $ARG2$ -t $ARG3$
}

  HostGroup: I created a new Host group because we have other service checks on our server such as SSH, HTTP, LOAD, etc and I want to separate my checks for Drupal sites.
  define hostgroup {
hostgroup_name Drupal
alias Drupal Sites
members MyWebServer
}

  Host: I defined new host for Drupal sites so i can configure and group my them on the same host where they belong.
  define host {
host_name MyWebServer
display_name MyWebServer
address HOSTNAME/IP ADDRESS HERE
hostgroups Drupal
check_command check-host-alive
contact_groups Admins
check_period 24x7
max_check_attempts 10
notification_interval 480
notification_period 24x7
notification_options d,r
notifications_enabled 1
}

  Service: Below is my service checks definition for checking Drupal sites, i only need to copy this and change supply parameters for domain, unique key and the timeout.
  define service {
service_description DRUPAL_SITE 1
host_name MyWebServer
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_drupal!mysite.example.com!mykeyhere!5
notifications_enabled 1
}


If your installation and configuration is correct you will get the Nagios service status similar below. It indicates number of modules, themes, users, nodes, etc.

DRUPAL OK, ADMIN:OK, CRON:OK
SAN=0;SAU=0;NOD=12;USR=7;MOD=23;THM=9

On my initial tests i received Nagios status (below) different than the above info and it was caused by my Apache configuration because i have a default Nagios installation before on my server that hosts my Drupal sites.

HTTP returned an error code. HTTP: HTTP/1.1 301 Moved Permanently
So you need to check first the url of your Nagios module installation ex: http://mysamplesite.com/nagios/, this will give you:

Nagios status page
nagios=UNKNOWN, DRUPAL:UNKNOWN=Unauthorized

For my objective i only need to check if Nginx process is running and will use the check_procs.  NRPE and Nagios Plugins were installed and i can check the Nginx process locally using the following commands:

/usr/local/nagios/libexec/check_procs -c 1:30 -C nginx
wherein :
-c 1:30 <– refers to the Critical range for number of Nginx processes. If there process count is below 1 and above 30 this will send me a Critical notice.  If you wan to add a Warning level you can use “-w 1:25″ – adjust the number of processes for you needs.
-C nginx <– this will check for the command name (nginx)

NOTE: For complete reference on this check and other samples please refer to the NagiosWiki page.

Below are my configurations:

NRPE(remote): /etc/nagios/nrpe_local.cfg
command[check_nginx]=/usr/local/nagios/libexec/check_procs -c 1:30 -C nginx

Nagios(host):  /usr/local/nagios/etc/objects/localhost.cfg
define service {
use generic-service ; Name of service template to use
host_name HOST/IPADDRESS
service_description CHECK_NGINX
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_nrpe!check_nginx
notifications_enabled 1
}

Nagios version is 3.0. Nagios monitoring and remote server are running Debian Etch.

Configuring Nagios
On the Nagios end of things we need to define a command to perform a check on a specific port of the server where the application is running. Add a line like this to the objects/commands.cfg file of your Nagios installation.

define command{
command_name check_your_application_name
command_line $USER1$/check_tcp -H $HOSTADDRESS$ -p $ARG1$ -e "This application is alive and well"
}

The -e parameter checks for a specific text that is to be returned by the application. This we can use later on to check for more detailed information. Next we need to add a service to Nagios for using this command. We do this by adding the following lines to the objects/localhost.cfg file. To keep this short I left out some lines which configure the frequency of the checks and the types of alerts.

define service {
use                    generic-service
host_name              your_server_name
service_description    your_service_name
check_command          check_your_application_name!2222
}

Creating a listener port in Java
In the second part I will show you the actual code to add to your application. Because this is a blog post I left out the package definition and the includes, but other than that the class itself is usable. To add the check to the Java app we need to add a listener thread to application. We do this by creating a class that is derived from Thread. This listener will open a port which is specified by the main application and a respond to any incoming data with a preset text. We really don’t care about the input on this end so any input will be  ignored:

public class NagiosChecker extends Thread {
    // Server socket
    private ServerSocket srv;

    // Flag that indicates whether the poller is running or not.
    private boolean isRunning = true;

    // Constructor.
    public OVMChecker(ServerSocket srv) {
        this.isRunning = true;
        this.srv = srv;
    }

    // Method for terminating the listener
    public void terminate() {
        this.isRunning = false;
    }

    /**
    * This method start the thread and performs all the operations.
    */
    public void run() {
        try {
            // Wait for connection from client.
            while (isRunning) {
                Socket socket = srv.accept();

            // Open a reader to receive (and ignore the input)
            BufferedReader rd = new BufferedReader(new InputStreamReader(socket.getInputStream()));

            // Write the status message to the outputstream
            try {
                BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
                wr.write("This application is alive and well");
                wr.flush();
            } catch (IOException e) {
                System.out.println(e.getMessage()));
            }

            // Close the inputstream since we really don't care about it
            rd.close();
        } catch (Exception e) {
            System.out.println(e.getMessage()));
        }
    }
}

In case you’re still reading this you’re probably interested in how to call this class. The following code should be executed in the initialization of the application. It creates the actual socket for port 2222 and starts the listener class. After this the listener class will run indefinitely until the application terminates.

ServerSocket srv = null;
try {
    srv = new ServerSocket(2222);
    NagiosChecker checker = new NagiosChecker(srv);
    checker.start();
} catch (Exception e) {
    System.out.println(e.getMessage());
}

So we let crontab run a shell script every minute which just executes one command:
mysql -e "show processlist;" > job.tmp

This command will let you track all user connections at that moment. Of course a lot can happen in a minute and there’s lots of stuff you won’t catch, but problems that are growing will manifest themselves here. To distill the number of running connections:
CONNS=`cut ${SCRIPTS_DIR}/jobs.tmp -f5 | grep "Query" | wc -l | cut -f1 -d"/"`

The number of locked out queries:
LOCKED=`cut ${SCRIPTS_DIR}/jobs.tmp -f7 | grep "Locked" | wc -l | cut -f1 -d"/"`

The longest running query:
LONGRUN=`grep "Query" ${SCRIPTS_DIR}/jobs.tmp | cut -f6 | sort -n | tail -1`

In all these cases a simple if statement will mail the processlist in case a threshold is passed. No better way to get your attention then your mailbox filling up with an alert per minute. Besides, seeing the processlist at the time things started going wrong can be very useful in identifying the culprit.

Yesterday we’ve setup another HP server and installed NRPE, check_hparray, and hpacucli, same process used on previous installations. NRPE worked fine locally and from the Nagios server as the local disk space check was configured properly, but when we tried check_hparray we got “check_hparray Error”. This error can have different causes like invalid slot value used or problems with permissions on executing hpacucli command.

We reviewed our setup and installations and we have the same settings (based on setup with other servers).

We run check_hparray from NRPE we got the error:

[root@web161 nagios]# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_raid
check_hparray Error.

and it worked fine if run check_hparray command directly:

[root@web161 nagios]# /usr/local/nagios/libexec/check_hparray -s 1
RAID OK – (Smart Array P400 in Slot 1 array A logicaldrive 1 (546.8 GB, RAID 1+0, OK))

Both of the commands above were tested using root and nagios users and they have the same results. Then we enabled NRPE DEBUG option to get details on the problem:

edit: /usr/local/nagios/etc/nrpe.cfg

# DEBUGGING OPTION
# This option determines whether or not debugging messages are logged to the
# syslog facility.
# Values: 0=debugging off, 1=debugging on

debug=1

and by looking on the system logs we saw the problem with our sudo:

Sep 10 04:11:35 hostname sudo: root : TTY=pts/2 ; PWD=/var/log ; USER=root ; COMMAND=/usr/sbin/hpacucli controller slot=1 ld all show
Sep 10 04:12:55 hostname sudo: nagios : sorry, you must have a tty to run sudo ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/hpacucli controller slot=1 ld all show”

and the solution was to comment out in /etc/sudoers file the line:

Defaults requiretty