The issue that we encountered was that some statistics like the InnoDB buffer pool activity were not displaying anything for one server. Other servers were displaying it just fine and other statistics for that server were also fine.

Among other things the SHOW ENGINE INNODB STATUS command shows deadlock information pertaining to the last deadlock that the InnoDB engine encountered. In some cases this information will be quite extensive and this causes a problem. The output of this command is one giant text field with a limit of 64KB. If the deadlock information is very large other information will get cut off which means certain statistics are lost. The easy fix for this is to restart the database server but in case this is not an option you can always use the innotop utility to wipe the deadlock information by causing a small deadlock.

Procedure:

Contacts: Create new contact definitions for your client.
define contact{
contact_name customer1
alias Customer One Admin
service_notification_period 24x7
host_notification_period 24x7
service_notification_options c,r
host_notification_options d,r
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email
email customer1@domain.tld
}

Groups: Create contact groups or you can add the new contact for you existing group, depending on the checks that you want to allow.
define contactgroup {
contactgroup_name Dedicated-Server1-Admins
alias Admins for Server 1
members customer1,hostingadmins
}

Hosts / Services: Use the new Contact Group with customers email and your main admin. Note that i used the existing Host Groups but you create new HostGroups if you want.
define host {
use generic-host
host_name Server1
alias Server1
address 10.0.0.2 // private or public ip
hostgroups DedicateServers
check_command check-host-alive
contact_groups Dedicated-Server1-Admins
check_period 24x7
max_check_attempts 10
notification_interval 480
notification_period 24x7
notification_options d,r
notifications_enabled 1
}
define service {
use generic-service
host_name Server1
service_description HTTP
is_volatile 0
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Dedicated-Server1-Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_http
notifications_enabled 1
}

In my case, I created a new group and add our admin contacts and customers, then update the contact groups for hosts and services. You can also create a new definitions for hosts, contacts, groups, and services with different names for the clients if you don’t want to edit your existing definitions.

Htaccess: Lastly, you need to add htaccess user to your htpasswd file (htpasswd.users). Username should match the name on your Contact. In this sample it is customer1. [Update] If you’ve implemented “Digest Authentication” you need to update your digest file instead of the htpasswd.

Don’t forget to restart you Nagios.

On the hardware side you need a DV camera with firewire port (IEEE 1394), firewire port equipped Pentium 4 or equivalent pc or laptop with loaded with Fedora 9 or 10 and a  firewire cable to connect camera to the computer. For software we will only need dvgrab and VLC

Install dvgrab from Fedora update repo:
yum install dvgrab

Install vlc from rpmfusion repo:
sudo rpm -ivh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm
sudo yum install vlc

Set DV camera audio to 16bit (default is 12 bit) to avoid garbled audio.

Turn on and connect video camera to computer and you should see something like this in /var/log/dmesg:
firewire_core: created device fw1: GUID 0800460102721e20, S100

To test that we are able to grab video/audio from camera and display in VLC player pipe output of dvgrab into vlc.

sudo dvgrab - -noavc -nostop | vlc - --no-sub-autodetect-file :demux=rawdv
After issuing this command you should see a 720×480 video feed with 16bit 48000Hz audio stream in vlc on your desktop

Now we setup vlc as a streaming server so that we can view the video/audio when away. Streaming a 720×480 video stream is a bit overkill as the video quality on the DV camera is pretty good when vlc streams video at lower resolutions like 320×240, I also reduce the audio quality to save on bandwidth. Here I used “cvlc” or command vlc to avoid opening a vlc window and “&” to put process into background.

sudo dvgrab - -noavc -nostop | cvlc - --no-sub-autodetect-file :demux=rawdv --sout '#transcode{vcodec=mp4v,vb=600,acodec=mp3,ab=56,scale=1,width=320,height=240,channels=2}:duplicate{dst=std{access=http,mux=ts,dst=192.168.1.102:3323}}' &

To view the feed locally via vlc open a http network location on ip and port you specified in the dst= section of the command above.

To view your feed from the Internet you will need to either configure vlc to stream on an outside interface or configure port forwarding.

At this time of writing, this module is still on a development version and there’s no guarantee that the installation guide will work out-of-the-box with your system.  And this post will mainly cover my own installation process on our Nagios monitoring server running on Debian and Nagios version 3.0, and Drupal version 6.x sites on web servers running CentOS 5.x.

Installation:
My installation is based on the included README file and with some adjustments to my liking.

Install the Drupal Module:

• Download the Nagios module from Drupal project page.
• Install the module to your Drupal site just like the other modules.  Download tarball, extract to modules directory ex: sites/all/modules/, go to admin->build->modules and enable the module.
• Configure your Nagios module and set the site’s UniqueID and Cron duration.

  UniqueID is your site identifier to be used by the Nagios (check_drupal) to authorize the service check and for security purposes. The author also suggests the use of MD5 or SHA1 string. Refer to README for more info on this parameter.

  Cron Duration – you need to supply the interval of your cron job that checks for Drupal updates. This value should match with your cron settings, ex: daily or every 3 hours..etc.

Configure Nagios checks:

• Copy the plugin file (check_drupal) found on the nagios-plugin directory of the module, to your Nagios plugins directory where the other Nagios check commands are located – in my case it’s on /usr/local/nagios/libexec/ (CentOS).

  If your Nagios installation is on a different machine than your Drupal server, you need to copy the check_drupal file in there.  You can also put it on the same server with Drupal sites and use NRPE instead.

  On my CentOS machine i received an error on check_drupal regarding the location of basename file – it’s on /bin/basename.  You can edit the check_drupal file directly to adjust the path to basename.
  ./check_drupal: line 14: /usr/bin/basename: No such file or directory.

• Add command, host, hostgroup, and service definition:

  Command (commands.cfg): I made small modification on the given commands from the README file to match my setup.
  define command{
command_name check_drupal
command_line $USER1$/check_drupal -H $ARG1$ -U $ARG2$ -t $ARG3$
}

  HostGroup: I created a new Host group because we have other service checks on our server such as SSH, HTTP, LOAD, etc and I want to separate my checks for Drupal sites.
  define hostgroup {
hostgroup_name Drupal
alias Drupal Sites
members MyWebServer
}

  Host: I defined new host for Drupal sites so i can configure and group my them on the same host where they belong.
  define host {
host_name MyWebServer
display_name MyWebServer
address HOSTNAME/IP ADDRESS HERE
hostgroups Drupal
check_command check-host-alive
contact_groups Admins
check_period 24x7
max_check_attempts 10
notification_interval 480
notification_period 24x7
notification_options d,r
notifications_enabled 1
}

  Service: Below is my service checks definition for checking Drupal sites, i only need to copy this and change supply parameters for domain, unique key and the timeout.
  define service {
service_description DRUPAL_SITE 1
host_name MyWebServer
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_drupal!mysite.example.com!mykeyhere!5
notifications_enabled 1
}


If your installation and configuration is correct you will get the Nagios service status similar below. It indicates number of modules, themes, users, nodes, etc.

DRUPAL OK, ADMIN:OK, CRON:OK
SAN=0;SAU=0;NOD=12;USR=7;MOD=23;THM=9

On my initial tests i received Nagios status (below) different than the above info and it was caused by my Apache configuration because i have a default Nagios installation before on my server that hosts my Drupal sites.

HTTP returned an error code. HTTP: HTTP/1.1 301 Moved Permanently
So you need to check first the url of your Nagios module installation ex: http://mysamplesite.com/nagios/, this will give you:

Nagios status page
nagios=UNKNOWN, DRUPAL:UNKNOWN=Unauthorized

While promises of having the A/C unit up and running soon were being thrown at me I wasn’t biting. Shutting down 3 non-essential servers did help things a bit.  The database servers with their 15K rpm disks were running their internal fans at nearly 100% coping with the heat. At this point I was starting to feel a little bit upbeat picturing what would happen if the servers were not servers and just consumer grade PC’s turned into servers. If you are reading this you must know what burned power supplies smell like! At some point I knew the A/C unit would be fixed and I would be able to re-power up the non-essential servers. Monitoring temperatures inside the cabinet would be nice, but we don’t have such a useful device. Next best thing was internal temperature sensors inside a direct attach storage array which has 6 sensors: 4 in the front and 2 in back. Cacti proved invaluable as I could monitor what was going on and most importantly see if the datacenter made good on their promise. Each raise in temperature indicates a period when the A/C was not working or working poorly.

temp_graph

1. Download and install latest (8.2.0-236) iLO kernel module.

rpm -ivh hp-ilo-8.2.0-236.rhel5.x86_64.rpm
service hp-ilo start

check that the module loaded properly

[root@server ~]$ lsmod
Module                  Size  Used by
hpilo                  43280  0

2. Download and install latest (1.70 B) iLO firmware.

[root@server ~]$ sh CP010302.scexe

FLASH_iLO2 v1.10 for Linux (Jan 12 2009)
Copyright 2009 Hewlett-Packard Development Company, L.P.
Firmware image: ilo2_170.bin
Current iLO 2 firmware version  1.29; Serial number ILOUSM1234567

Component XML file: CP010302.xml
CP010302.xml reports firmware version 1.70
This operation will update the firmware on the
iLO 2 in this server with version 1.70.
Continue (y/N)?y
Current firmware is 1.29 (Feb 28 2007 17:09:41)
Firmware image is 0x270000 bytes
Committing to flash part...
******** DO NOT INTERRUPT! ********
Flashing completed!
Attempting to reset device.
Succeeded.

3. Download and install latest (8.2.0-285) hp-OpenIPMI device driver.

rpm -ivh 8.2.0-285.rhel5.x86_64.rpm

4. Download and install latest (1.9.0-3) HP Lights-Out Online Configuration utility.

rpm -ivh hponcfg-1.9.0-3.noarch.rpm

5. Optional Download latest (2.10) SmartStart Scripting Toolkit to your pc.

The hponcfg utility allows configuration of iLO embedded server. All configuration is passed to iLO in XML format. Refer to SmartStart Scripting Toolkit for command reference.

In part 2 a sample configuration file will be prepared to provide access to iLO from outside even if the management port is not connected to the network.

• Look at the number of requests a page makes to the server
• Use yslow to measure page rendering (often a page performance is perceived, not just based on the server response time)

• Remove search, use alternate solutions such as Apache Solr or Google Search API
• Use CDN as much as possible
• Use Reverse Proxy Cache and memcache
• Obviously use drupal cache

Some other notes that are somewhat related to drupal performance and site performance management in a clustered hosting environment.

Manual updates and rollback

OLD WAY: tar, move/copy untar restart services

OLD WAY: rsync

BETTER WAY: Capistrano

Managing systems:

BETTER WAY: bcfg2

Monitoring Tools

• Capacity Load: analyzing trends, predicting load, checking results of configuration and software changes (cacti, munin)
• Failure: analyzing downtime, notification (nagios – using nrpe agents to monitor diverse services (do we use it this way?) , hyperin)

Use monitoring tools to closely observe cluster replication and cashing as the failures in this area are the most difficult to solve.

For my objective i only need to check if Nginx process is running and will use the check_procs.  NRPE and Nagios Plugins were installed and i can check the Nginx process locally using the following commands:

/usr/local/nagios/libexec/check_procs -c 1:30 -C nginx
wherein :
-c 1:30 <– refers to the Critical range for number of Nginx processes. If there process count is below 1 and above 30 this will send me a Critical notice.  If you wan to add a Warning level you can use “-w 1:25″ – adjust the number of processes for you needs.
-C nginx <– this will check for the command name (nginx)

NOTE: For complete reference on this check and other samples please refer to the NagiosWiki page.

Below are my configurations:

NRPE(remote): /etc/nagios/nrpe_local.cfg
command[check_nginx]=/usr/local/nagios/libexec/check_procs -c 1:30 -C nginx

Nagios(host):  /usr/local/nagios/etc/objects/localhost.cfg
define service {
use generic-service ; Name of service template to use
host_name HOST/IPADDRESS
service_description CHECK_NGINX
check_period 24x7
max_check_attempts 3
normal_check_interval 5
retry_check_interval 3
contact_groups Admins
notification_interval 480
notification_period 24x7
notification_options w,u,c,r
check_command check_nrpe!check_nginx
notifications_enabled 1
}

Nagios version is 3.0. Nagios monitoring and remote server are running Debian Etch.

It looks like they will be teaming up with  Free Software Foundation.

Hello,

This email details your current earnings from your participation in the Ad Bard Network.  We are also excited to announce major changes to our network, including general improvements and the direct participation of the Free Software Foundation.  However, our planned changes require that we temporarily suspend the entire network for the month of February.  As a member you will be receiving payment for your outstanding earnings balances, and then if you elect to participate in our newly structured network you will be required to sign up again.  We apologize for the inconvenience of this, but hope that it helps achieve the end goal of increasing the earnings of member websites and improving the desirability of the network for advertisers.

Statistics:
———–
LInux System Admin Blog average ad impressions:
Hourly:           nn
Daily:           nnn
Monthly:      nnnnn

Outstanding earnings:
$nnn

Due to the upgrade in process, please remove the JavaScript snippet from your website at this time.  No further advertisements will be displayed through this snippet, and before the end of February 2009 the handling for this javascript will be disabled and could result in an error on your website.  If you will require more than 2 weeks to remove the snippet, please send us an email and we will work with you as necessary.  A new snippet will be provided for the new website.

We will be issuing payments for all outstanding earnings through PayPal or via a check.  If your payment information has changed, please respond
to this email with updated details.  Please be sure to include your Ad Bard username in your email.

If you have converted earnings into unused coupons, please reply to this email with details so that we are sure to properly credit you
back.

Details about our enhanced network will be posted to http://adbard.net/ over the upcoming month.  You will also be receiving an update via email when it is possible to sign up for the new network.  A few of the planned changes include a limited number of advertising slots, the ability to participate in approving which FLOSS-appropriate advertisements are accepted, and improvements to our payment algorithms.
The Free Software Foundation is actively advising us in this effort, and will help campaign for the new network once it goes live.

Thank you for your patience and participation in our evolving network. We hope that you like the changes that will be happening this month, and
that you will continue to participate.

Cheers, -Jeremy

–
Jeremy Andrews
877-875-8824 x100
Tag1 Consulting, Inc.

Configuring Nagios
On the Nagios end of things we need to define a command to perform a check on a specific port of the server where the application is running. Add a line like this to the objects/commands.cfg file of your Nagios installation.

define command{
command_name check_your_application_name
command_line $USER1$/check_tcp -H $HOSTADDRESS$ -p $ARG1$ -e "This application is alive and well"
}

The -e parameter checks for a specific text that is to be returned by the application. This we can use later on to check for more detailed information. Next we need to add a service to Nagios for using this command. We do this by adding the following lines to the objects/localhost.cfg file. To keep this short I left out some lines which configure the frequency of the checks and the types of alerts.

define service {
use                    generic-service
host_name              your_server_name
service_description    your_service_name
check_command          check_your_application_name!2222
}

Creating a listener port in Java
In the second part I will show you the actual code to add to your application. Because this is a blog post I left out the package definition and the includes, but other than that the class itself is usable. To add the check to the Java app we need to add a listener thread to application. We do this by creating a class that is derived from Thread. This listener will open a port which is specified by the main application and a respond to any incoming data with a preset text. We really don’t care about the input on this end so any input will be  ignored:

public class NagiosChecker extends Thread {
    // Server socket
    private ServerSocket srv;

    // Flag that indicates whether the poller is running or not.
    private boolean isRunning = true;

    // Constructor.
    public OVMChecker(ServerSocket srv) {
        this.isRunning = true;
        this.srv = srv;
    }

    // Method for terminating the listener
    public void terminate() {
        this.isRunning = false;
    }

    /**
    * This method start the thread and performs all the operations.
    */
    public void run() {
        try {
            // Wait for connection from client.
            while (isRunning) {
                Socket socket = srv.accept();

            // Open a reader to receive (and ignore the input)
            BufferedReader rd = new BufferedReader(new InputStreamReader(socket.getInputStream()));

            // Write the status message to the outputstream
            try {
                BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
                wr.write("This application is alive and well");
                wr.flush();
            } catch (IOException e) {
                System.out.println(e.getMessage()));
            }

            // Close the inputstream since we really don't care about it
            rd.close();
        } catch (Exception e) {
            System.out.println(e.getMessage()));
        }
    }
}

In case you’re still reading this you’re probably interested in how to call this class. The following code should be executed in the initialization of the application. It creates the actual socket for port 2222 and starts the listener class. After this the listener class will run indefinitely until the application terminates.

ServerSocket srv = null;
try {
    srv = new ServerSocket(2222);
    NagiosChecker checker = new NagiosChecker(srv);
    checker.start();
} catch (Exception e) {
    System.out.println(e.getMessage());
}