From AdminWiki

Revision as of 15:08, 1 June 2006

HotSaNIC monitoring

Overview

HotSaNIC is system that reads data through a perl daemon and uses rrd to store and create graphs for the system. It is very similar to munin, but it lacks of a client-server system. As munin it can have plugins, but they are more difficult to write.

The advantage from HotSaNIC is the very high resolution, it reads the data every few seconds and gets therefore very accurate results. This is good for monitoring network traffic, processes and similar quick changing events.

The disadvantage is, that it creates the graphs on the server itself and cannot offload this task to a remote system. It therefore has to use some CPU time for it. This can be limited to create the graphs only every few hours, or depending on how often you want to check your graphs. In my opinion it would be better to have a cgi script like cricket has to create the graphs only when needed, when the page is viewed.

The current CVS version is recommended as it has all patches to work with current kernels. Hotsanic daemon runs as root and therefore has no problems with grs or other secure kernels.

Additional Modules / Changed Modules

NTP Patch to work with current ntp daemon

The ntp module from HotSaNIC works, but misses some data because of some output change from the ntpdaemon. I wrote a small patch to fix this.

Patches below

diagrams.pl

--- /usr/local/src/sys/HotSaNIC/modules/ntp/diagrams.pl 2004-06-03 08:25:10.000000000 +0900
+++ /usr/local/HotSaNIC/modules/ntp/diagrams.pl 2006-03-20 10:13:06.447836886 +0900
@@ -33,8 +33,8 @@

 # handle module-specific stuff
 #
-my  @GRAPHS=("stratum","distance","dispersion","frequency","stability");
-my  %LEGENDS=("stratum"=>" ","distance"=>"seconds","dispersion"=>"seconds","frequency"=>"ppm","stability"=>"ppm");
+my  @GRAPHS=("stratum","distance","dispersion","jitter","stability","broadcastdelay","authdelay");
+my  %LEGENDS=("stratum"=>" ","distance"=>"seconds","dispersion"=>"seconds","jitter"=>"seconds","stability"=>"ppm","broadcastdelay"=>"seconds","authdelay"=>"seconds");
 push @OPTIONS,(
   "--alt-autoscale-max");   # alternate scaling

makeindex.pl

--- /usr/local/src/sys/HotSaNIC/modules/ntp/makeindex.pl        2004-06-03 08:25:10.000000000 +0900
+++ /usr/local/HotSaNIC/modules/ntp/makeindex.pl        2006-03-20 10:09:42.065613912 +0900
@@ -29,7 +29,7 @@
 %MODCONFIG=HotSaNICmod::common::configure();
 @DIAGRAMS=("hour","6h","day","week","month","year");
 @TIMES=("6h","week");
-@GRAPHS=("stratum","distance","dispersion","frequency","stability");
+@GRAPHS=("stratum","distance","dispersion","jitter","stability","broadcastdelay","authdelay");

 # build time-based .html files
 #

makerrd

--- /usr/local/src/sys/HotSaNIC/modules/ntp/makerrd     2004-05-18 03:31:58.000000000 +0900
+++ /usr/local/HotSaNIC/modules/ntp/makerrd     2006-03-20 10:06:04.466164389 +0900
@@ -16,9 +16,11 @@
   $BINPATH/rrdtool create rrd/$PROC.rrd --step 10 \
     DS:dispersion:GAUGE:300:0:$MAX \
     DS:distance:GAUGE:300:0:$MAX \
-    DS:frequency:GAUGE:300:-10000:10000 \
+    DS:jitter:GAUGE:300:0:$MAX \
     DS:stability:GAUGE:300:0:$MAX \
     DS:stratum:GAUGE:300:0:20 \
+    DS:broadcastdelay:GAUGE:300:0:$MAX \
+    DS:authdelay:GAUGE:300:0:$MAX \
     RRA:AVERAGE:0:1:720 \
     RRA:AVERAGE:0.3:6:2880 \
     RRA:AVERAGE:0.3:60:2016 \

in the subdirectory platform common.pm

--- /usr/local/src/sys/HotSaNIC/modules/ntp/platform/common.pm  2004-06-03 08:25:10.000000000 +0900
+++ /usr/local/HotSaNIC/modules/ntp/platform/common.pm  2006-03-20 09:59:15.837619337 +0900
@@ -13,6 +13,7 @@
     my @list=HotSaNICparser::locate_files("bin/ntpdc");
     if (! @list) { @list=HotSaNICparser::locate_files("bin/xntpdc"); }
     $MODCONF{NTPCOMMAND}=pop @list;
+    chomp $MODCONF{NTPCOMMAND};
     }

   return %MODCONF;

default.pm

--- /usr/local/src/sys/HotSaNIC/modules/ntp/platform/default.pm 2004-07-06 05:40:31.000000000 +0900
+++ /usr/local/HotSaNIC/modules/ntp/platform/default.pm 2006-03-20 10:07:04.107497582 +0900
@@ -24,12 +24,14 @@
       $str=$value if $var eq "stratum";
       $dst=$value if $var eq "root distance";
       $dps=$value if $var eq "root dispersion";
-      $frq=$value if $var eq "frequency";
+      $jtr=$value if $var eq "jitter"; # changed that from frequency
       $stb=$value if $var eq "stability";
+      $bcdly=$value if $var eq "broadcastdelay";
+      $athdly=$value if $var eq "authdelay";
       }
     close FILE;
-
-    HotSaNICmod::do_rrd($dbname,"U",time,$dps,$dst,$frq,$stb,$str);
+    HotSaNICmod::do_rrd($dbname,"U",time,$dps,$dst,$jtr,$stb,$str,$bcdly,$athdly);
     }
   }

after patching the files and you have already a rrd file there you need to remove it (yes, you loose all data). You also need to run the makeindex.pl file again to regenerate the html files

Add more detailed error legend in Postfix queue log

The default postfix queue graph has already a nice list of errors, but there are some which can be also shown to be more detailed. Especially 4x errors.

Below some patches to add this to the vaniall mailq without the multiple postfix patch. The multiple postfix patch has this one already included.

diagrams.pl

--- /usr/local/src/HotSaNIC/modules/mailq/diagrams.pl   2004-05-24 08:53:39.000000000 +0900
+++ /usr/local/HotSaNIC/modules/mailq/diagrams.pl       2006-03-23 12:53:27.843497000 +0900
@@ -113,8 +113,12 @@
       "DEF:ctimeout=$DB_FILE:ctimeout:AVERAGE",
       "DEF:rtimeout=$DB_FILE:rtimeout:AVERAGE",
       "DEF:nohost=$DB_FILE:nohost:AVERAGE",
+      "DEF:noroute=$DB_FILE:noroute:AVERAGE",
+      "DEF:err450=$DB_FILE:err450:AVERAGE",
+      "DEF:err421=$DB_FILE:err421:AVERAGE",
+      "DEF:err4=$DB_FILE:err4:AVERAGE",
       "DEF:other=$DB_FILE:other:AVERAGE",
-      "CDEF:active=req,crefused,-,msrefused,-,ctimeout,-,rtimeout,-,nohost,-,other,-,");
+      "CDEF:active=req,crefused,-,msrefused,-,ctimeout,-,rtimeout,-,nohost,-,noroute,-,err450,-,err421,-,err4,-,other,-,");

     if ($range ne "1h") {
       push @COMMANDS,("DEF:maxreq=$DB_FILE:req:MAX",
@@ -128,8 +132,12 @@
       HotSaNICdiagram::insert_data("STACK","crefused" ,$MODULECONFIG{COLOR_AREA_CREFUSED} ,"connection refused  ",$legends,$LEGEND,1),
       HotSaNICdiagram::insert_data("STACK","other"    ,$MODULECONFIG{COLOR_AREA_OTHER}    ,"other               ",$legends,$LEGEND,1),
       HotSaNICdiagram::insert_data("STACK","nohost"   ,$MODULECONFIG{COLOR_AREA_NOHOST}   ,"host not found      ",$legends,$LEGEND,1),
+      HotSaNICdiagram::insert_data("STACK","noroute"  ,$MODULECONFIG{COLOR_AREA_NOROUTE}  ,"route not found     ",$legends,$LEGEND,1),
       HotSaNICdiagram::insert_data("STACK","ctimeout" ,$MODULECONFIG{COLOR_AREA_CTIMEOUT} ,"connection timed out",$legends,$LEGEND,1),
       HotSaNICdiagram::insert_data("STACK","rtimeout" ,$MODULECONFIG{COLOR_AREA_RTIMEOUT} ,"read timed out      ",$legends,$LEGEND,1),
+      HotSaNICdiagram::insert_data("STACK","err450"   ,$MODULECONFIG{COLOR_AREA_ERR450}   ,"450 mbox not okay   ",$legends,$LEGEND,1),
+      HotSaNICdiagram::insert_data("STACK","err421"   ,$MODULECONFIG{COLOR_AREA_ERR421}   ,"421 service not okay",$legends,$LEGEND,1),
+      HotSaNICdiagram::insert_data("STACK","err4"     ,$MODULECONFIG{COLOR_AREA_ERR4}     ,"general 4xx error   ",$legends,$LEGEND,1),
       HotSaNICdiagram::insert_data("STACK","active"   ,$MODULECONFIG{COLOR_AREA}          ,"active              ",$legends,$LEGEND,1),
       "LINE1:req#".$MODULECONFIG{COLOR_LINE}.":",
       HotSaNICdiagram::insert_lines(%MODULECONFIG));

makerrd

--- /usr/local/src/HotSaNIC/modules/mailq/makerrd       2004-04-19 01:35:24.000000000 +0900
+++ /usr/local/HotSaNIC/modules/mailq/makerrd   2006-03-23 12:13:07.296222250 +0900
@@ -18,6 +18,10 @@
     DS:nohost:GAUGE:300:0:U \
     DS:other:GAUGE:300:0:U \
     DS:msrefused:GAUGE:300:0:U \
+    DS:noroute:GAUGE:300:0:U \
+    DS:err450:GAUGE:300:0:U \
+    DS:err421:GAUGE:300:0:U \
+    DS:err4:GAUGE:300:0:U \
     RRA:AVERAGE:0:1:720 \
     RRA:AVERAGE:0.3:6:2880 \
     RRA:AVERAGE:0.3:60:2016 \

in the platform folder default.pm

--- /usr/local/src/HotSaNIC/modules/mailq/platform/default.pm   2004-07-01 19:28:50.000000000 +0900
+++ /usr/local/HotSaNIC/modules/mailq/platform/default.pm       2006-03-23 13:39:45.393083000 +0900
@@ -19,6 +19,10 @@
   $other=0;
   $msrefused=0;
   $active=0;
+  $noroute=0;
+  $err450=0;
+  $err421=0;
+  $err4=0;

   open FILE,"mailq|";
   while (<FILE>) {
@@ -29,15 +33,19 @@
       elsif (index($_,"Host not found") >=0 ) { $nohost++; }
       elsif (index($_,"read timeout") >=0 ) { $rtimeout++; }
       elsif (index($_,"server refused mail service") >=0 ) { $msrefused++; }
+      elsif (index($_,"No route to host") >=0 ) { $noroute++; }
+      elsif (index($_,"said: 450") >=0 ) { $err450++; }
+      elsif (index($_,"said: 421") >=0 ) { $err421++; }
+      elsif (index($_,"said: 4") >=0 ) { $err4++; } # all other 4xx errors are collected here
       else { $other++; }
       }
     elsif (/^--/o) { (undef,$kbytes)=split; }
     }
   close FILE;

-  my $req=$crefused+$ctimeout+$rtimeout+$nohost+$other+$msrefused+$active;
+  my $req=$crefused+$ctimeout+$rtimeout+$nohost+$other+$msrefused+$active+$noroute+$err450+$err421+$err4;

-  HotSaNICmod::do_rrd("queue","U",time,$kbytes,$req,$crefused,$ctimeout,$rtimeout,$nohost,$other,$msrefused);
+  HotSaNICmod::do_rrd("queue","U",time,$kbytes,$req,$crefused,$ctimeout,$rtimeout,$nohost,$other,$msrefused,$noroute,$err450,$err421,$err4);

   }

if you have already an rrd, you need to remove it, as this patche changes the rrd file layout. Old data will be lost.

Postfix Module patch to have it check 'n' postfix servers on one box

I have a special setup where I have ten postfix servers running on one box. I wanted to monitor them with the postfix module from HotSaNIC to get the current processes count and mails in the queue.

This is quite a majour change, it works fine for me. Remove the rrd and config files because the configuration has to be created new. Also the rrd file layout changes, so old rrd files and their data will be lost.

Patches for multiple postfixes module