System Integrity Monitor

[phpforfun]

For a while, I have had issues with HTTPD, it would lock up on me, no matter what I do, change the prefork settings, limit the CPU/MEM usage to httpd process, it doesnt matter, it always seems to lock up, but not die, and I always have to reboot it.

I just installed a new program, and it has a setting that I hope takes care of it.

System Integrity Monitor

Its a rather easy install, here is more specifics of the program from the website:

SIM is a system and services monitor for ‘SysVinit’ systems. It is designed to be intuitive and modular in nature, and to provide a clean and informative status system.

It does this by consistently verifying that services are online, load averages are in check, and log files are at reasonable sizes. Many other SIM modules sport different and in-depth features to bring a well rounded tool to your disposal to stop otherwise common issues daunting internet hosts. Features:
- Service monitoring of HTTP, FTP, DNS, SSH, MYSQL & more
- Event tracking and alert system
- Auto restart ability for downed services
- Checks against network sockets & process list to ensure services are online
- Advanced HTTP service monitoring, to prevent commonly encountered issues
- System load monitor with customizable warnings & actions
- Ability to auto restart system with definable critical load level
- Priority change configurable for services, at warning or critical load level
- Informative command line status display
- Easily customizable configuration file
- Auto configuration script
- Auto cronjob setup feature
- Simple & Informative installation script
- Integrated auto-update feature
- And more...

R-fx Networks - Internet Security Solutions - Projects » SIM

How to download/Install:
1) Download:
wget http://www.rfxnetworks.com/downloads/sim-current.tar.gz
2) Untar
tar xvfz sim-current.tar.gz
3) CD Into it
cd sim*
4) Run Setup
./setup -i
5) Run the easy auto conf
/usr/local/sim/autoconf


Here is the setup I used, I will highlight the change I made in bold. I think everyone should use these settings

root@svr1 [/]# /usr/local/sim/autoconf

SIM 2.5-4 Auto-Config Script

All questions default to value in brackets if no answer is given. If you
make a typo during the autoconf process, hit CTRL+C (^C) to abort and
rerun the autoconf script (/usr/local/sim/autoconf).

The below are general configuration options for SIM:
press return to continue...

Where is SIM installed ?
[/usr/local/sim]:

Where should the sim.log file be created ?
[/usr/local/sim/sim.log]:

Max size of sim.log before rotated ? (value in KB)
[128]:

What is the location of your kernel log ?
Found kernel log at /var/log/messages

Where should alerts be emailed to ? (e.g: root, user@domain)
[root]:

Disable alert emails after how many events, to avoid email flood ?
(Note: events stats are cleared daily)
[8]:

The below are configuration options for Service modules:
press return to continue...

Auto-restart services found to be offline ? (true=enable, false=disable)
[true]:

Enforce laxed service checking ? (true=enable, false=disable)
[true]:

Disable auto-restart after how many downed service events ?
(Note: events stats are cleared daily)
[10]:

Enable FTP service monitoring ? (true=enable, false=disable)
[false]:

Enable HTTP service monitoring ? (true=enable, false=disable)
[true]:

Name of the HTTP service as appears in 'ps' ?
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ
Found service name as httpd

TCP/IP port that HTTP operates on ?
Found service port as 80

Path to HTTP service init script ?
Found service init script at /etc/init.d/httpd

Enable DNS service monitoring ? (true=enable, false=disable)
[true]:

Name of the DNS service as appears in 'ps' ?
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ
Found service name as named

TCP/IP port that DNS operates on ?
Found service port as 53

Path to DNS service init script ?
Found service init script at /etc/init.d/named

Enable SSH service monitoring ? (true=enable, false=disable)
[false]:

Enable MYSQL service monitoring ? (true=enable, false=disable)
[true]:

Name of the MYSQL service as appears in 'ps' ?
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ
Found service name as mysqld

TCP/IP port that MYSQL operates on ?
Found service port as 3306

Path to MYSQL service init script ?
Found service init script at /etc/init.d/mysql

Enable SMTP service monitoring ? (true=enable, false=disable)
[false]:

Enable XINET service monitoring ? (true=enable, false=disable)
[false]:

Enable ENSIM service monitoring ? (true=enable, false=disable)
[false]:

Enable PGSQL service monitoring ? (true=enable, false=disable)
[false]:

The below are configuration options for Service Specific features:
press return to continue...

After an unclean HTTP shutdown, semaphore array's may remain allocated
and cause the service to fall into a looping restart cycle. Using this
feature clears semaphore arrays on HTTP restart.
Enable semaphore cleanup ?
[false]:

This is an implamented feature in the http module, its purpose is to
determine if/when the apache server locks up or otherwise stops
responding.
Enable URL aware monitoring ?
[false]:true

URL path to a local file ? (exclude HTTP://)
Note: This URL should be valid and reside on the local server, otherwise
HTTP will loop restarting
[127.0.0.1/index.html]:localhost/index.php

HTTP log files can grow large and cause the service to crash
(segfault), this feature will keep the main HTTP logs incheck.
Enable HTTP log monitor ?
[false]:false

MySQL uses a /tmp symlink of its mysql.sock socket file. This
feature verifies that the symlink exists from the main mysql.sock
file, and if not it is recreated.
Enable MySQL Socket correction ?
[false]:

The below are configuration options for System modules:
press return to continue...

Enable NETWORK monitoring ? (true=enable, false=disable)
[false]:

Enable LOAD monitor ? (true=enable, false=disable)
[true]:

Load level before status condition 'warning' ?
[25]:

Load level before status condition 'critical' ?
[45]:

Enable a global (wall) message at status condition 'warning' & 'critical' ?
[false]:

Renice services at status condition 'warning' or 'critical' ?
(3 values - warn, crit, false - false=disabled)
[warn]:

Stop nonessential services at status condition 'warning' or 'critical' ?
(3 values - warn, crit, false - false=disabled)
[crit]:

Reboot system on status condition 'warning' or 'critical' ?
(3 values - warn, crit, false - false=disabled)
[false]:

Configuration completed, saving conf.sim...
Done, conf.sim saved to /usr/local/sim.
root@svr1 [/]#

I will update this to see if its fixed my issue!

[phpforfun]

Already seems to be working, here is the log

[01/11/09 12:05:01]: MYSQL service is online.
[01/11/09 12:10:02]: LOAD 0.16 (status good)
[01/11/09 12:10:02]: HTTP service is online.
[01/11/09 12:10:02]: HTTP url request failed, assuming offline.
[01/11/09 12:10:02]: HTTP service is online.
[01/11/09 12:10:02]: HTTP url request failed, assuming offline.
[01/11/09 12:10:02]: Restarted HTTP service (2 HTTP events today).
[01/11/09 12:10:02]: Restarted HTTP service (2 HTTP events today).
[01/11/09 12:10:02]: DNS service is online.
[01/11/09 12:10:02]: MYSQL service is online.
[01/11/09 12:15:01]: LOAD 0.17 (status good)
[01/11/09 12:15:01]: HTTP service is online.
[01/11/09 12:15:01]: HTTP url request failed, assuming offline.
[01/11/09 12:15:01]: HTTP service is online.
[01/11/09 12:15:01]: HTTP url request failed, assuming offline.
[01/11/09 12:15:01]: Restarted HTTP service (3 HTTP events today).
[01/11/09 12:15:01]: Restarted HTTP service (3 HTTP events today).
[01/11/09 12:15:01]: DNS service is online.
[01/11/09 12:15:01]: MYSQL service is online.

NOTE: It was assuming HTTP was offline because the request failed, because it should have been localhost/index.html, not .php. my fault