Self-Monitoring Script with Python

Just last night, I faced a curious problem with one of my servers. It seemed to have the network interface up and running but I could not reach to the server with ssh, when I wanted to connect, the putty hanged on the username prompt , the server first closed port 80 and then the mysqld port and finally it closed ssh port that left me with no choice but to restart the server. We are not intending to address an optimization scenario, though the closure of httpd and mysqld port is related to an optimization issue which has haunted me several times.in this scenario, I am going to write and implement an internal monitoring and self-healing mechanism on my server with the help of python and crontab.

Which port is closed?

Netstat is the utility which comes in handy when we are to know about open/closed ports and we may pipe the output of the netstat to grep in order to filter the output, let’s see it in action

# netstat -ntulp|grep ':::80'

The command will look for port 80 and if the port is open the grep will filter and shows is in stdout so we will have a 1 line output, but we do not care about the what the output is, since the existence of output means the port is open, we just want to see if there is an output or not, right? Look at the following command:

# netstat -ntulp|grep ':::80'|wc –l

wc(Word Count) command counts the lines of output and if there is an output, it returns one (number of lines) and if it returns 0, it means the port is closed.so the question has been answered , we now know how to figure out which port is closed. let’s move on, shall we?

Mail the incident to yourself or your Manager

It is imperative to be informed about an incident, Specially when you are not at the Datacenter or you are at a remote location. The incident may be mailed to the managers to keep all of them informed. One mailing utility that I have used for a long time on my server is mail utility which is the most popular command to send emails from Linux terminal.one simple example for this command is:

# mail -s "Test Subject" user@example.com < /dev/null

The Above command uses option –s to define Subject of the email as “Test Subject” and sets user@example.com as the recipient and < /dev/null defines the body of the e-mail because you just want to send an empty e-mail with subject “Test Subject”.

# mail -s 'HTTP Port is Closed' mr.hdavoodi@gmail.com < report.txt

The Above  command sends an email to me, the body will be the contents of the file report.txt now let’s jump into python and write an extremely helpful script to report closing of an specific port.

PYTHON My Saviour

As you may know, python is very convenient and in recent distros , it is installed by default, it has an important role in Linux admins career, I personally believe that it is the best scripting language for Linux administrators, it is easy to learn , fast , flexible and has numerous libraries moreover, if you mix it with shell script, you will be invincible in your Linux career.

#!/usr/bin/python

import os

import subprocess

The shebang line in any script determines the script's ability to be executed like an standalone executable without typing python beforehand in the terminal or when double clicking it in a file manager (when configured properly). It isn't necessary but generally put there so when someone sees the file opened in an editor, they immediately know what they're looking at.the next two lines just add os and subprocess libraries which enable us to work with os commands and communicate with processes.

http = ssh = mysql = 0

http = subprocess.check_output("netstat -ntulp|grep ':::80'|wc -l", shell=True).strip()

ssh = subprocess.check_output("netstat -ntulp|grep ':::10242'|wc -l", shell=True).strip()

mysql = subprocess.check_output("netstat -ntulp|grep ':::3306'|wc -l", shell=True).strip()

we have defined 3 variables to keep the output for each Service we want to monitor in the first line , the next 3 lines use subprocess class to to check the output of netstat commands and the strip() method trims the spaces and unwanted characters around each value. if all the services are running then it means that their ports are open.

if int(ssh) != 1:

os.system("echo 'sshd port was closed @'" + "`date` >> /root/internalMonitoring/report.txt") 

os.system("mail -s 'SSH Port is Closed' mr.hdavoodi@gmail.com < /root/internalMonitoring/report.txt")

os.system("systemctl restart sshd.service")

if int(http) != 1:

os.system("echo 'httpd port was closed @'" + "`date` >> /root/internalMonitoring/report.txt")

os.system("mail -s 'HTTP Port is Closed' mr.hdavoodi@gmail.com < root/internalMonitoring/report.txt")

os.system("systemctl restart httpd.service")

if int(mysql) != 1:

os.system("echo 'mysqld port was closed @'" + "`date` >> /root/internalMonitoring/report.txt")

os.system("mail -s 'MYSQL Port is Closed' mr.hdavoodi@gmail.com < root/internalMonitoring/report.txt")

os.system("systemctl restart mysqld.service")

the Above code checks the value for each service and if the value is not 1, then it means that the port is closed and the script will first add a line to the file report.txt and then reports to me by email and in the next step, it tries to restart the service to make it up and running . now you may ask how does the script know when to start checking the ports?! Well, with the help of crontab which is powerful scheduling utility we run the script every one second so it checks internally the state of services and equivalent ports.

On the command line just type:
# crontab –e

and add a line like this:

* * * * * /root/internalMonitoring/interMonit.py

the stars mean every and in this case (every minute of every hour of every day of every month and the last start is day of week) this simply means always , do it always ! and in order to make it running you just have to save the file and exit crontab .one last thing ! you should make your python file executable by typing:

# chmod +x /root/internalMonitoring/interMonit.py

i have tested the script on my production server and you know what! It works astonishingly and from now on, I can sleep like a baby all night long, knowing that my server is taking care of itself, thanks to the magic of python and shell scripting J