Sysadmin Monitoring cronjobs and getting alerts on cronjob failures

Twitter question was posted asking how to monitor cronjob runs and alerts at Simon Fearby (Aussie DevSecOps) on Twitter and I replied with one 3rd party service I have used in the past at Healthchecks.io. However, the methodology used in healthchecks.io, can be replicated somewhat via just some custom shell scripting know-how

I quickly whipped up a custom cron-alert.sh script which takes the exit status of cronjob's script run and determines if the cronjob script successfully ran (exit status = 0) or failed to run and had errors (exit status != 0). Then in cron-alert.sh I configured it to send the cronjob script's run status to a custom Slack channel I setup via a Slack incoming webhook.

And there you have it a basic cronjob monitor and alert system

Very basic cronjob run would now be like below where cronjob.sh is cronjob script (can also be a command etc), exporting the cronjob.sh exit status to variable cron_err and then running cron-alert.sh which inspects cron_err for exit status and sends a Slack channel alert based on exit status = 0 (successful) vs != 0 (failed).

Code (Text):

./cronjob.sh ;export cron_err=$?; echo $cron_err; ./cron-alert.sh

Within custom cron-alerts Slack channel



A more advanced version where export script_link and script_name for full path to cronjob script and it's name of the file so that gets passed to cron-alert.sh to add to the Slack message sent.

Code (Text):

export script_link='/etc/centminmod/serverinfo.sh';
export script_name=$(basename $script_link); $script_link;
export cron_err=$?; echo $cron_err; /root/tools/crontest/cron-alert.sh

And passing on the server's hostname or a defined hostname alias which you can set in cron-alert.sh. So if no hostname alias is set, fall back to server hostname.



The wonders of simple shell scripting and Slack incoming webhooks