Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin Monitoring cronjobs and getting alerts on cronjob failures

Discussion in 'System Administration' started by eva2000, Jun 9, 2019.

  1. eva2000

    eva2000 Administrator Staff Member

    44,731
    10,196
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,804
    Local Time:
    12:55 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Twitter question was posted asking how to monitor cronjob runs and alerts at Simon Fearby (Aussie DevSecOps) on Twitter and I replied with one 3rd party service I have used in the past at Healthchecks.io. However, the methodology used in healthchecks.io, can be replicated somewhat via just some custom shell scripting know-how :)

    I quickly whipped up a custom cron-alert.sh script which takes the exit status of cronjob's script run and determines if the cronjob script successfully ran (exit status = 0) or failed to run and had errors (exit status != 0). Then in cron-alert.sh I configured it to send the cronjob script's run status to a custom Slack channel I setup via a Slack incoming webhook.

    And there you have it a basic cronjob monitor and alert system :D

    Very basic cronjob run would now be like below where cronjob.sh is cronjob script (can also be a command etc), exporting the cronjob.sh exit status to variable cron_err and then running cron-alert.sh which inspects cron_err for exit status and sends a Slack channel alert based on exit status = 0 (successful) vs != 0 (failed).
    Code (Text):
    ./cronjob.sh ;export cron_err=$?; echo $cron_err; ./cron-alert.sh


    Within custom cron-alerts Slack channel

    slack-cron-alert-01.png

    A more advanced version where export script_link and script_name for full path to cronjob script and it's name of the file so that gets passed to cron-alert.sh to add to the Slack message sent.
    Code (Text):
    export script_link='/etc/centminmod/serverinfo.sh';
    export script_name=$(basename $script_link); $script_link;
    export cron_err=$?; echo $cron_err; /root/tools/crontest/cron-alert.sh
    


    slack-cron-alert-02.png

    And passing on the server's hostname or a defined hostname alias which you can set in cron-alert.sh. So if no hostname alias is set, fall back to server hostname.

    slack-cron-alert-03.png

    The wonders of simple shell scripting and Slack incoming webhooks :D