Discover Centmin Mod today
Register Now

Sysadmin Pushover

Discussion in 'System Administration' started by Matt, May 30, 2014.

  1. Matt

    Matt Moderator Staff Member

    691
    319
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +444
    Local Time:
    4:05 PM
    1.7.1
    MariaDB 10
    There are currently 2 add-ons that work with Pushover and XenForo, but Pushover itself supports various different languages.

    https://pushover.net/

    So, I wrote my own little bash script to use their API to monitor the key services on my server, and alert me if they go down.

    First, create a list of the processes running
    Code:
    /usr/local/sbin/nginx
    /usr/sbin/mysqld
    /usr/local/etc/php-fpm.conf
    /usr/local/bin/memcached
    /usr/bin/java
    Then, the bash script:
    Code:
    #!/bin/bash
    
    echo "Checking all processes are running....";
    
    for i in `/bin/cat /root/scripts/process_list.txt`
    
    do
    
    if
    [ $(ps aux | grep -e "$i" | grep -v grep | wc -l | tr -s "\n") -eq 0 ];
    
    then
            printf "$i is down.....oh dear!!!\n";
            curl -s \
            -F "token=YOURAPITOKEN" \
            -F "user=YOURDEVICEID" \
            -F "message=$i is down.....oh dear!!!" \
            -F "priority=1"\
            https://api.pushover.net/1/messages.json
    
    else
    
            printf "$i running\n";
    
    fi
    
    done
    This then sends the alert out to my phone, makes it RED, and bypass the "quiet time" settings.
    photo 1.PNG


    photo 2.PNG
     
    • Like Like x 4
    • Informative Informative x 1
  2. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    Very nice :)
     
    • Like Like x 1
  3. palPalani

    palPalani Member

    30
    11
    8
    May 30, 2014
    Chennai, IN
    Ratings:
    +19
    Local Time:
    8:35 PM
    Nginx 1.7
    MariaDB 5.5
    Wow! very simple, but doing all my needs... will check that soon!
     
    • Like Like x 1
  4. Matt

    Matt Moderator Staff Member

    691
    319
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +444
    Local Time:
    4:05 PM
    1.7.1
    MariaDB 10
    Thank you. You can literally integrate it into any monitoring script you could possibly want to write.
     
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    now you'd giving me ideas for future scripts and Centmin Mod :D
     
    • Like Like x 1
  6. Matt

    Matt Moderator Staff Member

    691
    319
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +444
    Local Time:
    4:05 PM
    1.7.1
    MariaDB 10
    I use it at work as well for monitoring some of the servers we have running there.
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    awesome just added pushover notifications to my database backup script for the forums :cool: (y)
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    @Matt, do you know of anyway to get past the pushover.net API 512 character limit for message text ?
     
  9. Matt

    Matt Moderator Staff Member

    691
    319
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +444
    Local Time:
    4:05 PM
    1.7.1
    MariaDB 10
    No, I don't think you can get around it:

     
    • Like Like x 1
  10. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    Darn, my backup scripts also detail stats for the backup process like time taken, size and what and how many files were auto deleted once they hit the retention threshold. Was hoping to have that detail shown in pushover too. Guess just the backup was successful message is enough for now :)

    Sample backup email notification I get with full error detection at each backup process stage. If any errors are detected, auto retention deletion routine aborts before needlessly deleting old backups :)

    Code:
    /root/tools/mandbbackup_xf.sh backup dbname
    
    Successfully backed up database: dbname
    Successfully backed up web root directory for dbname
    Successfully backed up database schema: dbname
    pigz -4R -p4 /home/nginx/domains/domain.com/backup_dir/dbname_140614-195610.sql
    pigz -4R -p4 /home/nginx/domains/domain.com/backup_dir/dbname_schema_140614-195610.sql
    Number of backup files in /home/nginx/domains/domain.com/backup_dir before backup or deletion: 44
    Number of backup files in /home/nginx/domains/domain.com/backup_dir after backup or deletion: 47
    
    ----------------------------------------------------------------------------
    /root/tools/mandbbackup_xf.sh 0.1.0
    ----------------------------------------------------------------------------
    /home/nginx/domains/domain.com/backup_dir resides on partition which has
    62553 Megabytes of disk space available
    ----------------------------------------------------------------------------
    MySQL data directory located at: /var/lib/mysql/
    MySQL data directory size: 775 Megabytes
    ----------------------------------------------------------------------------
    [PASS] you have enough free disk space to store MySQL directory size on:
    /home/nginx/domains/domain.com/backup_dir
    ----------------------------------------------------------------------------
    
    ----------------------------------------------------------------------------
    Backup dbname Data: real: 1.09s cpu: 60% maxmem: 14544 KB cswaits: 4795
    ----------------------------------------------------------------------------
    Backup dbname associated web root directory: real: 5.73s cpu: 99% maxmem: 49008 KB cswaits: 4
    ----------------------------------------------------------------------------
    Backup dbname Schema: real: 0.50s cpu: 17% maxmem: 12960 KB cswaits: 4375
    ----------------------------------------------------------------------------
    Pigz dbname database: real: 0.26s cpu: 352% maxmem: 14688 KB cswaits: 415
    ----------------------------------------------------------------------------
    Backup MySQL user GRANT permissions for:
    #mysqlusername root @ 127.0.0.1
    
    *************************************************************************
    created MySQL database backups at: /home/nginx/domains/domain.com/backup_dir
    *************************************************************************
    Jun 14  19:56   6.5M   dbname_140614-195610.sql.gz
    Jun 14  19:56   19K    dbname_schema_140614-195610.sql.gz
    Jun 14  20:03   6.5M   dbname_140614-200334.sql.gz
    Jun 14  20:03   19K    dbname_schema_140614-200334.sql.gz
    
    *************************************************************************
    created web root directory backups at: /home/nginx/domains/domain.com/backup_dir
    *************************************************************************
    Jun 14  19:56   48M    backup_public_140614-195610.zip
    Jun 14  20:03   48M    backup_public_140614-200334.zip
    
    *************************************************************************
    created MySQL user GRANT backups at: /home/nginx/domains/domain.com/backup_dir
    *************************************************************************
    Jun 14  19:56   413    backup_mysqlgrants_140614-195610.sql.gz
    Jun 14  20:03   413    backup_mysqlgrants_140614-200334.sql.gz
    ------------------------------------------------------------------------
    Last backup sql / file(s) created at:
    /home/nginx/domains/domain.com/backup_dir/backup_public_140614-200334.zip
    /home/nginx/domains/domain.com/backup_dir/dbname_schema_140614-200334.sql.gz
    /home/nginx/domains/domain.com/backup_dir/backup_mysqlgrants_140614-200334.sql.gz
    ------------------------------------------------------------------------
    Backup Log:
    /home/nginx/domains/domain.com/backup_dirlogs/dbname_backup_140614-200334.log
    ------------------------------------------------------------------------
     
    Last edited: Jun 15, 2014
  11. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1
  12. pamamolf

    pamamolf Well-Known Member

    2,725
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    6:05 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Amazing ! Any plans to add anything related to Centminmod?
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1
  14. pamamolf

    pamamolf Well-Known Member

    2,725
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    6:05 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Do i have to run Matt bash script using cronjob every minute ?
     
  15. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    yeah every minute or at an interval you'd find acceptable for downtime for such monitored services
     
  16. pamamolf

    pamamolf Well-Known Member

    2,725
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    6:05 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Anything more than a minute i think is not acceptable :)

    I am afraid to use very low numbers for minutes as i do not want to kill my server running scripts all the time but i can't avoid it in this case :)

    Do you use it also every minute using cronjob or you implement it in any other way?
     
  17. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    Every minute via cron wouldn't cause much load for the particular script @Matt posted.
     
  18. pamamolf

    pamamolf Well-Known Member

    2,725
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    6:05 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    First, create a list of the processes running

    /usr/local/sbin/nginx
    /usr/sbin/mysqld
    /usr/local/etc/php-fpm.conf
    /usr/local/bin/memcached
    /usr/bin/java

    Confused :(

    The script will create this or i must create it?

    How can i select which services to check and send notifications?

    I need for nginx,php-fpm,mysql,memcached,Apc :)

    Does the script let me know which is service is down or that something is down?
     
    Last edited: Jun 18, 2014
  19. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    You have to enter the list of processes you want to monitor into file called /root/scripts/process_list.txt

    command below will show all processes on the server

    Code:
    ps aux
     
    • Like Like x 1
  20. eva2000

    eva2000 Administrator Staff Member

    30,186
    6,786
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,139
    Local Time:
    1:05 AM
    Nginx 1.13.x
    MariaDB 5.5
    Just checked my list for the forum server and it would be like

    Code:
    /usr/local/sbin/nginx
    /usr/sbin/mysqld
    /usr/local/etc/php-fpm.conf
    /usr/local/bin/memcached
    /usr/bin/java
    /usr/sbin/munin-node
    /var/www/cgi-bin/munin-cgi-html
    /var/www/cgi-bin/munin-cgi-graph
    lfd
    crond
    
     
    • Like Like x 1