/ Register

Learn about Centmin Mod LEMP Stack today
Become a Member

Sysadmin Pushover

Discussion in 'System Administration' started by Matt, May 30, 2014.

Tags:
Previous Thread Next Thread
Loading...
Page 1 of 3
  1. May 30, 2014 #1
    Matt

    Matt Well-Known Member

    926
    414
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +669
    Local Time:
    6:43 AM
    1.5.15
    MariaDB 10.2
    There are currently 2 add-ons that work with Pushover and XenForo, but Pushover itself supports various different languages.

    https://pushover.net/

    So, I wrote my own little bash script to use their API to monitor the key services on my server, and alert me if they go down.

    First, create a list of the processes running
    Code:
    /usr/local/sbin/nginx
/usr/sbin/mysqld
/usr/local/etc/php-fpm.conf
/usr/local/bin/memcached
/usr/bin/java
    Then, the bash script:
    Code:
    #!/bin/bash

echo "Checking all processes are running....";

for i in `/bin/cat /root/scripts/process_list.txt`

do

if
[ $(ps aux | grep -e "$i" | grep -v grep | wc -l | tr -s "\n") -eq 0 ];

then
        printf "$i is down.....oh dear!!!\n";
        curl -s \
        -F "token=YOURAPITOKEN" \
        -F "user=YOURDEVICEID" \
        -F "message=$i is down.....oh dear!!!" \
        -F "priority=1"\
        https://api.pushover.net/1/messages.json

else

        printf "$i running\n";

fi

done
    This then sends the alert out to my phone, makes it RED, and bypass the "quiet time" settings.
    photo 1.PNG


    photo 2.PNG
     
  2. May 30, 2014 #2
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Very nice :)
     
  3. May 31, 2014 #3
    palPalani

    palPalani Member

    30
    12
    8
    May 30, 2014
    Chennai, IN
    Ratings:
    +20
    Local Time:
    11:13 AM
    Nginx 1.7
    MariaDB 5.5
    Wow! very simple, but doing all my needs... will check that soon!
     
  4. May 31, 2014 #4
    Matt

    Matt Well-Known Member

    926
    414
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +669
    Local Time:
    6:43 AM
    1.5.15
    MariaDB 10.2
    Thank you. You can literally integrate it into any monitoring script you could possibly want to write.
     
  5. May 31, 2014 #5
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    now you'd giving me ideas for future scripts and Centmin Mod :D
     
  6. May 31, 2014 #6
    Matt

    Matt Well-Known Member

    926
    414
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +669
    Local Time:
    6:43 AM
    1.5.15
    MariaDB 10.2
    I use it at work as well for monitoring some of the servers we have running there.
     
  7. Jun 14, 2014 #7
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    awesome just added pushover notifications to my database backup script for the forums :cool: (y)
     
  8. Jun 14, 2014 #8
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @Matt, do you know of anyway to get past the pushover.net API 512 character limit for message text ?
     
  9. Jun 15, 2014 #9
    Matt

    Matt Well-Known Member

    926
    414
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +669
    Local Time:
    6:43 AM
    1.5.15
    MariaDB 10.2
    No, I don't think you can get around it:

     
  10. Jun 15, 2014 #10
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Darn, my backup scripts also detail stats for the backup process like time taken, size and what and how many files were auto deleted once they hit the retention threshold. Was hoping to have that detail shown in pushover too. Guess just the backup was successful message is enough for now :)

    Sample backup email notification I get with full error detection at each backup process stage. If any errors are detected, auto retention deletion routine aborts before needlessly deleting old backups :)

    Code:
    /root/tools/mandbbackup_xf.sh backup dbname

Successfully backed up database: dbname
Successfully backed up web root directory for dbname
Successfully backed up database schema: dbname
pigz -4R -p4 /home/nginx/domains/domain.com/backup_dir/dbname_140614-195610.sql
pigz -4R -p4 /home/nginx/domains/domain.com/backup_dir/dbname_schema_140614-195610.sql
Number of backup files in /home/nginx/domains/domain.com/backup_dir before backup or deletion: 44
Number of backup files in /home/nginx/domains/domain.com/backup_dir after backup or deletion: 47

----------------------------------------------------------------------------
/root/tools/mandbbackup_xf.sh 0.1.0
----------------------------------------------------------------------------
/home/nginx/domains/domain.com/backup_dir resides on partition which has
62553 Megabytes of disk space available
----------------------------------------------------------------------------
MySQL data directory located at: /var/lib/mysql/
MySQL data directory size: 775 Megabytes
----------------------------------------------------------------------------
[PASS] you have enough free disk space to store MySQL directory size on:
/home/nginx/domains/domain.com/backup_dir
----------------------------------------------------------------------------

----------------------------------------------------------------------------
Backup dbname Data: real: 1.09s cpu: 60% maxmem: 14544 KB cswaits: 4795
----------------------------------------------------------------------------
Backup dbname associated web root directory: real: 5.73s cpu: 99% maxmem: 49008 KB cswaits: 4
----------------------------------------------------------------------------
Backup dbname Schema: real: 0.50s cpu: 17% maxmem: 12960 KB cswaits: 4375
----------------------------------------------------------------------------
Pigz dbname database: real: 0.26s cpu: 352% maxmem: 14688 KB cswaits: 415
----------------------------------------------------------------------------
Backup MySQL user GRANT permissions for:
#mysqlusername root @ 127.0.0.1

*************************************************************************
created MySQL database backups at: /home/nginx/domains/domain.com/backup_dir
*************************************************************************
Jun 14  19:56   6.5M   dbname_140614-195610.sql.gz
Jun 14  19:56   19K    dbname_schema_140614-195610.sql.gz
Jun 14  20:03   6.5M   dbname_140614-200334.sql.gz
Jun 14  20:03   19K    dbname_schema_140614-200334.sql.gz

*************************************************************************
created web root directory backups at: /home/nginx/domains/domain.com/backup_dir
*************************************************************************
Jun 14  19:56   48M    backup_public_140614-195610.zip
Jun 14  20:03   48M    backup_public_140614-200334.zip

*************************************************************************
created MySQL user GRANT backups at: /home/nginx/domains/domain.com/backup_dir
*************************************************************************
Jun 14  19:56   413    backup_mysqlgrants_140614-195610.sql.gz
Jun 14  20:03   413    backup_mysqlgrants_140614-200334.sql.gz
------------------------------------------------------------------------
Last backup sql / file(s) created at:
/home/nginx/domains/domain.com/backup_dir/backup_public_140614-200334.zip
/home/nginx/domains/domain.com/backup_dir/dbname_schema_140614-200334.sql.gz
/home/nginx/domains/domain.com/backup_dir/backup_mysqlgrants_140614-200334.sql.gz
------------------------------------------------------------------------
Backup Log:
/home/nginx/domains/domain.com/backup_dirlogs/dbname_backup_140614-200334.log
------------------------------------------------------------------------
     
    Last edited: Jun 15, 2014
  11. Jun 18, 2014 #11
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  12. Jun 18, 2014 #12
    pamamolf

    pamamolf Premium Member Premium Member

    4,069
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    8:43 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Amazing ! Any plans to add anything related to Centminmod?
     
  13. Jun 18, 2014 #13
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  14. Jun 18, 2014 #14
    pamamolf

    pamamolf Premium Member Premium Member

    4,069
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    8:43 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Do i have to run Matt bash script using cronjob every minute ?
     
  15. Jun 18, 2014 #15
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah every minute or at an interval you'd find acceptable for downtime for such monitored services
     
  16. Jun 18, 2014 #16
    pamamolf

    pamamolf Premium Member Premium Member

    4,069
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    8:43 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Anything more than a minute i think is not acceptable :)

    I am afraid to use very low numbers for minutes as i do not want to kill my server running scripts all the time but i can't avoid it in this case :)

    Do you use it also every minute using cronjob or you implement it in any other way?
     
  17. Jun 18, 2014 #17
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Every minute via cron wouldn't cause much load for the particular script @Matt posted.
     
  18. Jun 18, 2014 #18
    pamamolf

    pamamolf Premium Member Premium Member

    4,069
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    8:43 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    First, create a list of the processes running

    /usr/local/sbin/nginx
    /usr/sbin/mysqld
    /usr/local/etc/php-fpm.conf
    /usr/local/bin/memcached
    /usr/bin/java

    Confused :(

    The script will create this or i must create it?

    How can i select which services to check and send notifications?

    I need for nginx,php-fpm,mysql,memcached,Apc :)

    Does the script let me know which is service is down or that something is down?
     
    Last edited: Jun 18, 2014
  19. Jun 18, 2014 #19
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    You have to enter the list of processes you want to monitor into file called /root/scripts/process_list.txt

    command below will show all processes on the server

    Code:
    ps aux
     
  20. Jun 18, 2014 #20
    eva2000

    eva2000 Administrator Staff Member

    53,537
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    3:43 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Just checked my list for the forum server and it would be like

    Code:
    /usr/local/sbin/nginx
/usr/sbin/mysqld
/usr/local/etc/php-fpm.conf
/usr/local/bin/memcached
/usr/bin/java
/usr/sbin/munin-node
/var/www/cgi-bin/munin-cgi-html
/var/www/cgi-bin/munin-cgi-graph
lfd
crond
     
Previous Thread Next Thread
Loading...
Page 1 of 3

.