Get the most out of your Centmin Mod LEMP stack
Become a Member

Nginx Nginx Amplify Open Source Monitoring Service

Discussion in 'Nginx and PHP-FPM news & discussions' started by eva2000, Nov 18, 2015.

  1. pamamolf

    pamamolf Premium Member Premium Member

    3,481
    335
    83
    May 31, 2014
    Ratings:
    +642
    Local Time:
    12:50 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok it is better to have a fail2ban compatible log file than for Amplify :)
     
  2. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  3. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    nice

     
    • Like Like x 1
  4. GamerJota

    GamerJota Member

    56
    7
    8
    Mar 1, 2016
    Chile
    Ratings:
    +18
    Local Time:
    6:50 PM
    @eva2000
    So as I understand centminmod works out of the box with nginx Amplify by detecting the main_ext that comes installed by default as explained in Beta Branch - switch nginx access_log to custom main_ext format by default 123.09beta01.

    So I enabled PHP-FPM status PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS.

    I set up everything, also including the MySQL monitoring, looks like it works fine except for two nginx metrics:
    • NGINX HTTP Version
    • NGINX Request time
    I get "Metrics unavailable. Please check the docs" for both.

    This is the relevant configuration of nginx.conf, completely vanilla and untouched:

    Code:
    log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" '
                          '"$host" sn="$server_name" '
                          'rt=$request_time '
                          'ua="$upstream_addr" us="$upstream_status" '
                          'ut="$upstream_response_time" ul="$upstream_response_leng$
                          'cs=$upstream_cache_status' ;
    
    access_log  off;
    error_log   logs/error.log warn;
    I don't think that access_log being off and the error_log without main_ext is okay?
    Or is it? How does centminmod auto-detects the main_ext?
     
  5. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    log_format main_ext is an optional format so you change and edit your nginx vhosts to use main_ext similar to how example at Module ngx_http_log_module changes access_log to use custom log_format = compression
     
  6. GamerJota

    GamerJota Member

    56
    7
    8
    Mar 1, 2016
    Chile
    Ratings:
    +18
    Local Time:
    6:50 PM
    Oh! So I have to change the access_log from combined to main_ext for every vhost conf and also virtual.conf to make it work with nginx Amplify?
    error_log should be fine at error level as I see.

    I thought this was "out of the box" too, haha.
     
    Last edited: Feb 24, 2018
  7. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yup. I did have it as default temporarily but it may break fail2ban and it's scanning of logged regular expressions Security - fail2ban for Centmin Mod + CSF Firewall / Cloudflare API
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    3,481
    335
    83
    May 31, 2014
    Ratings:
    +642
    Local Time:
    12:50 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Better to support fail2ban than Amplify :)
     
    • Agree Agree x 1
  9. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Technically you can support both by having duplicate access_log defined one for default combine and one for main_ext
    Code (Text):
    access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
    access_log /home/nginx/domains/domain.com/log/access-ext.log main_ext buffer=256k flush=5m;
    

    which will mean double the disk space consumed by nginx access logs and more resources used for logging on high traffic sites.
     
    • Informative Informative x 1
  10. rdan

    rdan Well-Known Member

    4,672
    1,122
    113
    May 25, 2014
    Ratings:
    +1,664
    Local Time:
    5:50 AM
    Mainline
    10.2
    That's why I turn it off in all vhost :D.
     
    • Like Like x 1
  11. pamamolf

    pamamolf Premium Member Premium Member

    3,481
    335
    83
    May 31, 2014
    Ratings:
    +642
    Local Time:
    12:50 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Which metrics will help me to know if i am close to a limit for:

    1)Nginx worker_connections 50000;
    2)Phpfpm pm.max_children = 80

    ?
     
  12. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  13. MakeShyft

    MakeShyft New Member

    8
    2
    3
    Apr 13, 2018
    Ratings:
    +4
    Local Time:
    5:50 PM
    My amplify service periodically reports that NGINX is down, but its obviously running because sites are running. When I use the restart command everything seems to work fine again. Anyone else experiencing that?....

    I've only been running CentMinMod for a few days....very happy so far.....but still a total noob. its not a big deal i guess...but i do enjoy having the amplify service for monitoring. unless there is a better one I am unaware of. Cheers
     
    • Informative Informative x 1
  14. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah only one of my servers at OVH runs into that for unexplained reasons sometimes but every other server seems fine. Haven't been able to figure out why but the OVH server has no live traffic it's just for testing so maybe related.
     
  15. MakeShyft

    MakeShyft New Member

    8
    2
    3
    Apr 13, 2018
    Ratings:
    +4
    Local Time:
    5:50 PM
    let me know if u ever figure it out ..... and i will do the same..... is there a way to automate the restart somehow? this is where some of my linux noobness is showing. :)
     
  16. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    You mean the amplify-agent service Amplify | Library ? or nginx ? You can write your own custom script via cronjob to check if service is running and restart it or you can use something like monit which you need to install and configure yourself Easy, proactive monitoring of processes, programs, files, directories, filesystems and hosts | Monit

    Though for me amplify-agent service is always running even when amplify doesn't report stats so checking to see if amplify-agent service is running might not work.
     
  17. MakeShyft

    MakeShyft New Member

    8
    2
    3
    Apr 13, 2018
    Ratings:
    +4
    Local Time:
    5:50 PM
    I meant the amplify agent ..... i will look into creating cron job to check and restart. thanks for the hint.
     
  18. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    This is with monit monitoring configured on centos 7.4 for some centmin mod services
    Code (Text):
    monit summary
    Monit 5.25.1 uptime: 1m
    ┌─────────────────────────────────┬────────────────────────────┬───────────────┐
    │ Service Name                    │ Status                     │ Type          │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ hostname                        │ OK                         │ System        │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ sshd                            │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ postfix                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ php-fpm                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ nginx                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ memcached                       │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ mysqld                          │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ lfd                             │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ crond                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ amplify-agent                   │ OK                         │ Process       │
    └─────────────────────────────────┴────────────────────────────┴───────────────┘


    Code (Text):
    monit status
    Monit 5.25.1 uptime: 1m
    
    Process 'sshd'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          5589
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       4d 11h 12m
      threads                      1
      children                     2
      cpu                          -
      cpu total                    -
      memory                       0.2% [3.2 MB]
      memory total                 0.6% [11.9 MB]
      security attribute           (null)
      disk read                    0 B/s [293.4 MB total]
      disk write                   0 B/s [416.5 MB total]
      port response time           9.524 ms to 127.0.0.1:22 type TCP/IP protocol SSH
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'postfix'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          3033
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       4d 10h 53m
      threads                      1
      children                     7
      cpu                          -
      cpu total                    -
      memory                       0.1% [1.1 MB]
      memory total                 1.5% [30.1 MB]
      security attribute           (null)
      disk read                    0 B/s [56.6 MB total]
      disk write                   0 B/s [1.5 MB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'php-fpm'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          20712
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       4h 48m
      threads                      1
      children                     1
      cpu                          -
      cpu total                    -
      memory                       0.5% [9.4 MB]
      memory total                 0.9% [19.3 MB]
      security attribute           (null)
      disk read                    0 B/s [6.1 MB total]
      disk write                   0 B/s [12 kB total]
      port response time           0.313 ms to localhost:9000 type TCP/IP protocol DEFAULT
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'nginx'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          20700
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       4h 48m
      threads                      1
      children                     2
      cpu                          -
      cpu total                    -
      memory                       2.2% [45.2 MB]
      memory total                 8.7% [178.6 MB]
      security attribute           (null)
      disk write                   0 B/s [4 kB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'memcached'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          17999
      parent pid                   1
      uid                          1001
      effective uid                1001
      gid                          1002
      uptime                       4d 10h 51m
      threads                      10
      children                     0
      cpu                          -
      cpu total                    -
      memory                       0.1% [1.2 MB]
      memory total                 0.1% [1.2 MB]
      security attribute           (null)
      disk read                    0 B/s [352 kB total]
      port response time           0.217 ms to localhost:11211 type TCP/IP protocol DEFAULT
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'mysqld'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          5136
      parent pid                   1
      uid                          996
      effective uid                996
      gid                          994
      uptime                       4d 10h 49m
      threads                      24
      children                     0
      cpu                          -
      cpu total                    -
      memory                       3.2% [65.6 MB]
      memory total                 3.2% [65.6 MB]
      security attribute           (null)
      disk read                    0 B/s [23.2 MB total]
      disk write                   0 B/s [64.8 MB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'lfd'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          14513
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       20h 51m
      threads                      1
      children                     0
      cpu                          -
      cpu total                    -
      memory                       1.2% [24.3 MB]
      memory total                 1.2% [24.3 MB]
      security attribute           (null)
      disk read                    0 B/s [88 kB total]
      disk write                   0 B/s [12 kB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'crond'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          5652
      parent pid                   1
      uid                          0
      effective uid                0
      gid                          0
      uptime                       4d 11h 12m
      threads                      1
      children                     0
      cpu                          -
      cpu total                    -
      memory                       0.1% [1.3 MB]
      memory total                 0.1% [1.3 MB]
      security attribute           (null)
      disk read                    0 B/s [504.0 MB total]
      disk write                   0 B/s [120.3 MB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    Process 'amplify-agent'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      pid                          22522
      parent pid                   1
      uid                          1000
      effective uid                1000
      gid                          1001
      uptime                       1h 10m
      threads                      3
      children                     0
      cpu                          -
      cpu total                    -
      memory                       1.8% [37.1 MB]
      memory total                 1.8% [37.1 MB]
      security attribute           (null)
      disk write                   0 B/s [1.1 MB total]
      data collected               Tue, 17 Apr 2018 20:51:52
    
    System 'hostname'
      status                       OK
      monitoring status            Monitored
      monitoring mode              active
      on reboot                    start
      load average                 [0.04] [0.03] [0.00]
      cpu                          0.0%us 0.0%sy 0.0%wa
      memory usage                 522.1 MB [25.5%]
      swap usage                   4.6 MB [0.2%]
      uptime                       4d 11h 30m
      boot time                    Fri, 13 Apr 2018 09:21:43
      data collected               Tue, 17 Apr 2018 20:51:52


    Test monit auto restarts of monitored services.

    Manually kill amplify-agent service
    Code (Text):
    kill -9 $(pidof amplify-agent)
    

    check monit summary there's a 60s delay and 120 second interval check for downed services before restarts configured so
    Code (Text):
    monit summary
    Monit 5.25.1 uptime: 9m
    ┌─────────────────────────────────┬────────────────────────────┬───────────────┐
    │ Service Name                    │ Status                     │ Type          │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ hostname.domainname.com         │ OK                         │ System        │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ sshd                            │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ postfix                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ php-fpm                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ nginx                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ memcached                       │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ mysqld                          │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ lfd                             │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ crond                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ amplify-agent                   │ Does not exist             │ Process       │
    └─────────────────────────────────┴────────────────────────────┴───────────────┘
    

    monit detects amplify-agent is down so restarts it automatically after 60 seconds delay from default 120 sec interval checks so ~3 mins until restart happens
    Code (Text):
    tail -100 /var/log/messages | grep monit | sed -e "s|$(hostname -s)|hostname|g" | grep -v Skipping
    
    Apr 17 21:20:22 hostname monit[2096]: 'amplify-agent' process is not running
    Apr 17 21:20:22 hostname monit: 'amplify-agent' process is not running
    Apr 17 21:20:22 hostname monit[2096]: 'amplify-agent' trying to restart
    Apr 17 21:20:22 hostname monit[2096]: 'amplify-agent' start: '/etc/init.d/amplify-agent start'
    Apr 17 21:20:22 hostname monit: 'amplify-agent' trying to restart
    Apr 17 21:20:22 hostname monit: 'amplify-agent' start: '/etc/init.d/amplify-agent start'
    Apr 17 21:22:25 hostname monit[2096]: 'amplify-agent' process is running with pid 3275
    Apr 17 21:22:25 hostname monit: 'amplify-agent' process is running with pid 3275
    

    recheck monit summary
    Code (Text):
    monit summary
    Monit 5.25.1 uptime: 21m
    ┌─────────────────────────────────┬────────────────────────────┬───────────────┐
    │ Service Name                    │ Status                     │ Type          │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ hostname.domainname.com         │ OK                         │ System        │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ sshd                            │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ postfix                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ php-fpm                         │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ nginx                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ memcached                       │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ mysqld                          │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ lfd                             │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ crond                           │ OK                         │ Process       │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ amplify-agent                   │ OK                         │ Process       │
    └─────────────────────────────────┴────────────────────────────┴───────────────┘
    
     
  19. negative

    negative Active Member

    347
    39
    28
    Apr 11, 2015
    Ratings:
    +81
    Local Time:
    12:50 AM
    1.9.10
    10.1.11
    @eva2000

    I've installed the "monit" service and after when i type the monit summary command;

    Code (Text):
    [15:05][[email protected] ~]# monit summary
    Monit 5.25.1 uptime: 1m
    ┌─────────────────────────────────┬────────────────────────────┬───────────────┐
    │ Service Name                    │ Status                     │ Type          │
    ├─────────────────────────────────┼────────────────────────────┼───────────────┤
    │ server                        │ OK                         │ System        │
    └─────────────────────────────────┴────────────────────────────┴───────────────┘


    Why other process don't show ?
     
  20. eva2000

    eva2000 Administrator Staff Member

    41,755
    9,401
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,446
    Local Time:
    7:50 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    not enough to install monit, it needs to be configured specifically for each service you want to monitor :)
     
    • Optimistic Optimistic x 1