Learn about Centmin Mod LEMP Stack today
Register Now

Nginx PHP-FPM Some strange rate limiting

Discussion in 'Install & Upgrades or Pre-Install Questions' started by jair, Jul 9, 2019.

  1. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.17.1 (latest)
    • PHP Version Installed: 7.3.7
    • MariaDB MySQL Version Installed: 10.3.16
    • When was last time updated Centmin Mod code base ? : yesterday
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      LETSENCRYPT_DETECT='y'
      
    I read pretty much all there is about rate nginx rate limiting, both here on SO and most of the links that seemed relevant to me. Still, I cannot solve the issue.

    Background: I switched from VPS to dedicated. The dedicated has like 2x the resources of the VPS so by transferring most of the settings I assumed that I should be fine. Alas, this is not the case and apparently I am hitting a severe rate limit that causes many frequent visitors to my forum to complain, I can notice it as well.

    The lowest level I can replicate the issue is at php status page via lynx in the console:
    Code (Text):
    lynx --dump  http://127.0.0.1/phpstatus
    


    If I do that really fast, within a minute it stops giving results. I actually noticed that when I ran the netdata monitoring, which for some reason did not give me phpfpm stats. When I debugged it as per the author instructions, this is what I got:

    Code (Text):
    2019-07-08 16:09:57: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
    2019-07-08 16:09:59: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))
    


    So this leads me to believe that either nginx and/or php-fpm are rate limited and even localhost requests are blocked by it.

    When I restart nginx it works briefly for few seconds, then again the same error. Full log after nginx is restarted:

    Code:
    [16:09][[email protected] nginx]$ /usr/libexec/netdata/plugins.d/python.d.plugin debug 1 phpfpm
    2019-07-08 16:12:30: python.d INFO: plugin[main] : using python v2
    2019-07-08 16:12:30: python.d INFO: plugin[main] : starting setup
    2019-07-08 16:12:30: python.d INFO: plugin[main] : checking for config in ['/etc/netdata', '/usr/lib/netdata/conf.d']
    2019-07-08 16:12:30: python.d INFO: plugin[main] : config found, loading config '/usr/lib/netdata/conf.d/python.d.conf'
    2019-07-08 16:12:30: python.d INFO: plugin[main] : config successfully loaded
    2019-07-08 16:12:30: python.d INFO: plugin[main] : starting checker process (1 module(s) to check)
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : starting...
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : checking
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : source successfully loaded
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : found config file '/etc/netdata/python.d/phpfpm.conf'
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : created 1 job(s) from the config
    2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm[local] : check successful
    2019-07-08 16:12:30: python.d INFO: plugin[checker] : terminating...
    2019-07-08 16:12:30: python.d INFO: plugin[main] : stopping checker process
    2019-07-08 16:12:30: python.d INFO: plugin[main] : setup complete, 1 active module(s) : '['phpfpm']'
    2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm : created 1 job(s)
    2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
    2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : init successful
    2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : check successful
    CHART netdata.runtime_phpfpm_local '' 'Execution time for phpfpm_local' 'ms' 'python.d' netdata.pythond_runtime line 145000 1
    DIMENSION run_time 'run time' absolute 1 1
    
    2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : started in thread
    2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : started, update frequency: 1
    CHART phpfpm_local.connections '' 'PHP-FPM Active Connections' 'connections' 'active connections' 'phpfpm.connections' line 60000 1 '' 'python.d.plugin' 'phpfpm'
    DIMENSION 'active' 'active' absolute 1 1 ' '
    DIMENSION 'maxActive' 'max active' absolute 1 1 ' '
    DIMENSION 'idle' 'idle' absolute 1 1 ' '
    
    BEGIN phpfpm_local.connections 0
    SET 'active' = 5
    SET 'maxActive' = 42
    SET 'idle' = 11
    END
    
    CHART phpfpm_local.performance '' 'PHP-FPM Performance' 'status' 'performance' 'phpfpm.performance' line 60002 1 '' 'python.d.plugin' 'phpfpm'
    DIMENSION 'reached' 'max children reached' absolute 1 1 ' '
    DIMENSION 'slow' 'slow requests' absolute 1 1 ' '
    
    BEGIN phpfpm_local.performance 0
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    CHART phpfpm_local.requests '' 'PHP-FPM Requests' 'requests/s' 'requests' 'phpfpm.requests' line 60001 1 '' 'python.d.plugin' 'phpfpm'
    DIMENSION 'requests' 'requests' incremental 1 1 ' '
    
    BEGIN phpfpm_local.requests 0
    SET 'requests' = 7077
    END
    
    BEGIN netdata.runtime_phpfpm_local 0
    SET run_time = 3
    END
    
    2019-07-08 16:12:31: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 3, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000133
    SET 'active' = 6
    SET 'maxActive' = 42
    SET 'idle' = 10
    END
    
    BEGIN phpfpm_local.performance 1000133
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000133
    SET 'requests' = 7080
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000133
    SET run_time = 2
    END
    
    2019-07-08 16:12:32: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000042
    SET 'active' = 9
    SET 'maxActive' = 42
    SET 'idle' = 7
    END
    
    BEGIN phpfpm_local.performance 1000042
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000042
    SET 'requests' = 7085
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000042
    SET run_time = 2
    END
    
    2019-07-08 16:12:33: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999927
    SET 'active' = 7
    SET 'maxActive' = 42
    SET 'idle' = 9
    END
    
    BEGIN phpfpm_local.performance 999927
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999927
    SET 'requests' = 7087
    END
    
    BEGIN netdata.runtime_phpfpm_local 999927
    SET run_time = 2
    END
    
    2019-07-08 16:12:34: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000054
    SET 'active' = 17
    SET 'maxActive' = 42
    SET 'idle' = 0
    END
    
    BEGIN phpfpm_local.performance 1000054
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000054
    SET 'requests' = 7099
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000054
    SET run_time = 2
    END
    
    2019-07-08 16:12:35: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999990
    SET 'active' = 11
    SET 'maxActive' = 42
    SET 'idle' = 5
    END
    
    BEGIN phpfpm_local.performance 999990
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999990
    SET 'requests' = 7104
    END
    
    BEGIN netdata.runtime_phpfpm_local 999990
    SET run_time = 2
    END
    
    2019-07-08 16:12:36: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000015
    SET 'active' = 12
    SET 'maxActive' = 42
    SET 'idle' = 4
    END
    
    BEGIN phpfpm_local.performance 1000015
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000015
    SET 'requests' = 7106
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000015
    SET run_time = 2
    END
    
    2019-07-08 16:12:37: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999943
    SET 'active' = 12
    SET 'maxActive' = 42
    SET 'idle' = 4
    END
    
    BEGIN phpfpm_local.performance 999943
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999943
    SET 'requests' = 7108
    END
    
    BEGIN netdata.runtime_phpfpm_local 999943
    SET run_time = 2
    END
    
    2019-07-08 16:12:38: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000041
    SET 'active' = 14
    SET 'maxActive' = 42
    SET 'idle' = 2
    END
    
    BEGIN phpfpm_local.performance 1000041
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000041
    SET 'requests' = 7111
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000041
    SET run_time = 2
    END
    
    2019-07-08 16:12:39: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999998
    SET 'active' = 13
    SET 'maxActive' = 42
    SET 'idle' = 3
    END
    
    BEGIN phpfpm_local.performance 999998
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999998
    SET 'requests' = 7113
    END
    
    BEGIN netdata.runtime_phpfpm_local 999998
    SET run_time = 2
    END
    
    2019-07-08 16:12:40: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_duration' was suppressed due to non updating
    2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_mem' was suppressed due to non updating
    2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_cpu' was suppressed due to non updating
    BEGIN phpfpm_local.connections 999941
    SET 'active' = 14
    SET 'maxActive' = 42
    SET 'idle' = 2
    END
    
    BEGIN phpfpm_local.performance 999941
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999941
    SET 'requests' = 7115
    END
    
    BEGIN netdata.runtime_phpfpm_local 999941
    SET run_time = 2
    END
    
    2019-07-08 16:12:41: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000089
    SET 'active' = 15
    SET 'maxActive' = 42
    SET 'idle' = 1
    END
    
    BEGIN phpfpm_local.performance 1000089
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000089
    SET 'requests' = 7118
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000089
    SET run_time = 2
    END
    
    2019-07-08 16:12:42: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999980
    SET 'active' = 15
    SET 'maxActive' = 42
    SET 'idle' = 1
    END
    
    BEGIN phpfpm_local.performance 999980
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999980
    SET 'requests' = 7120
    END
    
    BEGIN netdata.runtime_phpfpm_local 999980
    SET run_time = 2
    END
    
    2019-07-08 16:12:43: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999999
    SET 'active' = 14
    SET 'maxActive' = 42
    SET 'idle' = 2
    END
    
    BEGIN phpfpm_local.performance 999999
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999999
    SET 'requests' = 7122
    END
    
    BEGIN netdata.runtime_phpfpm_local 999999
    SET run_time = 2
    END
    
    2019-07-08 16:12:44: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000020
    SET 'active' = 15
    SET 'maxActive' = 42
    SET 'idle' = 1
    END
    
    BEGIN phpfpm_local.performance 1000020
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000020
    SET 'requests' = 7124
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000020
    SET run_time = 2
    END
    
    2019-07-08 16:12:45: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 999139
    SET 'active' = 17
    SET 'maxActive' = 42
    SET 'idle' = 0
    END
    
    BEGIN phpfpm_local.performance 999139
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 999139
    SET 'requests' = 7128
    END
    
    BEGIN netdata.runtime_phpfpm_local 999139
    SET run_time = 2
    END
    
    2019-07-08 16:12:46: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000829
    SET 'active' = 17
    SET 'maxActive' = 42
    SET 'idle' = 0
    END
    
    BEGIN phpfpm_local.performance 1000829
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000829
    SET 'requests' = 7133
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000829
    SET run_time = 2
    END
    
    2019-07-08 16:12:47: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    BEGIN phpfpm_local.connections 1000020
    SET 'active' = 17
    SET 'maxActive' = 42
    SET 'idle' = 0
    END
    
    BEGIN phpfpm_local.performance 1000020
    SET 'reached' = 1
    SET 'slow' = 0
    END
    
    BEGIN phpfpm_local.requests 1000020
    SET 'requests' = 7136
    END
    
    BEGIN netdata.runtime_phpfpm_local 1000020
    SET run_time = 2
    END
    
    2019-07-08 16:12:48: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    2019-07-08 16:12:51: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))
    2019-07-08 16:12:51: python.d DEBUG: phpfpm[local] : get_data() returned no data
    2019-07-08 16:12:51: python.d DEBUG: phpfpm[local] : update => [FAILED] (elapsed time: -, failed retries in a row: 1)
    2019-07-08 16:12:54: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))
    2019-07-08 16:12:54: python.d DEBUG: phpfpm[local] : get_data() returned no data
    2019-07-08 16:12:54: python.d DEBUG: phpfpm[local] : update => [FAILED] (elapsed time: -, failed retries in a row: 2)
    ^CTraceback (most recent call last):
      File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 733, in <module>
        main()
      File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 727, in main
        plugin.run()
      File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 462, in run
        self.serve()
      File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 619, in serve
        time.sleep(1)
    KeyboardInterrupt
    
    
    Any help is appreciated!
     
  2. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    my nginx.conf

    Code (Text):
    user              nginx nginx;
    worker_processes 16;
    worker_cpu_affinity auto;
    worker_priority -10;
    
    worker_rlimit_nofile 520000;
    timer_resolution 100ms;
    
    pcre_jit on;
    include /usr/local/nginx/conf/dynamic-modules.conf;
    
    
    pid         logs/nginx.pid;
    
    events {
        worker_connections  80000;
        accept_mutex off;
        accept_mutex_delay 200ms;
        use epoll;
        #multi_accept on;
    }
    
    http {
     map_hash_bucket_size 128;
     map_hash_max_size 4096;
     server_names_hash_bucket_size 128;
     server_names_hash_max_size 2048;
     variables_hash_max_size 2048;
    
     resolver 127.0.0.1 valid=10m;
     resolver_timeout 10s;
    #limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=200r/m;
    #limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;
    
    # sets Centmin Mod headers via headers more nginx module
    # https://github.com/openresty/headers-more-nginx-module
    # don't remove the first 2 lines as centmin mod checks to see if they're
    # missing and re-adds them anyway. Just uncomment the 3rd & 4th lines
    # which is used to override the Server header to what you want = nginx
    # and remove the X-Powered-By header + restart nginx service
    # do not disable headers more nginx module itself as it's required for
    # other centmin mod features like redis nginx level caching & letsencrypt
    # integration in vhosts created by addons/acmetool.sh
    more_set_headers "Server: nginx centminmod";
    more_set_headers "X-Powered-By: centminmod";
    #more_set_headers "Server: nginx";
    #more_clear_headers "X-Powered-By";
    
    # uncomment cloudflare.conf include if using cloudflare for
    # server and/or vhost site + setup cron job for command
    # /usr/local/src/centminmod/tools/csfcf.sh auto
    # run the auto command once to populate cloudflare ips
    include /usr/local/nginx/conf/cloudflare.conf;
    # uncomment incapsula.conf include if using incapsula for
    # server and/or vhost site + setup cron job for command
    # /usr/local/src/centminmod/tools/csfincapsula.sh auto
    # run the auto command once to popular incapsula ips
    #include /usr/local/nginx/conf/incapsula.conf;
    include /usr/local/nginx/conf/maintenance.conf;
    #include /usr/local/nginx/conf/vts_http.conf;
    include /usr/local/nginx/conf/geoip.conf;
    include /usr/local/nginx/conf/webp.conf;
    include /usr/local/nginx/conf/ssl_include.conf;
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
    include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
    include /usr/local/nginx/conf/default_phpupstream.conf;
    
    log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    log_format  ddos-proxy '$remote_addr for $http_x_real_ip - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" '
                          '"$host" sn="$server_name" '
                          'rt=$request_time '
                          'ua="$upstream_addr" us="$upstream_status" '
                          'ut="$upstream_response_time" ul="$upstream_response_length" '
                          'cs=$upstream_cache_status' ;
    
    # only uncomment include line to enable it you have enabled ngx_brotli module
    # which is disabled by default https://community.centminmod.com/threads/10688/
    #include /usr/local/nginx/conf/log_format_brotli.conf;
    
    access_log  off;
    error_log   logs/error.log warn;
    
        index  index.php index.html index.htm;
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
    
    sendfile on;
    sendfile_max_chunk 512k;
    tcp_nopush  on;
    tcp_nodelay on;
    server_tokens off;
    server_name_in_redirect off;
    
    keepalive_timeout  5s;
    keepalive_requests  1000;
    lingering_time 20s;
    lingering_timeout 5s;
    keepalive_disable msie6;
    
    gzip on;
    gzip_vary   on;
    gzip_disable msie6;
    gzip_static on;
    gzip_min_length   1400;
    gzip_buffers      1024 8k;
    gzip_http_version 1.1;
    gzip_comp_level 5;
    gzip_proxied    any;
    gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    
    client_body_buffer_size 256k;
    client_body_in_file_only off;
    client_body_timeout 10s;
    client_header_buffer_size 64k;
    ## how long a connection has to complete sending
    ## it's headers for request to be processed
    client_header_timeout  10s;
    client_max_body_size 1024m;
    connection_pool_size  512;
    directio  4m;
    directio_alignment 4096;
    ignore_invalid_headers on;
    large_client_header_buffers 8 64k;
    output_buffers   1 512k;
    postpone_output  1460;
    proxy_temp_path  /tmp/nginx_proxy/;
    request_pool_size  32k;
    reset_timedout_connection on;
    send_timeout     60s;
    types_hash_max_size 2048;
    
    # for nginx proxy backends to prevent redirects to backend port
    # port_in_redirect off;
    
    open_file_cache max=50000 inactive=60s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;
    open_log_file_cache max=10000 inactive=30s min_uses=2;
    
    ## limit number of concurrency connections per ip to 16
    ## add to your server {} section the next line
    ## limit_conn limit_per_ip 16;
    ## uncomment below line allows 500K sessions
    # limit_conn_log_level error;
    #######################################
    # use limit_zone for Nginx <v1.1.7 and lower
    # limit_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    # use limit_conn_zone for Nginx >v1.1.8 and higher
    # limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    
     include /usr/local/nginx/conf/conf.d/*.conf;
    }
    
    



    my phpfpm.conf

    Code (Text):
    [global]
    ; Log level
    ; Possible Values: alert, error, warning, notice, debug
    ; Default Value: notice
    log_level = warning
    pid = /var/run/php-fpm/php-fpm.pid
    error_log = /var/log/php-fpm/www-error.log
    emergency_restart_threshold = 10
    emergency_restart_interval = 1m
    process_control_timeout = 10s
    ;include=/usr/local/nginx/conf/phpfpmd/*.conf
    
    [www]
    user = nginx
    group = nginx
    
    listen = 127.0.0.1:9000
    listen.allowed_clients = 127.0.0.1
    listen.backlog = 8151
    
    ;listen = /tmp/php5-fpm.sock
    listen.owner = nginx
    listen.group = nginx
    listen.mode = 0660
    
    pm = dynamic
    pm.max_children = 16
    ; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
    pm.start_servers = 10
    pm.min_spare_servers = 4
    pm.max_spare_servers = 16
    pm.max_requests = 200
    
    ; PHP 5.3.9 setting
    ; The number of seconds after which an idle process will be killed.
    ; Note: Used only when pm is set to 'ondemand'
    ; Default Value: 10s
    pm.process_idle_timeout = 10s;
    
    rlimit_files = 65536
    rlimit_core = 0
    ; The timeout for serving a single request after which the worker process will
    ; be killed. This option should be used when the 'max_execution_time' ini option
    ; does not stop script execution for some reason. A value of '0' means 'off'.
    ; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
    ; Default Value: 0
    ;request_terminate_timeout = 0
    ; Default Value: 0
    ;request_slowlog_timeout = 0
    slowlog = /var/log/php-fpm/www-slow.log
    
    pm.status_path = /phpstatus
    ping.path = /phpping
    ping.response = pong
    
    ; Limits the extensions of the main script FPM will allow to parse. This can
    ; prevent configuration mistakes on the web server side. You should only limit
    ; FPM to .php extensions to prevent malicious users to use other extensions to
    ; exectute php code.
    ; Note: set an empty value to allow all extensions.
    ; Default Value: .php
    security.limit_extensions = .php
    
    ; catch_workers_output = yes
    php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
    php_admin_value[disable_functions] = shell_exec,exec,system,passthru,pcntl_exec,popen,proc_open
    
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    41,667
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,413
    Local Time:
    7:32 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    that looks like netdata is rate limiting connections not Centmin Mod Nginx

    how was netdata setup/installed and configured ? are you using Centmin Mod Nginx as reverse proxy in front of netdata running instance ? it could be you incorrectly configured such that netdata is also listening and serving requests from /phpstatus which it shouldn't be doing

    if you temporarily remove netdata Nginx reverse proxy config does it fix /phpstatus access ? if so then you found your problem.

    unfortunately netdata setup and configuration is left to end users to figure out :)
     
  4. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    I just used their one line installer, no idea how it setups the webserver :/

    Well, weird thing is that nginx stats work without any issues and they use similar approach - accessing and parsing the status page. Also, I haven't modified any netdata settings about rate limit, so it is weird that netdata itself will limit its own requests in the default settings :)

    Anyway, I am puzzled, I will remove netdata and see if it solves the issue. I fear that netdata symptoms could be red herring, stopping the netdata service definitely did not help.
     
  5. eva2000

    eva2000 Administrator Staff Member

    41,667
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,413
    Local Time:
    7:32 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    only time that would happen is if php-fpm server is overloaded or doesn't have enough php-fpm process/resources to pass onto nginx to be served. If using Centmin Mod 123.09beta01 on CentOS 7 you can also get your php-fpm stats via fpmstats command i.e.
    Code (Text):
    fpmstats
    Processes active: 0, idle: 30, Requests: 3281, slow: 0, Traffic: 0req/sec
    pool:                 www
    process manager:      static
    start time:           19/Jun/2019:17:49:46 +0000
    start since:          1638667
    accepted conn:        3283
    listen queue:         0
    max listen queue:     1
    listen queue len:     4095
    idle processes:       29
    active processes:     1
    total processes:      30
    max active processes: 7
    max children reached: 0
    slow requests:        0
    

    but that still uses /phpstatus at localhost to read them
     
  6. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    Yeap, doing

    Code (Text):
     lynx --dump  http://localhost/nginx_status
    


    many times has absolutely no problem. Phpfpm must be the bottleneck. But my settings are not that unreasonable and my server is practically idle. There are no errors in the log since yesterday when I upgraded to 7.3.7 due to bug giving segmentation faults. Now back to 7.2.x, but issue is not resolved. I will play with the children and processes and see what is going on. This migration has been a nightmare :)
     
  7. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    OK, this is getting more and more confusing. Take a look at my php-fpm stats (I have to wait like 20 seconds for this to show up):

    Code (Text):
    Processes active: 21, idle: 0, Requests: 1040, slow: 0, Traffic: 1.3req/sec
    pool:                 www
    process manager:      dynamic
    start time:           08/Jul/2019:17:24:28 +0000
    start since:          537
    accepted conn:        1065
    listen queue:         2
    max listen queue:     5
    listen queue len:     511
    idle processes:       0
    active processes:     23
    total processes:      23
    max active processes: 31
    max children reached: 1
    slow requests:        0
    


    So after start up I reach max children almost immediately, in the current case it is 20. But the requests themselves are low number - 1000 requests for around 10 min, thats like 2req/s on average. I think much lower amount of php-fpm processes should be able to handle this number of requests...and yet, almost immediately after start all children are busy and the status page starts lagging...

    Sorry for rambling, I find that it kinda helps me when I think out loud :)
     
  8. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    OK, just for testing I bumped max.children from 20 to 40. Just for testing, my server can take it (32 cores, although the model is 6y old). The issue disappeared and here are the results at the same timestamp as above:

    Code (Text):
    Processes active: 24, idle: 2, Requests: 1414, slow: 0, Traffic: 1.9req/sec
    pool:                 www
    process manager:      dynamic
    start time:           08/Jul/2019:17:39:40 +0000
    start since:          537
    accepted conn:        1435
    listen queue:         0
    max listen queue:     1
    listen queue len:     511
    idle processes:       1
    active processes:     25
    total processes:      26
    max active processes: 39
    max children reached: 0
    slow requests:        0
    
    


    So almost 50% more requests handled and there is no lag at the moment. The server is still not loaded, so I can afford to leave it like that. Still the interesting questions are:

    - Such amount of requests should be handled with lower amount of processes, no? Like 3 requests per second does not seem high to me.
    - Why on my previous VPS the same traffic was handled with lower amount of resources and processes?
     
  9. eva2000

    eva2000 Administrator Staff Member

    41,667
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,413
    Local Time:
    7:32 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    hard to know if you didn't monitor and track your php-fpm usage long term to compare as VPS specs and dedicated specs can vary - it's quite each for a high end spec'd VPS to out perform a low/old/slow dedicated too.

    that's where php-fpm tuning for your web app usage requirements comes in :)
     
    • Like Like x 1
  10. jair

    jair Member

    48
    7
    8
    Jan 8, 2017
    Ratings:
    +17
    Local Time:
    12:32 AM
    Well, here are the stats from my previous VPS:

    Code (Text):
    pool:                 www
    process manager:      dynamic
    start time:           16/May/2019:09:19:18 +0000
    start since:          4612387
    accepted conn:        7248947
    listen queue:         0
    max listen queue:     129
    listen queue len:     128
    idle processes:       9
    active processes:     1
    total processes:      10
    max active processes: 20
    max children reached: 26
    slow requests:        0
    
    


    I guess the explanation that the VPS was able to better process the requests due to its high-end CPU is plausible. Maybe thats why I have to increase the children amount on the dedi - to take advantage of the more cores at a lower processing power.

    VPS: 4x Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz (4 cores each, cache size: 8192 KB) vs
    Dedi: 4x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz (8 cores each, cache size: 20480 KB)

    You learn something new every day :)
     
  11. eva2000

    eva2000 Administrator Staff Member

    41,667
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,413
    Local Time:
    7:32 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    As suspected. FYI, a single Xeon E3-1230v1 with 4C/8T is around speed of 2x Xeon E5-26xx v1 Sandy Bridge 6C/12T x 2 = 12C/24T and that is already 10-25% as faster than 2x Xeon E5-2450 v1s. And Xeon E3-1230v3 is around 15-30% faster than E3-1230v1. You do the maths :) But depends on your work loads.

    I like using supermarket checkout lanes as analogy
    • dual xeon E5-2620 = 2x hexa core x 2 HT = 24 cpu threads = 24 lane checkout
    • xeon e3-1270v6 = 1x quad core x 2HT = 8 cpu thread = 8 lane check out
    dual xeon E5-2620
    - each of 24 lanes can process 3 customers per hour (20min per customer) = 24x 3 = 72 customers per hour

    xeon e3-1270v6
    - each of 8 lanes can process 8 customers per hour (10min per customer) = 8x 8 = 64 customers per hour

    Situation #1
    144 customers come through checkout
    - dual xeon E5-2620 handles those 144 customers in 144/72 = 2hrs
    - xeon e3-1270v6 handles those 144 customers in 144/64 = 2.25hrs

    Situation #2
    6 customers come through checkout each going to one check out lane
    - dual xeon E5-2620 handles those 6 customers @ rate of 20min per customer = within 20 mins
    - xeon e3-1270v6 handles those 6 customers @ rate of 10min per customer = within 10 mins

    FYI, Xeon E3-1270v6 which is ~20-40% faster than Xeon E3-1230v1 is based on Intel Kaby Lake architecture which does more work per cpu clock cycle + is higher clocked than older generation Intel E5-2620 Sandy Bridge-EP based processors.

    More concurrent loads, the dual Xeon E5-2620 will handle better than E3-1270v6, but at lower concurrency levels, E3-1270v6 would be faster.

    From end user perspective if you're not at max all cpu thread utilisation (8 cpu threads for E3-1270v6 and 24 cpu threads for dual Xeon E5-2620), then end users will notice that a forum or web site is faster on E3-1270v6 than on dual Xeon E5-2620.

    As you get closer to maxing out all cpu thread utilisation, then visitors will notice more responsiveness on dual Xeon E5-2620 than E3-1270v6.
     
    • Like Like x 1
    • Informative Informative x 1