Nginx PHP-FPM Some strange rate limiting

• CentOS Version: CentOS 7 64bit
• Centmin Mod Version Installed: 123.09beta01
• Nginx Version Installed: 1.17.1 (latest)
• PHP Version Installed: 7.3.7
• MariaDB MySQL Version Installed: 10.3.16
• When was last time updated Centmin Mod code base ? : yesterday
• Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
  
  Code (Text):

  LETSENCRYPT_DETECT='y'
  

I read pretty much all there is about rate nginx rate limiting, both here on SO and most of the links that seemed relevant to me. Still, I cannot solve the issue.

Background: I switched from VPS to dedicated. The dedicated has like 2x the resources of the VPS so by transferring most of the settings I assumed that I should be fine. Alas, this is not the case and apparently I am hitting a severe rate limit that causes many frequent visitors to my forum to complain, I can notice it as well.

The lowest level I can replicate the issue is at php status page via lynx in the console:

Code (Text):

lynx --dump  http://127.0.0.1/phpstatus

---

If I do that really fast, within a minute it stops giving results. I actually noticed that when I ran the netdata monitoring, which for some reason did not give me phpfpm stats. When I debugged it as per the author instructions, this is what I got:

Code (Text):

2019-07-08 16:09:57: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
2019-07-08 16:09:59: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))

So this leads me to believe that either nginx and/or php-fpm are rate limited and even localhost requests are blocked by it.

When I restart nginx it works briefly for few seconds, then again the same error. Full log after nginx is restarted:

Code:

[16:09][netdata@concorde nginx]$ /usr/libexec/netdata/plugins.d/python.d.plugin debug 1 phpfpm
2019-07-08 16:12:30: python.d INFO: plugin[main] : using python v2
2019-07-08 16:12:30: python.d INFO: plugin[main] : starting setup
2019-07-08 16:12:30: python.d INFO: plugin[main] : checking for config in ['/etc/netdata', '/usr/lib/netdata/conf.d']
2019-07-08 16:12:30: python.d INFO: plugin[main] : config found, loading config '/usr/lib/netdata/conf.d/python.d.conf'
2019-07-08 16:12:30: python.d INFO: plugin[main] : config successfully loaded
2019-07-08 16:12:30: python.d INFO: plugin[main] : starting checker process (1 module(s) to check)
2019-07-08 16:12:30: python.d INFO: plugin[checker] : starting...
2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : checking
2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : source successfully loaded
2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : found config file '/etc/netdata/python.d/phpfpm.conf'
2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm : created 1 job(s) from the config
2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
2019-07-08 16:12:30: python.d INFO: plugin[checker] : phpfpm[local] : check successful
2019-07-08 16:12:30: python.d INFO: plugin[checker] : terminating...
2019-07-08 16:12:30: python.d INFO: plugin[main] : stopping checker process
2019-07-08 16:12:30: python.d INFO: plugin[main] : setup complete, 1 active module(s) : '['phpfpm']'
2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm : created 1 job(s)
2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : urllib3 version: 1.21.1
2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : init successful
2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : check successful
CHART netdata.runtime_phpfpm_local '' 'Execution time for phpfpm_local' 'ms' 'python.d' netdata.pythond_runtime line 145000 1
DIMENSION run_time 'run time' absolute 1 1

2019-07-08 16:12:30: python.d INFO: plugin[main] : phpfpm[local] : started in thread
2019-07-08 16:12:30: python.d DEBUG: phpfpm[local] : started, update frequency: 1
CHART phpfpm_local.connections '' 'PHP-FPM Active Connections' 'connections' 'active connections' 'phpfpm.connections' line 60000 1 '' 'python.d.plugin' 'phpfpm'
DIMENSION 'active' 'active' absolute 1 1 ' '
DIMENSION 'maxActive' 'max active' absolute 1 1 ' '
DIMENSION 'idle' 'idle' absolute 1 1 ' '

BEGIN phpfpm_local.connections 0
SET 'active' = 5
SET 'maxActive' = 42
SET 'idle' = 11
END

CHART phpfpm_local.performance '' 'PHP-FPM Performance' 'status' 'performance' 'phpfpm.performance' line 60002 1 '' 'python.d.plugin' 'phpfpm'
DIMENSION 'reached' 'max children reached' absolute 1 1 ' '
DIMENSION 'slow' 'slow requests' absolute 1 1 ' '

BEGIN phpfpm_local.performance 0
SET 'reached' = 1
SET 'slow' = 0
END

CHART phpfpm_local.requests '' 'PHP-FPM Requests' 'requests/s' 'requests' 'phpfpm.requests' line 60001 1 '' 'python.d.plugin' 'phpfpm'
DIMENSION 'requests' 'requests' incremental 1 1 ' '

BEGIN phpfpm_local.requests 0
SET 'requests' = 7077
END

BEGIN netdata.runtime_phpfpm_local 0
SET run_time = 3
END

2019-07-08 16:12:31: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 3, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000133
SET 'active' = 6
SET 'maxActive' = 42
SET 'idle' = 10
END

BEGIN phpfpm_local.performance 1000133
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000133
SET 'requests' = 7080
END

BEGIN netdata.runtime_phpfpm_local 1000133
SET run_time = 2
END

2019-07-08 16:12:32: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000042
SET 'active' = 9
SET 'maxActive' = 42
SET 'idle' = 7
END

BEGIN phpfpm_local.performance 1000042
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000042
SET 'requests' = 7085
END

BEGIN netdata.runtime_phpfpm_local 1000042
SET run_time = 2
END

2019-07-08 16:12:33: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999927
SET 'active' = 7
SET 'maxActive' = 42
SET 'idle' = 9
END

BEGIN phpfpm_local.performance 999927
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999927
SET 'requests' = 7087
END

BEGIN netdata.runtime_phpfpm_local 999927
SET run_time = 2
END

2019-07-08 16:12:34: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000054
SET 'active' = 17
SET 'maxActive' = 42
SET 'idle' = 0
END

BEGIN phpfpm_local.performance 1000054
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000054
SET 'requests' = 7099
END

BEGIN netdata.runtime_phpfpm_local 1000054
SET run_time = 2
END

2019-07-08 16:12:35: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999990
SET 'active' = 11
SET 'maxActive' = 42
SET 'idle' = 5
END

BEGIN phpfpm_local.performance 999990
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999990
SET 'requests' = 7104
END

BEGIN netdata.runtime_phpfpm_local 999990
SET run_time = 2
END

2019-07-08 16:12:36: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000015
SET 'active' = 12
SET 'maxActive' = 42
SET 'idle' = 4
END

BEGIN phpfpm_local.performance 1000015
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000015
SET 'requests' = 7106
END

BEGIN netdata.runtime_phpfpm_local 1000015
SET run_time = 2
END

2019-07-08 16:12:37: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999943
SET 'active' = 12
SET 'maxActive' = 42
SET 'idle' = 4
END

BEGIN phpfpm_local.performance 999943
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999943
SET 'requests' = 7108
END

BEGIN netdata.runtime_phpfpm_local 999943
SET run_time = 2
END

2019-07-08 16:12:38: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000041
SET 'active' = 14
SET 'maxActive' = 42
SET 'idle' = 2
END

BEGIN phpfpm_local.performance 1000041
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000041
SET 'requests' = 7111
END

BEGIN netdata.runtime_phpfpm_local 1000041
SET run_time = 2
END

2019-07-08 16:12:39: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999998
SET 'active' = 13
SET 'maxActive' = 42
SET 'idle' = 3
END

BEGIN phpfpm_local.performance 999998
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999998
SET 'requests' = 7113
END

BEGIN netdata.runtime_phpfpm_local 999998
SET run_time = 2
END

2019-07-08 16:12:40: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_duration' was suppressed due to non updating
2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_mem' was suppressed due to non updating
2019-07-08 16:12:41: python.d ERROR: phpfpm[local] : chart 'phpfpm_local.request_cpu' was suppressed due to non updating
BEGIN phpfpm_local.connections 999941
SET 'active' = 14
SET 'maxActive' = 42
SET 'idle' = 2
END

BEGIN phpfpm_local.performance 999941
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999941
SET 'requests' = 7115
END

BEGIN netdata.runtime_phpfpm_local 999941
SET run_time = 2
END

2019-07-08 16:12:41: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000089
SET 'active' = 15
SET 'maxActive' = 42
SET 'idle' = 1
END

BEGIN phpfpm_local.performance 1000089
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000089
SET 'requests' = 7118
END

BEGIN netdata.runtime_phpfpm_local 1000089
SET run_time = 2
END

2019-07-08 16:12:42: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999980
SET 'active' = 15
SET 'maxActive' = 42
SET 'idle' = 1
END

BEGIN phpfpm_local.performance 999980
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999980
SET 'requests' = 7120
END

BEGIN netdata.runtime_phpfpm_local 999980
SET run_time = 2
END

2019-07-08 16:12:43: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999999
SET 'active' = 14
SET 'maxActive' = 42
SET 'idle' = 2
END

BEGIN phpfpm_local.performance 999999
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999999
SET 'requests' = 7122
END

BEGIN netdata.runtime_phpfpm_local 999999
SET run_time = 2
END

2019-07-08 16:12:44: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000020
SET 'active' = 15
SET 'maxActive' = 42
SET 'idle' = 1
END

BEGIN phpfpm_local.performance 1000020
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000020
SET 'requests' = 7124
END

BEGIN netdata.runtime_phpfpm_local 1000020
SET run_time = 2
END

2019-07-08 16:12:45: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 999139
SET 'active' = 17
SET 'maxActive' = 42
SET 'idle' = 0
END

BEGIN phpfpm_local.performance 999139
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 999139
SET 'requests' = 7128
END

BEGIN netdata.runtime_phpfpm_local 999139
SET run_time = 2
END

2019-07-08 16:12:46: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000829
SET 'active' = 17
SET 'maxActive' = 42
SET 'idle' = 0
END

BEGIN phpfpm_local.performance 1000829
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000829
SET 'requests' = 7133
END

BEGIN netdata.runtime_phpfpm_local 1000829
SET run_time = 2
END

2019-07-08 16:12:47: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
BEGIN phpfpm_local.connections 1000020
SET 'active' = 17
SET 'maxActive' = 42
SET 'idle' = 0
END

BEGIN phpfpm_local.performance 1000020
SET 'reached' = 1
SET 'slow' = 0
END

BEGIN phpfpm_local.requests 1000020
SET 'requests' = 7136
END

BEGIN netdata.runtime_phpfpm_local 1000020
SET run_time = 2
END

2019-07-08 16:12:48: python.d DEBUG: phpfpm[local] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
2019-07-08 16:12:51: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))
2019-07-08 16:12:51: python.d DEBUG: phpfpm[local] : get_data() returned no data
2019-07-08 16:12:51: python.d DEBUG: phpfpm[local] : update => [FAILED] (elapsed time: -, failed retries in a row: 1)
2019-07-08 16:12:54: python.d ERROR: phpfpm[local] : Url: http://localhost/phpstatus. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /phpstatus (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=80): Read timed out. (read timeout=1)",))
2019-07-08 16:12:54: python.d DEBUG: phpfpm[local] : get_data() returned no data
2019-07-08 16:12:54: python.d DEBUG: phpfpm[local] : update => [FAILED] (elapsed time: -, failed retries in a row: 2)
^CTraceback (most recent call last):
  File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 733, in <module>
    main()
  File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 727, in main
    plugin.run()
  File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 462, in run
    self.serve()
  File "/usr/libexec/netdata/plugins.d/python.d.plugin", line 619, in serve
    time.sleep(1)
KeyboardInterrupt

Any help is appreciated!

 

my nginx.conf

Code (Text):

user              nginx nginx;
worker_processes 16;
worker_cpu_affinity auto;
worker_priority -10;

worker_rlimit_nofile 520000;
timer_resolution 100ms;

pcre_jit on;
include /usr/local/nginx/conf/dynamic-modules.conf;


pid         logs/nginx.pid;

events {
    worker_connections  80000;
    accept_mutex off;
    accept_mutex_delay 200ms;
    use epoll;
    #multi_accept on;
}

http {
 map_hash_bucket_size 128;
 map_hash_max_size 4096;
 server_names_hash_bucket_size 128;
 server_names_hash_max_size 2048;
 variables_hash_max_size 2048;

 resolver 127.0.0.1 valid=10m;
 resolver_timeout 10s;
#limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=200r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;

# sets Centmin Mod headers via headers more nginx module
# https://github.com/openresty/headers-more-nginx-module
# don't remove the first 2 lines as centmin mod checks to see if they're
# missing and re-adds them anyway. Just uncomment the 3rd & 4th lines
# which is used to override the Server header to what you want = nginx
# and remove the X-Powered-By header + restart nginx service
# do not disable headers more nginx module itself as it's required for
# other centmin mod features like redis nginx level caching & letsencrypt
# integration in vhosts created by addons/acmetool.sh
more_set_headers "Server: nginx centminmod";
more_set_headers "X-Powered-By: centminmod";
#more_set_headers "Server: nginx";
#more_clear_headers "X-Powered-By";

# uncomment cloudflare.conf include if using cloudflare for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfcf.sh auto
# run the auto command once to populate cloudflare ips
include /usr/local/nginx/conf/cloudflare.conf;
# uncomment incapsula.conf include if using incapsula for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfincapsula.sh auto
# run the auto command once to popular incapsula ips
#include /usr/local/nginx/conf/incapsula.conf;
include /usr/local/nginx/conf/maintenance.conf;
#include /usr/local/nginx/conf/vts_http.conf;
include /usr/local/nginx/conf/geoip.conf;
include /usr/local/nginx/conf/webp.conf;
include /usr/local/nginx/conf/ssl_include.conf;
#include /usr/local/nginx/conf/pagespeedadmin.conf;
include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
include /usr/local/nginx/conf/default_phpupstream.conf;

log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  ddos-proxy '$remote_addr for $http_x_real_ip - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$host" sn="$server_name" '
                      'rt=$request_time '
                      'ua="$upstream_addr" us="$upstream_status" '
                      'ut="$upstream_response_time" ul="$upstream_response_length" '
                      'cs=$upstream_cache_status' ;

# only uncomment include line to enable it you have enabled ngx_brotli module
# which is disabled by default https://community.centminmod.com/threads/10688/
#include /usr/local/nginx/conf/log_format_brotli.conf;

access_log  off;
error_log   logs/error.log warn;

    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

sendfile on;
sendfile_max_chunk 512k;
tcp_nopush  on;
tcp_nodelay on;
server_tokens off;
server_name_in_redirect off;

keepalive_timeout  5s;
keepalive_requests  1000;
lingering_time 20s;
lingering_timeout 5s;
keepalive_disable msie6;

gzip on;
gzip_vary   on;
gzip_disable msie6;
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

client_body_buffer_size 256k;
client_body_in_file_only off;
client_body_timeout 10s;
client_header_buffer_size 64k;
## how long a connection has to complete sending
## it's headers for request to be processed
client_header_timeout  10s;
client_max_body_size 1024m;
connection_pool_size  512;
directio  4m;
directio_alignment 4096;
ignore_invalid_headers on;
large_client_header_buffers 8 64k;
output_buffers   1 512k;
postpone_output  1460;
proxy_temp_path  /tmp/nginx_proxy/;
request_pool_size  32k;
reset_timedout_connection on;
send_timeout     60s;
types_hash_max_size 2048;

# for nginx proxy backends to prevent redirects to backend port
# port_in_redirect off;

open_file_cache max=50000 inactive=60s;
open_file_cache_valid 120s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
open_log_file_cache max=10000 inactive=30s min_uses=2;

## limit number of concurrency connections per ip to 16
## add to your server {} section the next line
## limit_conn limit_per_ip 16;
## uncomment below line allows 500K sessions
# limit_conn_log_level error;
#######################################
# use limit_zone for Nginx <v1.1.7 and lower
# limit_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################
# use limit_conn_zone for Nginx >v1.1.8 and higher
# limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################

 include /usr/local/nginx/conf/conf.d/*.conf;
}

my phpfpm.conf

Code (Text):

[global]
; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = warning
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/www-error.log
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
;include=/usr/local/nginx/conf/phpfpmd/*.conf

[www]
user = nginx
group = nginx

listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
listen.backlog = 8151

;listen = /tmp/php5-fpm.sock
listen.owner = nginx
listen.group = nginx
listen.mode = 0660

pm = dynamic
pm.max_children = 16
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 10
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 200

; PHP 5.3.9 setting
; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
pm.process_idle_timeout = 10s;

rlimit_files = 65536
rlimit_core = 0
; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0
; Default Value: 0
;request_slowlog_timeout = 0
slowlog = /var/log/php-fpm/www-slow.log

pm.status_path = /phpstatus
ping.path = /phpping
ping.response = pong

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
security.limit_extensions = .php

; catch_workers_output = yes
php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
php_admin_value[disable_functions] = shell_exec,exec,system,passthru,pcntl_exec,popen,proc_open

 

I just used their one line installer, no idea how it setups the webserver :/

Well, weird thing is that nginx stats work without any issues and they use similar approach - accessing and parsing the status page. Also, I haven't modified any netdata settings about rate limit, so it is weird that netdata itself will limit its own requests in the default settings

Anyway, I am puzzled, I will remove netdata and see if it solves the issue. I fear that netdata symptoms could be red herring, stopping the netdata service definitely did not help.

 

Yeap, doing

Code (Text):

 lynx --dump  http://localhost/nginx_status

many times has absolutely no problem. Phpfpm must be the bottleneck. But my settings are not that unreasonable and my server is practically idle. There are no errors in the log since yesterday when I upgraded to 7.3.7 due to bug giving segmentation faults. Now back to 7.2.x, but issue is not resolved. I will play with the children and processes and see what is going on. This migration has been a nightmare

 

OK, this is getting more and more confusing. Take a look at my php-fpm stats (I have to wait like 20 seconds for this to show up):

Code (Text):

Processes active: 21, idle: 0, Requests: 1040, slow: 0, Traffic: 1.3req/sec
pool:                 www
process manager:      dynamic
start time:           08/Jul/2019:17:24:28 +0000
start since:          537
accepted conn:        1065
listen queue:         2
max listen queue:     5
listen queue len:     511
idle processes:       0
active processes:     23
total processes:      23
max active processes: 31
max children reached: 1
slow requests:        0

So after start up I reach max children almost immediately, in the current case it is 20. But the requests themselves are low number - 1000 requests for around 10 min, thats like 2req/s on average. I think much lower amount of php-fpm processes should be able to handle this number of requests...and yet, almost immediately after start all children are busy and the status page starts lagging...

Sorry for rambling, I find that it kinda helps me when I think out loud

 

OK, just for testing I bumped max.children from 20 to 40. Just for testing, my server can take it (32 cores, although the model is 6y old). The issue disappeared and here are the results at the same timestamp as above:

Code (Text):

Processes active: 24, idle: 2, Requests: 1414, slow: 0, Traffic: 1.9req/sec
pool:                 www
process manager:      dynamic
start time:           08/Jul/2019:17:39:40 +0000
start since:          537
accepted conn:        1435
listen queue:         0
max listen queue:     1
listen queue len:     511
idle processes:       1
active processes:     25
total processes:      26
max active processes: 39
max children reached: 0
slow requests:        0

So almost 50% more requests handled and there is no lag at the moment. The server is still not loaded, so I can afford to leave it like that. Still the interesting questions are:

- Such amount of requests should be handled with lower amount of processes, no? Like 3 requests per second does not seem high to me.
- Why on my previous VPS the same traffic was handled with lower amount of resources and processes?

 

Well, here are the stats from my previous VPS:

Code (Text):

pool:                 www
process manager:      dynamic
start time:           16/May/2019:09:19:18 +0000
start since:          4612387
accepted conn:        7248947
listen queue:         0
max listen queue:     129
listen queue len:     128
idle processes:       9
active processes:     1
total processes:      10
max active processes: 20
max children reached: 26
slow requests:        0

I guess the explanation that the VPS was able to better process the requests due to its high-end CPU is plausible. Maybe thats why I have to increase the children amount on the dedi - to take advantage of the more cores at a lower processing power.

VPS: 4x Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz (4 cores each, cache size: 8192 KB) vs
Dedi: 4x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz (8 cores each, cache size: 20480 KB)

You learn something new every day