Welcome to Centmin Mod Community
Register Now

Nginx PHP-FPM upstream prematurely closed connection while reading response header from upstream

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by ezak, Jun 21, 2020.

  1. ezak

    ezak New Member

    5
    0
    1
    May 26, 2018
    Ratings:
    +1
    Local Time:
    9:05 AM
    1.13.12
    5.5
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit ?
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.19.0
    • PHP Version Installed: 7.3.19
    • MariaDB MySQL Version Installed: 10.3.23-MariaDB
    • When was last time updated Centmin Mod code base ? : just now
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      cat /etc/centminmod/custom_config.inc
      [/LIST]
      LETSENCRYPT_DETECT='y'
      MARCH_TARGETNATIVE='n'
      



      I have installed 10 centmin server, all server same config just change domain name, and all serving download files with secure encrypted PHP code link .. all is working fine, but this server have a strange problem, and I don't know what and where is the problem?


      the server response is very slow
      here is error_log
      Code:
      2020/06/21 03:28:01 [error] 36244#36244: *125318 readv() failed (104: Connection reset by peer) while reading upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:30:11 [error] 36245#36245: *125406 upstream prematurely closed connection while reading response header from upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:30:18 [error] 36245#36245: *125414 upstream prematurely closed connection while reading response header from upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:30:37 [error] 36245#36245: *125438 upstream prematurely closed connection while reading response header from upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:32:19 [error] 36244#36244: *125950 readv() failed (104: Connection reset by peer) while reading upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:37:01 [error] 36244#36244: *127632 readv() failed (104: Connection reset by peer) while reading upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:37:17 [error] 36244#36244: *127640 upstream prematurely closed connection while reading response header from upstream, client: 94.26.205.210, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:38:07 [error] 36245#36245: *127664 upstream prematurely closed connection while reading response header from upstream, client: 136.228.175.87, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      2020/06/21 03:38:07 [error] 36245#36245: *127675 upstream prematurely closed connection while reading response header from upstream, client: 136.228.175.87, server: servx.domain.com, request: "GET /d.php?id=xxx HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "servx.domain.com", referrer: "https://servx.domain.com/d.php?id=xxx"
      
      after enable debug in error_log

      Code:
      2020/06/21 09:24:22 [info] 53165#53165: *226 client 110.139.23.194 closed keepalive connection
      2020/06/21 09:24:24 [info] 53165#53165: *229 client 110.139.23.194 closed keepalive connection
      2020/06/21 09:24:25 [info] 53166#53166: *260 client 110.139.23.194 closed keepalive connection
      2020/06/21 09:24:36 [info] 53168#53168: *268 client timed out (110: Connection timed out) while SSL handshaking, client: 110.139.23.194, server: 0.0.0.0:443
      2020/06/21 09:24:37 [info] 53165#53165: *267 client timed out (110: Connection timed out) while waiting for request, client: 110.139.23.194, server: 0.0.0.0:443
      2020/06/21 09:24:46 [info] 53164#53164: *281 client 110.139.23.194 closed keepalive connection
      2020/06/21 09:25:02 [info] 53165#53165: *317 client 110.139.23.194 closed keepalive connection
      2020/06/21 09:25:02 [info] 53166#53166: *323 client canceled stream 1 while sending response to client, client: 165.56.182.10, server: servx.domain.com, request: "GET /d.php?id=xxx"
      2020/06/21 09:25:02 [info] 53166#53166: *323 client prematurely closed connection while sending response to client, client: 165.56.182.10, server: servx.domain.com, request: "GET /d.php?id=xxx"
      2020/06/21 09:25:02 [info] 53168#53168: *78 client prematurely closed connection while sending response to client, client: 103.111.225.155, server: servx.domain.com, request: "GET /d.php?id=xxx"
      2020/06/21 09:25:08 [info] 53164#53164: *248 client prematurely closed connection (32: Broken pipe) while sending response to client, client: 41.102.93.114, server: servx.domain.com, request: "GET /d.php?id=xxx"
      
     
  2. eva2000

    eva2000 Administrator Staff Member

    50,891
    11,793
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,248
    Local Time:
    5:05 PM
    Nginx 1.25.x
    MariaDB 10.x
    Your php issues most likely are due to PHP-FPM needing tuning. So best to start read thoroughly the guide at PHP-FPM - How to troubleshoot & optimize PHP-FPM server? :) If you're using default /usr/local/nginx/conf/php.conf include file in Centmin Mod generated Nginx vhosts, you can tune it via command shortcut = phpinc which opens nano editor for file /usr/local/nginx/conf/php.conf which has fastcgi connect, send and read timeouts. Though raising them too high just masks the issue, you should ideally not use PHP to serve large download files themselves but code PHP to tell Nginx to serve the file via X-Accel / X-Sendfile headers X-Accel | NGINX. Example https://stackoverflow.com/questions/16189758/serve-large-file-with-php-and-nginx-x-accel-redirect
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable PHP-FPM status page output outlined at
    Enabling PHP-FPM status also allows setting up 3rd party PHP-FPM status metric monitoring from services like:

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    50,891
    11,793
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,248
    Local Time:
    5:05 PM
    Nginx 1.25.x
    MariaDB 10.x
  4. ezak

    ezak New Member

    5
    0
    1
    May 26, 2018
    Ratings:
    +1
    Local Time:
    9:05 AM
    1.13.12
    5.5
    thanks for your reply sir, also I want to mention that I already use
    X-Accel-Redirect in all this server which using centminmod all is working fine without any problem
    also, many of them have a heavy load, these servers didn't have any problem with default config in centminmod

    this server the only one have this problem, also there is no error in phpfpm error_log
    the connection is very slow to any request is always disconnected
    I thought there is a problem in Hardware Network in the node server (its dedicated server), but I'm not sure how to check that
     
  5. eva2000

    eva2000 Administrator Staff Member

    50,891
    11,793
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,248
    Local Time:
    5:05 PM
    Nginx 1.25.x
    MariaDB 10.x
    all servers with same web host ? same hardware specs ? can you identify the IP addresses for the ones with slowness ? any users you can discuss with that report such issues ? could be that the visitor's connection is slow on their end so download time is slower too - as a download is only as fast as it's slowest side (visitor or server).
     
  6. ezak

    ezak New Member

    5
    0
    1
    May 26, 2018
    Ratings:
    +1
    Local Time:
    9:05 AM
    1.13.12
    5.5
    I have requested new server, and move all data .. it's working fine for now
    thanks for your interesting