Welcome to Centmin Mod Community
Become a Member

PHP-FPM Random 502 bad gateway error with no load at all on the server

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Mar 20, 2017.

  1. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit ?
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: i.e. 1.11.10
    • PHP Version Installed: i.e. 5.6.30 and 7.0.15
    • MariaDB MySQL Version Installed: 10.1.21
    • When was last time updated Centmin Mod code base ? : today

    Hi :)

    I am getting a lot of 502 errors :(

    Server load is very low and i can't see anything to seem related and can cause that :(


    I think there is a bug somewhere .....

    Reading on the net i got some info and i don't know if something is related to Centminmod like:

    The packages setup must be install on some order otherwise can cause that 502

    They solve it by moving some code:

    Some others recommend:

    fastcgi_buffers 16 512k;
    fastcgi_buffer_size 512k;

    Other users recommend to use sockets for connection....

    Another thing is to increase timeout (which one?)

    Some nice info:

    https://www.datadoghq.com/blog/nginx-502-bad-gateway-errors-php-fpm/
    Nginx + PHP-FPM = 502 Bad Gateway

    Sockets are faster than TCP (true?)

    bug in pthreads ?


    From my logs i can see this error:

    Code:
    tail -20 /home/nginx/domains/domain.com/log/error.log
    Code:
    upstream sent unexpected FastCGI record: 3 while reading response header from upstream
    Any ideas?

    Thank you
     
    Last edited: Mar 20, 2017
  2. Sunka

    Sunka Well-Known Member

    1,150
    325
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +525
    Local Time:
    12:00 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    You have installed php version lower than 7.0.16?
     
  3. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    I am using the latest version 7.0.17 and 5.6.30 and i have on both servers this issue :(

    I know that there was a related fix for phpfpm but i am using the latest versions......

    Maybe related to aio threads that i use?

    I really don't know why i am getting this the last 1-2 months :(
     
  4. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Don't know why but there was no erorrs at phpfpm log files and that makes me think that it may be related to Nginx...

    And there the only thing that i have enable as extra is the aio threads....

    I am thinking also that the issue may be related to:

    Code:
    try_files        $uri $uri/ /index.php;
    as the 502 errors most of the times are related to the IPB admin panel that uses links like:

    Code:
    /forum/?page=1
    and may need to use another option like:

    Code:
    try_files $uri $uri/ /index.php?args;
    or

    Code:
    try_files $uri $uri/ /index.php?q=$request_uri;
    or even:

    Code:
    try_files $uri $uri/ /index.php?q=$uri&$args;
    What other ipb 4 users use? @Revenge , @Jimmy ?

    Thanks
     
    Last edited: Mar 20, 2017
  5. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:00 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  6. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Last edited: Mar 20, 2017
  7. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Phpfpm and Nginx debug mode ON :)

    I got some info in any case that anyone has an idea :

    Code:
    2017/03/20 20:09:06 [error] 8745#8745: *9489 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: 131.161.11.204, server: domain.com, request: "GET /profile/125721-zach-binx/?tab=gallery HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "www.domain.com"
    
    2017/03/20 20:09:06 [warn] 8745#8745: *9489 upstream server temporarily disabled while reading response header from upstream, client: 131.161.11.204, server: domain.com, request: "GET /profile/155721-zach-binx/?tab=gallery HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "www.domain.com"
    
    The upstream server temporarily disabled while reading response header
     
    Last edited: Mar 21, 2017
  8. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:00 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    try turning off php-fpm multiple pools and go back to single php-fpm pool for troubleshooting
     
  9. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    I did but nothing.....

    I will plan to downgrade phpfpm and nginx to older versions (2 months ago) that all was ok and see....

    I am sure that it is specific to the server software as i can see the same errors on IPB, Wordpress, Opencart....
     
  10. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    All problems solved by disabling multiple pools :)

    I got tricked the first time when i was disabled them as i got some 502 but now i check and all are related to one simple think so maybe something block it as i thing Centminmod uses a kind of protection for that file as it was used for Ddos attacks:

    Code:
    "POST /xmlrpc.php?for=jetpack&token=ajPjE%40i03yH%21FYYKMoUa%23%2364jVM8Q%24d7%3A1%3A0&timestamp=1490292995&nonce=I3ymrBuB2N&body-hash=K1jPFZ%2BIik361eqqJintTEayeMg%3D&signature=x6W0BUMgDvB4px8WGKWxQJv5mtw%3D HTTP/1.1" 502 166 "https://www.domain.com/xmlrpc.php?for=jetpack&token=ajPjE%40i03yH%21FYYKMoUa%23%2364jVM8Q%24d7%3A1%3A0&timestamp=1490292995&nonce=I3ymrBuB2N&body-hash=K1jPFZ%2BIik361eqqJintTEayeMg%3D&signature=x6W0BUMgDvB4px8WGKWxQJv5mtw%3D" "Jetpack by WordPress.com" "192.0.100.89" rt=0.100 ua="127.0.0.1:9000" us="502" ut="0.100" ul="0" cs=-
    No other 502 errors than that ! :)

    Do i have to edit something related to it?

    Thank you
     
  11. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:00 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    centmin.sh menu option 22 auto wordpress installs disable xmlrpc.php requests via wp plugin by default ;)
     
  12. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    Don't remember if i was use menu 22 for this Wordpress site to be honest so is it normal to get that 502 error and should i ignore it?
     
  13. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:00 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    it's same reason as https://community.centminmod.com/threads/502-error-with-one-user-online.9359/page-2#post-39818

    it's normal in that it's how multiple php-fpm pools is meant to failover to the next pool if php-fpm pools are unavilable
     
  14. pamamolf

    pamamolf Premium Member Premium Member

    4,100
    428
    83
    May 31, 2014
    Ratings:
    +836
    Local Time:
    1:00 AM
    Nginx-1.26.x
    MariaDB 10.6.x
    I do not use anymore multiple pools and i got this one.
     
  15. eva2000

    eva2000 Administrator Staff Member

    55,155
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:00 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    check if xmlrpc.php requests are blocked via wp plugin.. also centmin.sh menu option 22 auto installed wordpress nginx config also rates limits xmlrpc.php requests as well