Join the community today
Register Now

502 error with one user online

Discussion in 'Bug Reports' started by pamamolf, Nov 3, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    k i think i can reproduce the error if you create a dummp 503.php file in web root i.e. public/503.php with just
    Code (Text):
    <?php http_response_code(503) ?>

    and refresh the page at domain.com/503.php - gives 502 bad gateway. Seems it's because php-fpm looks at 503 as php being down which is what 502 bad gateway reports

     
  2. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    Great :)

    Hope that you find a fix for it :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    well haven't figured out the solution but maybe this work around might work, basically rewrite the 502 bad gateways to 503 maintenance page so whenever php is down with 503 error, visits get a nginx maintenance page instead

    you can take advantage of existing include file /usr/local/nginx/conf/503include-main.conf and uncomment it in nginx domain vhost file

    and change it to look like this
    Code (Text):
        # only uncomment if you do not need to exclude images or js
        # from 503 redirect rewrites
         #include /usr/local/nginx/conf/503include-only.conf;
         error_page 503 @maintenance;
         error_page 502 @maintenance502;
         location @maintenance {
              rewrite ^ /maintenance.html break;
         }
         location @maintenance502 {
              rewrite ^ /maintenance502.html break;
         }
    

    then go into public/ directory for domain and make a copy of maintenance page as maintenance502.html
    Code (Text):
    cd /home/nginx/domains/newdomain.com/public
    cp maintenance.html maintenance502.html
    

    so when site and php go into 503 error site unavailable status, static pages returns the maintenance.html page for 503 and maintenance502.html for 502 bad gateway pages.

    Not really a permanent solution but for this issue with multiple php-fpm pools enabled, might do for the time being but that would put entire site into maintenance mode page unaccessible to even admin users heh - unless admins logged in do not get 503 status from php which means admins would still have access

    FYI, the 502 bad gateway comes from multiple php-fpm pools setting for fastcgi_next_upstream which defines which php-fpm error codes will trigger rotating to another php-fpm pool if one is down, i.e. if php-fpm pool2 is down and sees http 503 error returned by forum closure, rotate to pool3 and so forth
    Code (Text):
    fastcgi_next_upstream error timeout http_500 http_503;
    

    if you changed to remove http_503 then it won't rotate on 503 php-fpm errors but that would defeat purpose of multiple php-fpm pools for scaling
    Code (Text):
    fastcgi_next_upstream error timeout http_500;
    


    @Matt might have some ideas ?
     
    Last edited: Nov 11, 2016
  4. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    I will wait for a better fix as that fix makes the error more nice but the error shouldn't exist :)

    but the php is not down or at least shouldn't with no traffic at all :(
     
  5. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    503 status code shown in forum close will be seen by php as site not available and the
    fastcgi_next_upstream setting i mentioned in previous post triggers
     
  6. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    ok got it :)

    Maybe posting at phpfpm support will help?
     
  7. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah when i have more free time will take a closer look at it :)
     
  8. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    Ok take your time :)
     
  9. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    looks like pretty much what i suspected Re: multiple php-fpm pool upstream alternating 503 & 502 errors :)

     
    Last edited: Nov 11, 2016
  10. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    FYI, i've implemented the workaround in post #23 on this forum itself as well as i am using multiple php-fpm pools. Basically, whenever 502 bad gateway errors occur, visitor gets the maintenance page message :)
     
  11. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    I am getting many errors when a new Invision upgrade is out... :-(

    I have to refresh 20 times minimum to get it done....

    And i get then without setting forum as offline....

    Damn there is no a proper solution for this one and go ahead and finish upgrade as it does on Apache server......

    Don't know what other IPB Centminmod users do for this one....
     
  12. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    if it's same issue discussed in this thread then it only happens with multiple php-fpm pools and the work around in this thread maybe used. It's what i use on this forum since it's using multiple php-fpm pools too.
    use this solution quoted link above or just go back to using non-multiple php-fpm pool default setup for the upgrades only or 24/7
     
    Last edited: Feb 4, 2017
  13. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    So there is no way for Multiple pools setup to get that fixed and don't use the workaround?
     
  14. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    no as it's due to how multiple php-fpm pools need to work see https://community.centminmod.com/posts/39826/
     
  15. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    I disable the multiple pools and i have the same issue again:

    Code:
    ;include=/usr/local/nginx/conf/phpfpmd/*.conf
    and then run :

    Code:
    nprestart
     
  16. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    you didn't revert php.conf include file changes as that is where the 502 comes from with regards to fastcgi_next_upstream
     
  17. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    Ok i will :)

    What about nginx config entries:

    Code:
    upstream phpbackend {
    ip_hash;
    keepalive 5;
      server 127.0.0.1:9000 weight=50;
      server 127.0.0.1:9002 weight=50;
      server 127.0.0.1:9003 weight=50;
      server 127.0.0.1:9004 weight=50;
      server 127.0.0.1:9005 weight=50;
    }
    should i revert them also?
     
    Last edited: Feb 6, 2017
  18. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    no to revert though as you edit php.conf include will not reference upstream = phpbackend
     
  19. pamamolf

    pamamolf Well-Known Member

    4,101
    428
    83
    May 31, 2014
    Ratings:
    +838
    Local Time:
    6:11 PM
    Nginx-1.26.x
    MariaDB 10.6.x
    Is there an issue to remove the 'http_503' option from 'proxy_next_upstream' ?

    Code:
    If you remove 'http_503' option from 'proxy_next_upstream' nginx will
    not mark a server in an upstream as failed and you won't get http 502.
    Or we will have other issues or maybe other type of error for the specific one?
     
  20. eva2000

    eva2000 Administrator Staff Member

    55,383
    12,255
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,835
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    that would defeat purpose of multiple php-fpm pools as when php pool is unavilable it won't be able to use the next available php-fpm pool without http_503 defined - so you'd get 502 error anyway on the first unavailable php-fpm pool access