Welcome to Centmin Mod Community
Register Now

502 error with one user online

Discussion in 'Bug Reports' started by pamamolf, Nov 3, 2016.

  1. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hello :)

    I have some steps that you can follow and always get a 502 error.....

    So if you install on Centos 7.x 64bit Wordpress or Invision and set Maintenance mode ON after refreshing main page a few times you will get once result OK and once 502 error...

    Don't know why but i can reproduce it anytime :(

    I use one time installer latest Centminmod beta version....

    What is the special thing i do?

    After installation i am enabling io threads and enable multiple pools....

    fpm settings are reasonable per pool like :

    Code:
    pm = dynamic
    pm.max_children = 70
    pm.start_servers = 10
    pm.min_spare_servers = 5
    pm.max_spare_servers = 68
    pm.max_requests = 4000
    So it will be great if anyone can test it so we may found what is causing this and fix it :)

    If i disable maintenance mode i never get that error even if i got 500 users online....

    Also if i am logged in i do not have that issue ! Only as guest i have the problem....

    Don't know if it is related to any cookies or any cached systems like Memcached or Zend Opcache.....

    Thanks
     
    Last edited: Nov 3, 2016
  2. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
  3. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Yes that's the exact steps that i follow :)
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    there's 2 sets of steps there, list your exact steps done
     
  5. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Post #1
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    i meant exact steps for enabling multiple php-fpm pools
     
  7. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Code:
    nano /usr/local/etc/php-fpm.conf
    uncomment:
    Code:
    include=/usr/local/nginx/conf/phpfpmd/*.conf
    Code:
    nano /usr/local/nginx/conf/nginx.conf
    under http { i add:
    Code:
    upstream phpbackend {
    ip_hash;
    keepalive 5;
      server 127.0.0.1:9000 weight=50;
      server 127.0.0.1:9002 weight=50;
      server 127.0.0.1:9003 weight=50;
      server 127.0.0.1:9004 weight=50;
      server 127.0.0.1:9005 weight=50;
    }
    
    Code:
    nano /usr/local/nginx/conf/php.conf
    comment :
    Code:
    #fastcgi_pass   127.0.0.1:9000;
    under it i add:
    Code:
    fastcgi_next_upstream error timeout http_500 http_503;
    fastcgi_pass phpbackend;
    fastcgi_keep_conn on;
    
    Code:
    nano /usr/local/nginx/conf/phpstatus.conf
    and i add:
    Code:
                location ~ ^/(phpstatus-pool2|phpping-pool2)$ {
                    access_log          off;
                    fastcgi_pass        127.0.0.1:9002;
                    fastcgi_param     SCRIPT_FILENAME $fastcgi_script_name;
                    include          fastcgi_params;
                    allow              127.0.0.1;
                    #allow YOURIPADDRESS;
                    deny                all;
                }
    
                location ~ ^/(phpstatus-pool3|phpping-pool3)$ {
                    access_log          off;
                    fastcgi_pass        127.0.0.1:9003;
                    fastcgi_param     SCRIPT_FILENAME $fastcgi_script_name;
                    include          fastcgi_params;
                    allow              127.0.0.1;
                    #allow YOURIPADDRESS;
                    deny                all;
                }
    
                location ~ ^/(phpstatus-pool4|phpping-pool4)$ {
                    access_log          off;
                    fastcgi_pass        127.0.0.1:9004;
                    fastcgi_param     SCRIPT_FILENAME $fastcgi_script_name;
                    include          fastcgi_params;
                    allow              127.0.0.1;
                    #allow YOURIPADDRESS;
                    deny                all;
                }
    
                location ~ ^/(phpstatus-pool5|phpping-pool5)$ {
                    access_log          off;
                    fastcgi_pass        127.0.0.1:9005;
                    fastcgi_param     SCRIPT_FILENAME $fastcgi_script_name;
                    include          fastcgi_params;
                    allow              127.0.0.1;
                    #allow YOURIPADDRESS;
                    deny                all;
                }
    
    Code:
    nano /usr/local/nginx/conf/conf.d/virtual.conf
    and i uncomment:
    Code:
    include /usr/local/nginx/conf/phpstatus.conf;
    Then i restart phpfpm and Nginx also !
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    for this issue with 502 errors for php-fpm when using multiple php-fpm pools and sitestatus 503 maintenance feature, if you edit /usr/local/nginx/conf/php.conf and change

    from
    Code (Text):
    fastcgi_intercept_errors on;
    

    to
    Code (Text):
    fastcgi_intercept_errors off;
    

    restart nginx and php-fpm
    Code (Text):
    nprestart
    

    does it help or fix the issue ?

    Module ngx_http_fastcgi_module
     
  9. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Nope it doesn't work :(

    When i click after the edit and nprestart to open the page is ok and then when i refresh 502 error and then on refresh ok and the 502 again and so on :(
     
  10. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I disable also the multiple pools:

    Code:
    ;include=/usr/local/nginx/conf/phpfpmd/*.conf
    then nprestart and then again the same issue :(
     
  11. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    what if you edit /usr/local/nginx/conf/503include-main.conf to add
    Code (Text):
    open_file_cache off;
    

    so it becomes
    Code (Text):
        # only uncomment if you do not need to exclude images or js
        # from 503 redirect rewrites
         #include /usr/local/nginx/conf/503include-only.conf;
         error_page 503 @maintenance;
         location @maintenance {
              #if ($maintenance = 1) {
              rewrite ^ /maintenance.html break;
              open_file_cache off;
              #ry_files /maintenance.html =503;
              #}
         }
    

    restart nginx
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    Check your nginx and php error logs for any clues

    Nginx in your access.log and error.log in vhost listed paths in your nginx vhost where domain.com is your domain name at
    Code (Text):
    tail -25 /home/nginx/domains/domain.com/log/access.log
    tail -25 /home/nginx/domains/domain.com/log/error.log
    

    PHP-FPM in
    Code (Text):
    tail -25 /var/log/php-fpm/www-error.log
    

    and/or
    Code (Text):
    tail -25 /var/log/php-fpm/www-php.error.log
    

    tail -25 lists last 25 lines of the log.
     
  13. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I don't use that file as i have commented it:

    Code:
    #include /usr/local/nginx/conf/503include-main.conf;
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    for now but when you had enabled maintenance mode that file was uncommented right ? you need to uncomment 2 files for 503 sitestatus maintenance mode to work a /usr/local/nginx/conf/503include-main.conf and /usr/local/nginx/conf/503include-only.conf include files. From example at Beta Branch - sitestatus maintenance mode | Centmin Mod Community
    Code (Text):
      root /home/nginx/domains/testdomain.com/public;
      include /usr/local/nginx/conf/503include-main.conf;
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
    ...
      }
     
  15. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Code:
    /home/nginx/domains/domain.com/log/access.log
    Code:
    123.456.789.000 - - [10/Nov/2016:04:01:58 +0000] "GET / HTTP/2.0" 503 7077 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 OPR/41.0.2353.46" "-" rt=0.100 ua="127.0.0.1:9003, 127.0.0.1:9004, 127.0.0.1:9005, 127.0.0.1:9002, 127.0.0.1:9000" us="502, 502, 502, 502, 503" ut="0.000, 0.000, 0.000, 0.000, 0.100" ul="0, 0, 0, 0, 6832" cs=-
    123.456.789.000 - - [10/Nov/2016:04:02:02 +0000] "GET / HTTP/2.0" 502 668 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 OPR/41.0.2353.46" "-" rt=0.000 ua="127.0.0.1:9000, phpbackend" us="502, 502" ut="0.000, 0.000" ul="0, 0" cs=-
    123.456.789.000 - - [10/Nov/2016:04:02:03 +0000] "GET / HTTP/2.0" 503 6933 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 OPR/41.0.2353.46" "-" rt=0.000 ua="127.0.0.1:9003, 127.0.0.1:9004, 127.0.0.1:9005, 127.0.0.1:9002, 127.0.0.1:9000" us="502, 502, 502, 502, 503" ut="0.000, 0.000, 0.000, 0.000, 0.000" ul="0, 0, 0, 0, 6837" cs=-
    123.456.789.000 - - [10/Nov/2016:04:02:04 +0000] "GET / HTTP/2.0" 502 668 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 OPR/41.0.2353.46" "-" rt=0.000 ua="127.0.0.1:9000, phpbackend" us="502, 502" ut="0.000, 0.000" ul="0, 0" cs=-
    123.456.789.000 - - [10/Nov/2016:04:02:05 +0000] "GET / HTTP/2.0" 503 6933 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 OPR/41.0.2353.46" "-" rt=0.000 ua="127.0.0.1:9003, 127.0.0.1:9004, 127.0.0.1:9005, 127.0.0.1:9002, 127.0.0.1:9000" us="502, 502, 502, 502, 503" ut="0.000, 0.000, 0.000, 0.000, 0.000" ul="0, 0, 0, 0, 6837" cs=

    Code:
    /home/nginx/domains/domain.com/log/error.log

    Code:
    2016/11/10 04:02:04 [error] 12003#12003: *1 no live upstreams while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET / HTTP/2.0", upstream: "fastcgi://phpbackend", host: "domain.com"
    2016/11/10 04:02:05 [error] 12003#12003: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET / HTTP/2.0", upstream: "fastcgi://127.0.0.1:9003", host: "domain.com"
    2016/11/10 04:02:05 [error] 12003#12003: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET / HTTP/2.0", upstream: "fastcgi://127.0.0.1:9004", host: "domain.com"
    2016/11/10 04:02:05 [error] 12003#12003: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET / HTTP/2.0", upstream: "fastcgi://127.0.0.1:9005", host: "domain.com"
    2016/11/10 04:02:05 [error] 12003#12003: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET / HTTP/2.0", upstream: "fastcgi://127.0.0.1:9002", host: "domain.com"
    2016/11/10 04:02:06 [error] 12004#12004: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET /?app=core&module=system&controller=ajax&do=instantNotifications&csrfKey=aa7d5381d26b1fcbe0bbda3d8da4b96c&notifications=0&messages=0 HTTP/2.0", upstream: "fastcgi://127.0.0.1:9003", host: "domain.com", referrer: "https://domain.com/"
    2016/11/10 04:02:06 [error] 12004#12004: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET /?app=core&module=system&controller=ajax&do=instantNotifications&csrfKey=aa7d5381d26b1fcbe0bbda3d8da4b96c&notifications=0&messages=0 HTTP/2.0", upstream: "fastcgi://127.0.0.1:9004", host: "domain.com", referrer: "https://domain.com/"
    2016/11/10 04:02:06 [error] 12004#12004: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET /?app=core&module=system&controller=ajax&do=instantNotifications&csrfKey=aa7d5381d26b1fcbe0bbda3d8da4b96c&notifications=0&messages=0 HTTP/2.0", upstream: "fastcgi://127.0.0.1:9005", host: "domain.com", referrer: "https://domain.com/"
    2016/11/10 04:02:06 [error] 12004#12004: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 123.456.789.000, server: domain.com, request: "GET /?app=core&module=system&controller=ajax&do=instantNotifications&csrfKey=aa7d5381d26b1fcbe0bbda3d8da4b96c&notifications=0&messages=0 HTTP/2.0", upstream: "fastcgi://127.0.0.1:9002", host: "domain.com", referrer: "https://domain.com/"
    

    Code:
    /var/log/php-fpm/www-error.log
    empty

    Code:
    /var/log/php-fpm/www-php.error.log
    empty
     
  16. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I am not talking about Centminmod maintenance mode and i have commented all lines related to that file......

    I am talking about Invision set forum as offline and maintenance mode for Wordpress....

    Or do you mean something else?
     
  17. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    looks like pools 2-5 the additional pools aren't responding or refusing connections/down so give 502
    oh i was talking about sitestatus 503 maintenance mode heh
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,195
    6,789
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,144
    Local Time:
    2:12 PM
    Nginx 1.13.x
    MariaDB 5.5
    what does your Invision board nginx vhost look like for domain.com.conf and/or domain.com.ssl.conf ?
     
  19. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I can get the same issue even with multiple pools disabled by:

    Code:
    #include /usr/local/nginx/conf/503include-main.conf;
     
  20. pamamolf

    pamamolf Well-Known Member

    2,726
    243
    63
    May 31, 2014
    Ratings:
    +434
    Local Time:
    7:12 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Code:
    #x# HTTPS-DEFAULT
     server {
     
       server_name domain.com www.domain.com;
       return 302 https://$server_name$request_uri;
       include /usr/local/nginx/conf/staticfiles-hsts.conf;
     }
    
    server {
      listen 443 ssl http2;
      server_name domain.com www.domain.com;
    
      include /usr/local/nginx/conf/ssl/domain.com/domain.com.crt.key.conf;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      http2_max_field_size 16k;
      http2_max_header_size 32k;
      # mozilla recommended
      ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
    
      # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
      add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
      #add_header X-Frame-Options SAMEORIGIN;
      #add_header X-Xss-Protection "1; mode=block" always;
      #add_header X-Content-Type-Options "nosniff" always;
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
     
      # enable ocsp stapling
      resolver 8.8.8.8 8.8.4.4 valid=10m;
      resolver_timeout 10s;
      ssl_stapling on;
      ssl_stapling_verify on;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/domain.com/log/access.log main_ext buffer=256k flush=60m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf;
      root /home/nginx/domains/domain.com/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      #include /usr/local/nginx/conf/503include-main.conf;
    
      location / {
      #include /usr/local/nginx/conf/503include-only.conf;
    
    # block common exploits, sql injections etc
    include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Enable for vBulletin usage WITHOUT vbSEO installed
      try_files    $uri $uri/ /index.php;
    
      }
    
      include /usr/local/nginx/conf/staticfiles-hsts.conf;
      #include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    But it is very easy to produce it by installing Centminmod 09 beta and then wordpress and then set it to maintenance mode and check.......

    As you may be able to find the cause more easy if you can produce it yourself.... ?

    The problem exist on http also ! Tested on another server with Wordpress and http....