Get the most out of your Centmin Mod LEMP stack
Become a Member

Nginx 504 Gateway Timeout error

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by danielsl, May 19, 2016.

  1. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    Error 502 after adding new domain and editing redirects in conf

    hi
    encountered the same behaviour
    logs
    error access emty
    but from time to to time error.log
    Code:
    2016/05/18 09:15:41 [error] 21060#21060: *103947 open() "/home/nginx/domains/e36v.ru/public/assets/components/bootstrap/dist/js/bootstrap.min.js" failed (2: No such file or directory), client: 128.71.178.200, server: e36v.ru, request: "GET /assets/components/bootstrap/dist/js/bootstrap.min.js HTTP/1.1", host: "e36v.ru", referrer: "http://e36v.ru/"
    2016/05/18 09:15:48 [error] 21060#21060: *103943 open() "/home/nginx/domains/e36v.ru/public/assets/components/bootstrap/dist/js/bootstrap.min.js" failed (2: No such file or directory), client: 128.71.178.200, server: e36v.ru, request: "GET /assets/components/bootstrap/dist/js/bootstrap.min.js HTTP/1.1", host: "e36v.ru", referrer: "http://e36v.ru/index.php?id=10"
    2016/05/18 09:45:24 [error] 22224#22224: *44 open() "/home/nginx/domains/e36v.ru/public/assets/components/bootstrap/dist/js/bootstrap.min.js" failed (2: No such file or directory), client: 37.9.122.201, server: e36v.ru, request: "GET /assets/components/bootstrap/dist/js/bootstrap.min.js HTTP/1.1", host: "e36v.ru"
    2016/05/18 10:34:59 [error] 22224#22224: *74 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 50.121.214.104, server: e36v.ru, request: "GET /index.php?id=10 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "e36v.ru"
    2016/05/18 10:35:40 [error] 22224#22224: *78 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 50.121.214.104, server: e36v.ru, request: "GET /index.php?id=12 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "e36v.ru"
    2016/05/18 10:36:33 [error] 22224#22224: *80 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 50.121.214.104, server: e36v.ru, request: "GET /index.php?id=9 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "e36v.ru"
    2016/05/18 12:14:18 [notice] 22224#22224: *189 "^/(.*)$" matches "/admin/", client: 89.21.79.152, server: e36v.ru, request: "GET /admin/ HTTP/1.1", host: "e36v.ru"
    
    09 BETA
    PHP7.0.1
    CENTMIN32B 6.5

    all worked just fine for a year until i added new domain and started editing conf redirects rules
    and after few
    Code:
    service nginx restart;s
    all my domains become unavailable except
    mysql works and accessible remotely
    static files are served
    node and jetty proxied just fine
    but no response from php scripts
    i tryed to restart php-fpm(no errors)
    i updated downgraded php version,
    deleted domain conf, restarted and even tryed to create it again via sh console but it doesnt create this domain conf anymore despite sh console running without errors.
    Code:
    tryed yum update
    but i get
    Code:
    1716 packages excluded due to repository priority protections
    and randomly (after nginx restart)i get some domains work some not even made on the same cms
     
    Last edited: May 19, 2016
  2. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    Is that the new domain with new rewrites added ? try disable that domain temporarily by backing up and removing the nginx vhost files for them at /usr/local/nginx/conf/conf.d/e36v.ru.conf and /usr/local/nginx/conf/conf.d/e36v.ru.ssl.conf and restart nginx and php-fpm and see

    When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)
    • Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf
    • Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf
    • Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com
    • Vhost public web root will be at /home/nginx/domains/newdomain.com/public
    • Vhost log directory will be at /home/nginx/domains/newdomain.com/log
    Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)
     
  3. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    fyi, visited the site i see 504 not 502 which is a timeout error due to php-fpm most likely with incorrect rewrite rules and/or your web app/php script setup
     
  4. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    MODX CMF Manager Login i have 502
    but on 4steps.ru i have 504
    Daniel Ko Photography proxied to jetty just fine

    this one i never edited since making it running under nginx
    Code:
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    #server {
    #            listen   80;
    #            server_name 4steps.ru;
    #            return 301 $scheme://www.4steps.ru$request_uri;
    #       }
    
    server {
      server_name 4steps.ru www.4steps.ru;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/4steps.ru/log/access.log combined buffer=32k;
      error_log /home/nginx/domains/4steps.ru/log/error.log;
    
      root /home/nginx/domains/4steps.ru/public;
    
    if (!-e $request_filename){
                            rewrite ^/(.*)$ /index.php?q=$1 last;
                }
    
      location / {
    
    # block common exploits, sql injections etc
    #include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Enable for vBulletin usage WITHOUT vbSEO installed
      #try_files        $uri $uri/ /index.php;
    
      }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      include /usr/local/nginx/conf/errorpage.conf;
    }
    
    e36v
    edited but then returned on default to the state it worked before the system ruined
    Code:
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    #server {
    #            listen   80;
    #            server_name e36v.ru;
    #            return 301 $scheme://www.e36v.ru$request_uri;
    #       }
    
    server {
        listen   80;
            server_name e36v.ru;
        rewrite_log on;
        index    index.php index.html;
        root /home/nginx/domains/e36v.ru/public;
    
    
    #    return 301 $scheme://e36v.ru$request_uri;
    #ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/e36v.ru/log/access.log combined buffer=32k;
      error_log /home/nginx/domains/e36v.ru/log/error.log notice;
    
    
        # редирект с ww на без-www
    #   if ($host != 'e36v.ru' ) {
    #       rewrite  ^/(.*)$  http://e36v.ru/$1  permanent;
    #  }
    #location ~ ^/(.*?)/index.html$ {
    #        rewrite            ^/(.*?)/ /$1/ permanent;
    #    } 
        # если нет запрошенного файла - слать в локацию rewrite
        location / {
            root /home/nginx/domains/e36v.ru/public;
            try_files    $uri $uri/ @rewrite;
        }
    
        # отсюда слать всё на index.php - это есть быть friendly urls
        location @rewrite {
            rewrite        ^/(.*)$ /index.php?q=$1;
        }
    
        # а вот и наша защита системных директорий по ip
    #    location ~* ^\/(manager|core|connectors)\/(?:.*)$ {
    #        allow        184.55.68.97;
    #        deny        all;
    #    location ~* \.php$ {
    #           include        fastcgi_params;
    #            fastcgi_param    SCRIPT_FILENAME $document_root$fastcgi_script_name;
    #            fastcgi_pass    127.0.0.1:9000;
    #        }
    #    }
    
        # а это выполнение всех php файлов, не попавших в предыдущий блок
        location ~* \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
    
        #limit_req    zone=one  burst=3;
            include        fastcgi_params;
            fastcgi_param    SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_pass   127.0.0.1:9000;
        #include fastcgi_params;
            fastcgi_ignore_client_abort on;
            fastcgi_param  SERVER_NAME $http_host; 
        }
    
        # статику отдаём без участия MODX и php
        location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|bmp|woff|eot)$ {
           access_log        off;
           expires            10d;
           break;
        }
    
        # на всякий случай запрещаем чтение .htaccess - вдруг у вас там еще и апач есть?
        location ~ /\.ht {
            deny            all;
        }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      include /usr/local/nginx/conf/errorpage.conf;
    }
    
    Also i even deleted in from conf folder, centmin.sh couldnot recreat it again via 2 menu, the folder simply didnt have it after adding

    no ssl

    i tryed disabling all except one or two old domains but it didnt make any change.

    just after countless nginx restart e36v.ru started working but others not
     
    Last edited: May 19, 2016
  5. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    you have to restart both nginx and php-fpm as 504 timeouts is php-fpm related and nginx con't connect/timed out
    Code (Text):
    nprestart
     
  6. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    you also have duplicate php location matches like
    Code (Text):
        # а это выполнение всех php файлов, не попавших в предыдущий блок
        location ~* \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
    
        #limit_req    zone=one  burst=3;
            include        fastcgi_params;
            fastcgi_param    SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_pass   127.0.0.1:9000;
        #include fastcgi_params;
            fastcgi_ignore_client_abort on;
            fastcgi_param  SERVER_NAME $http_host;
        }

    you don't need them as centmin mod auto generated nginx vhosts have their own match in include file for php.conf
    Code (Text):
      include /usr/local/nginx/conf/php.conf;
     
  7. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    i deleted all ..conf ..except those mentioned 3 domains
    i deleted duplicated location as you mentioned(but it worked in the past flawlessly) in e36v.ru
    i nprestartED

    but my old domain no help
    4steps.ru
    timeout

    ----------------------
    also when this problem first arose. i checked Htop processess. centmin.sh has been taking 30% cpu for 528Hours, and i terminated it and restarted vps
     
  8. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    Might try nginx debug mode to dig deeper

    Latest update on September 14, 2015 to Centmin Mod 1.2.3-eva2000.08 stable (123.08stable) and higher has added Nginx debug mode support. Nginx debug mode is disabled by default.

    To enable, edit centmin.sh and change variable NGINX_DEBUG=n default to NGINX_DEBUG=y and recompile Nginx via centmin.sh menu option 4 and then setting error_log in nginx vhosts to debug as outlined at nginx.org/en/docs/debugging_log.html & wiki.nginx.org/Debugging.

    You do not want to leave Nginx debug mode running forever, so after debugging, set NGINX_DEBUG=n in centmin.sh and recompile Nginx again via centmin.sh menu option 4 and remove error_log debugging mode to disable Nginx debug mode again.

    To update your Centmin Mod builds code for Nginx debug mode support if you do not have a NGINX_DEBUG variable in centmin.sh, follow instructions at centminmod.com/upgrade.html and respective version threads below:

    Centmin Mod is provided as is, but you can try debugging mode for Nginx for further troubleshooting if you have problems with Nginx (i.e. segfaults / signal 11 issues) as outlined at nginx.org/en/docs/debugging_log.html & wiki.nginx.org/Debugging.
     
  9. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    Also anything in /home/nginx/domains/4steps.ru/log/error.log ?
     
  10. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    very strange in the log request to different domain name 9ah.ru which i also have
    Code:
    2016/05/18 23:49:52 [error] 23421#23421: *30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "9ah.ru"
    2016/05/18 23:50:07 [error] 23421#23421: *41 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 128.71.178.231, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "4steps.ru"
    2016/05/18 23:50:12 [error] 23421#23421: *43 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "9ah.ru"
    2016/05/18 23:53:22 [error] 23421#23421: *51 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 128.71.178.231, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "4steps.ru"
    also when i open new web link via google chrome it sometimes redirect me to previous adrress, i tryed on different PCs it sometimes happens(all right after this misfortunate accident)

    ----
    recompiled with option 4
    added
    Code:
      error_log /home/nginx/domains/4steps.ru/log/error.log;
    error_log /home/nginx/domains/4steps.ru/log/errord.log debug;
    
    but
    service nginx-debug start
    -->unrecognized service
     
  11. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    Just nginx service restart not nginx-debug
     
  12. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    clear your web browser caches and try in incognito mode
    what does your 9ah.ru nginx vhost conf files look like ?
     
  13. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    9ah.ru conf is not in conf folder now for debugging purposes
    Code:
    # Centmin Mod Getting Started Guide
    # must read http://centminmod.com/getstarted.html
    
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    # if unsure use return 302 before using return 301
    #server {
    #            listen   80;
    #            server_name 9ah.ru;
    #            return 301 $scheme://www.9ah.ru$request_uri;
    #       }
    
    server {
    
      server_name 9ah.ru www.9ah.ru;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      #add_header X-Frame-Options SAMEORIGIN;
      #add_header X-Xss-Protection "1; mode=block" always;
      #add_header X-Content-Type-Options "nosniff" always;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/9ah.ru/log/access.log combined buffer=256k flush=60m;
      error_log /home/nginx/domains/9ah.ru/log/error.log;
    
      root /home/nginx/domains/9ah.ru/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      include /usr/local/nginx/conf/wpincludes/9ah.ru/wpcacheenabler_9ah.ru.conf;
      #include /usr/local/nginx/conf/wpincludes/9ah.ru/wpsupercache_9ah.ru.conf;
      # https://community.centminmod.com/posts/18828/
      #include /usr/local/nginx/conf/wpincludes/9ah.ru/rediscache_9ah.ru.conf;
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
    
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # for wordpress super cache plugin
      #try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;
    
      # for wp cache enabler plugin
      try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args;  
    
      # Wordpress Permalinks
      #try_files $uri $uri/ /index.php?q=$uri&$args;
    
      # Nginx level redis Wordpress
      # https://community.centminmod.com/posts/18828/
      #try_files $uri $uri/ /index.php?$args;
    
      }
    
    location ~* /(wp-login\.php) {
        limit_req zone=xwplogin burst=1 nodelay;
        #limit_conn xwpconlimit 30;
        #auth_basic "Private";
        #auth_basic_user_file /home/nginx/domains/9ah.ru/htpasswd_wplogin;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /(xmlrpc\.php) {
        limit_req zone=xwprpc burst=45 nodelay;
        #limit_conn xwpconlimit 30;
        include /usr/local/nginx/conf/php-wpsc.conf;
        # https://community.centminmod.com/posts/18828/
        #include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
      include /usr/local/nginx/conf/wpincludes/9ah.ru/wpsecure_9ah.ru.conf;
      include /usr/local/nginx/conf/php-wpsc.conf;
      # https://community.centminmod.com/posts/18828/
      #include /usr/local/nginx/conf/php-rediscache.conf;
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    
    Also I just opened iexplorer inprivate mode without result
    nginx restarted
    ----------
    error debug output is here also
    /wp-cron.php is from 9ah.ru or icholding.ru(also deleted for testing purposes) wordpress installation. it should not be in 4steps.ru error debug path. i.m shocked
    Code:
    2016/05/19 00:16:00 [notice] 25983#25983: *5 "^/(.*)$" matches "/shop/feed/", client: 66.249.66.98, server: 4steps.ru, request: "GET /shop/feed/?product_tag=tee HTTP/1.1", host: "9ah.ru"
    2016/05/19 00:16:00 [notice] 25983#25983: *5 rewritten data: "/index.php", args: "q=shop/feed/&product_tag=tee", client: 66.249.66.98, server: 4steps.ru, request: "GET /shop/feed/?product_tag=tee HTTP/1.1", host: "9ah.ru"
    2016/05/19 00:16:23 [info] 25983#25983: *7 client 5.143.231.133 closed keepalive connection
    2016/05/19 00:16:52 [notice] 25983#25983: *8 "^/(.*)$" matches "/wp-cron.php", client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", host: "9ah.ru"
    2016/05/19 00:16:52 [notice] 25983#25983: *8 rewritten data: "/index.php", args: "q=wp-cron.php&doing_wp_cron=1", client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", host: "9ah.ru"
    2016/05/19 00:17:12 [notice] 25983#25983: *10 "^/(.*)$" matches "/wp-cron.php", client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", host: "9ah.ru"
    2016/05/19 00:17:12 [notice] 25983#25983: *10 rewritten data: "/index.php", args: "q=wp-cron.php&doing_wp_cron=1", client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", host: "9ah.ru"
    2016/05/19 00:19:00 [info] 25983#25983: *5 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 66.249.66.98, server: 4steps.ru, request: "GET /shop/feed/?product_tag=tee HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "9ah.ru"
    2016/05/19 00:19:52 [error] 25983#25983: *8 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "9ah.ru"
    2016/05/19 00:19:52 [info] 25983#25983: *8 client 185.83.218.184 closed keepalive connection
    2016/05/19 00:20:12 [error] 25983#25983: *10 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 185.83.218.184, server: 4steps.ru, request: "GET /wp-cron.php?doing_wp_cron=1 HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "9ah.ru"
    2016/05/19 00:20:12 [info] 25983#25983: *10 client 185.83.218.184 closed keepalive connection
    2016/05/19 00:20:35 [error] 25983#25983: *12 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 178.79.138.229, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.4steps.ru"
    2016/05/19 00:20:35 [info] 25983#25983: *12 client 178.79.138.229 closed keepalive connection
    2016/05/19 00:20:57 [error] 25983#25983: *14 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 178.79.138.229, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.4steps.ru"
    2016/05/19 00:21:20 [info] 26197#26197: *3 client closed connection while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:21:20 [info] 26197#26197: *5 client closed connection while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:21:20 [info] 26197#26197: *4 client closed connection while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:22:47 [notice] 26197#26197: *6 "^/(.*)$" matches "/xmlrpc.php", client: 86.105.50.40, server: 4steps.ru, request: "POST /xmlrpc.php HTTP/1.1", host: "icholding.ru"
    2016/05/19 00:22:47 [notice] 26197#26197: *6 rewritten data: "/index.php", args: "q=xmlrpc.php", client: 86.105.50.40, server: 4steps.ru, request: "POST /xmlrpc.php HTTP/1.1", host: "icholding.ru"
    2016/05/19 00:22:55 [info] 26197#26197: *6 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 86.105.50.40, server: 4steps.ru, request: "POST /xmlrpc.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "icholding.ru"
    2016/05/19 00:24:01 [error] 26197#26197: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 128.71.178.231, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "4steps.ru"
    2016/05/19 00:25:07 [notice] 26197#26197: *8 "^/(.*)$" matches "/cond_calculate", client: 180.76.15.150, server: 4steps.ru, request: "GET /cond_calculate HTTP/1.1", host: "icholding.ru"
    2016/05/19 00:25:07 [notice] 26197#26197: *8 rewritten data: "/index.php", args: "q=cond_calculate", client: 180.76.15.150, server: 4steps.ru, request: "GET /cond_calculate HTTP/1.1", host: "icholding.ru"
    2016/05/19 00:26:01 [info] 26197#26197: *11 client closed connection while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:26:01 [info] 26197#26197: *13 client closed connection while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:28:07 [error] 26197#26197: *8 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 180.76.15.150, server: 4steps.ru, request: "GET /cond_calculate HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "icholding.ru"
    2016/05/19 00:28:07 [info] 26197#26197: *8 recv() failed (104: Connection reset by peer) while sending response to client, client: 180.76.15.150, server: 4steps.ru, request: "GET /cond_calculate HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "icholding.ru"
    2016/05/19 00:28:32 [info] 26197#26197: *16 client timed out (110: Connection timed out) while waiting for request, client: 128.71.178.231, server: 0.0.0.0:80
    2016/05/19 00:28:41 [error] 26197#26197: *10 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 128.71.178.231, server: 4steps.ru, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "4steps.ru"
    
    it seems i need to reinstall server and all addons
     
  14. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah you'd need to troubelshoot your site vhosts and your nginx rewrite rules as alot of them are related to 4steps.ru before the connection timesout
     
  15. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    i never changed any rules in vhost since it worked for a year. i have this random strange problem just after adding new domain, it seems configuration got corrupted somewhere else. thanks anyway. is it possible to reinstall clean delete and fresh install of centminmod without domain directories deleting. i have db backups
     
  16. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    best to backup all site data and vhost files and then reload CentOS OS clean and install fresh centmin mod 123.09beta01 (latest improvements) Beta Branch - Centmin Mod .09 beta branch Testing | Centmin Mod Community

    You can do this if you want by setting up a new server i.e. VPS and install on that, then slowly move data from old to new server. This gives you chance to compare before and after configuration/settings etc
     
  17. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    Hi again.
    after reinstalling centos 6.7 min+centminmod 09 beta01+few of web domain--> got back to the same issue.
    random dissapearance. 500 error
    Code:
     tail -f /var/log/php-fpm/www-error.log
    [21-May-2016 14:38:03] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [21-May-2016 14:41:10] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [21-May-2016 14:42:54] WARNING: [pool www] server reached max_children setting (4), consider raising it
    
    not sure if its a problem as i had this present config for a year
    Code:
    var/log/nginx
    
    tail -f /var/log/php-fpm/www-error.log
    
    2016/05/21 03:39:25 [error] 25390#25390: *5733 open() "/usr/local/nginx/html/contacts.html" failed (2: No such file or directory), client: 144.76.61.21, server: pan.0dan.ru, request: "GET /contacts.html HTTP/1.0", host: "f0ta.ru"
    2016/05/21 03:39:26 [error] 25390#25390: *5734 open() "/usr/local/nginx/html/portfolio-album.html" failed (2: No such file or directory), client: 144.76.61.21, server: pan.0dan.ru, request: "GET /portfolio-album.html HTTP/1.0", host: "f0ta.ru"
    2016/05/21 03:39:28 [error] 25390#25390: *5735 open() "/usr/local/nginx/html/replica.html" failed (2: No such file or directory), client: 144.76.61.21, server: pan.0dan.ru, request: "GET /replica.html HTTP/1.0", host: "f0ta.ru"
    2016/05/21 04:08:23 [error] 25390#25390: *5908 open() "/usr/local/nginx/html/sitemap.xml" failed (2: No such file or directory), client: 66.249.66.27, server: pan.0dan.ru, request: "GET /sitemap.xml HTTP/1.1", host: "f.0dan.ru"
    2016/05/21 05:07:52 [error] 25390#25390: *6261 open() "/usr/local/nginx/html/minimal-base/tours/extreme.html" failed (2: No such file or directory), client: 207.46.13.190, server: pan.0dan.ru, request: "GET /minimal-base/tours/extreme.html HTTP/1.1", host: "www.go2ch.ru"
    2016/05/21 05:14:03 [error] 25390#25390: *6301 open() "/usr/local/nginx/html/vip.html" failed (2: No such file or directory), client: 220.181.108.151, server: pan.0dan.ru, request: "GET /vip.html?color=27&dir=asc&occasion=31&order=name HTTP/1.1", host: "magento.0dan.ru"
    2016/05/21 06:27:52 [error] 25390#25390: *6737 open() "/usr/local/nginx/html/img/portfolio/money.jpg" failed (2: No such file or directory), client: 5.255.253.86, server: pan.0dan.ru, request: "GET /img/portfolio/money.jpg HTTP/1.1", host: "f0ta.ru"
    2016/05/21 06:29:52 [error] 25390#25390: *6748 open() "/usr/local/nginx/html/img/portfolio/food3.jpg" failed (2: No such file or directory), client: 141.8.183.19, server: pan.0dan.ru, request: "GET /img/portfolio/food3.jpg HTTP/1.1", host: "f0ta.ru"
    2016/05/21 06:54:41 [error] 25390#25390: *6894 "/usr/local/nginx/html/catalog/product_compare/add/product/425/uenc/aHR0cDovL21hZ2VudG8uMGRhbi5ydS8,/form_key/4x9fwz0dzllULYtP/index.php" is not found (2: No such file or directory), client: 123.125.71.14, server: pan.0dan.ru, request: "GET /catalog/product_compare/add/product/425/uenc/aHR0cDovL21hZ2VudG8uMGRhbi5ydS8,/form_key/4x9fwz0dzllULYtP/ HTTP/1.1", host: "magento.0dan.ru"
    2016/05/21 07:30:12 [error] 25390#25390: *7103 "/usr/local/nginx/html/all-products/index.php" is not found (2: No such file or directory), client: 157.55.39.203, server: pan.0dan.ru, request: "GET /all-products/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:30:12 [error] 25390#25390: *7103 "/usr/local/nginx/html/shop-contact/index.php" is not found (2: No such file or directory), client: 157.55.39.203, server: pan.0dan.ru, request: "GET /shop-contact/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:30:12 [error] 25390#25390: *7103 "/usr/local/nginx/html/blog/index.php" is not found (2: No such file or directory), client: 157.55.39.203, server: pan.0dan.ru, request: "GET /blog/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:30:16 [error] 25390#25390: *7106 "/usr/local/nginx/html/123/duis-ornare-at-lobortis/index.php" is not found (2: No such file or directory), client: 207.46.13.90, server: pan.0dan.ru, request: "GET /123/duis-ornare-at-lobortis/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:30:16 [error] 25390#25390: *7106 "/usr/local/nginx/html/login/index.php" is not found (2: No such file or directory), client: 207.46.13.90, server: pan.0dan.ru, request: "GET /login/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:30:16 [error] 25390#25390: *7106 "/usr/local/nginx/html/15346/hitrices-orci-leo-dolor/index.php" is not found (2: No such file or directory), client: 207.46.13.90, server: pan.0dan.ru, request: "GET /15346/hitrices-orci-leo-dolor/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:31:08 [error] 25390#25390: *7110 "/usr/local/nginx/html/sample-page/index.php" is not found (2: No such file or directory), client: 207.46.13.136, server: pan.0dan.ru, request: "GET /sample-page/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:31:08 [error] 25390#25390: *7110 "/usr/local/nginx/html/shop-testimonials/index.php" is not found (2: No such file or directory), client: 207.46.13.136, server: pan.0dan.ru, request: "GET /shop-testimonials/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:31:09 [error] 25390#25390: *7110 "/usr/local/nginx/html/date/2012/09/19/index.php" is not found (2: No such file or directory), client: 207.46.13.136, server: pan.0dan.ru, request: "GET /date/2012/09/19/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:36:33 [error] 25390#25390: *7145 "/usr/local/nginx/html/all-products/terms-and-conditions/index.php" is not found (2: No such file or directory), client: 157.55.39.237, server: pan.0dan.ru, request: "GET /all-products/terms-and-conditions/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:36:33 [error] 25390#25390: *7145 "/usr/local/nginx/html/15138/luna-ipsum-nulla-dolor/index.php" is not found (2: No such file or directory), client: 157.55.39.237, server: pan.0dan.ru, request: "GET /15138/luna-ipsum-nulla-dolor/ HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:40:37 [error] 25390#25390: *7167 open() "/usr/local/nginx/html/wp-includes/wlwmanifest.xml" failed (2: No such file or directory), client: 5.255.253.67, server: pan.0dan.ru, request: "GET /wp-includes/wlwmanifest.xml HTTP/1.1", host: "9ah.ru"
    2016/05/21 07:42:16 [error] 25390#25390: *7179 open() "/usr/local/nginx/html/сервисное-обслуживание" failed (2: No such file or directory), client: 144.76.8.132, server: pan.0dan.ru, request: "GET /%D1%81%D0%B5%D1%80%D0%B2%D0%B8%D1%81%D0%BD%D0%BE%D0%B5-%D0%BE%D0%B1%D1%81%D0%BB%D1%83%D0%B6%D0%B8%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5 HTTP/1.0", host: "icholding.ru"
    2016/05/21 07:42:19 [error] 25390#25390: *7180 open() "/usr/local/nginx/html/install" failed (2: No such file or directory), client: 144.76.8.132, server: pan.0dan.ru, request: "GET /install HTTP/1.0", host: "icholding.ru"
    2016/05/21 07:42:20 [error] 25390#25390: *7181 open() "/usr/local/nginx/html/services.html" failed (2: No such file or directory), client: 144.76.8.132, server: pan.0dan.ru, request: "GET /services.html HTTP/1.0", host: "icholding.ru"
    2016/05/21 08:04:54 [error] 25390#25390: *7323 open() "/usr/local/nginx/html/img/portfolio/book_cover.jpg" failed (2: No such file or directory), client: 66.249.66.73, server: pan.0dan.ru, request: "GET /img/portfolio/book_cover.jpg HTTP/1.1", host: "f0ta.ru"
    2016/05/21 08:30:45 [error] 25390#25390: *7333 open() "/usr/local/nginx/html/2November2014MartialArt" failed (2: No such file or directory), client: 89.163.148.58, server: pan.0dan.ru, request: "GET /2November2014MartialArt HTTP/1.0", host: "f.0dan.ru"
    2016/05/21 08:30:47 [error] 25390#25390: *7334 open() "/usr/local/nginx/html/3011282" failed (2: No such file or directory), client: 89.163.148.58, server: pan.0dan.ru, request: "GET /3011282 HTTP/1.0", host: "f.0dan.ru"
    
    
    
    
    servers and hosts are mixed somehow-> this log really puzzles me

    I usually ssh and use mc to edit files via putty
    at the same time sftp via far manager to edit files
    and tunel mysql via plink to 3307:127.0.0.1:3306 to update upload mysql dumps

    use openVZ vps
     
  18. eva2000

    eva2000 Administrator Staff Member

    28,968
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,758
    Local Time:
    3:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    ah that explains it means php-fpm is overloaded and crashes and those = 504 gateway timeouts

    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Nginx 502 or 504 Bad Gateway Errors



    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children | Centmin Mod Community and PHP-FPM - WARNING: [pool www] server reached max_children setting (50), consider raising it | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
    tail -10 /usr/local/nginx/logs/access.log
    tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
    tail -10 /home/nginx/domains/domainname.com/log/access.log
    tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
    ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root root 0 Aug 29 15:33 tallylog
    -rw------- 1 root root 0 Aug 29 15:33 spooler
    drwx------ 3 root root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root 500 0 Oct 8 18:13 dmesg.old
    -rw------- 1 root 500 0 Oct 8 18:13 boot.log
    -rw-r--r-- 1 root 500 0 Oct 8 18:14 dmesg
    drwx------ 2 root root 4.0K Oct 8 18:14 httpd
    drwxr-xr-x 2 root root 4.0K Oct 8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct 9 12:38 mysqld.log
    -rw------- 1 root root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root utmp 94K Nov 7 22:59 btmp
    drwxr-xr-x 2 root root 4.0K Nov 8 00:00 sa
    -rw------- 1 root root 269K Nov 8 21:39 messages
    -rw------- 1 root root 110K Nov 8 23:08 secure
    -rw-rw-r-- 1 root utmp 43K Nov 8 23:08 wtmp
    -rw-r--r-- 1 root root 144K Nov 8 23:08 lastlog
    -rw------- 1 root root 69K Nov 8 23:08 lfd.log
    -rw------- 1 root root 332K Nov 8 23:08 maillog
    -rw------- 1 root 500 1.6M Nov 8 23:10 cron
    
    For PHP-FPM error log:
    Code:
    tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
    /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
    tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
    tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
    tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
    tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
    • Like Like x 1
  19. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    that;s more then i expected .
    Thank you very Much
     
    • Like Like x 1
  20. danielsl

    danielsl New Member

    20
    2
    3
    Nov 7, 2014
    Ratings:
    +3
    Local Time:
    8:22 AM
    centminmod 123.09beta01
    5.5
    but in this case , why another web name works?????
    icholding.ru the one i set the last. I thought everything should go down. I.m i mistaken