/ Register

Welcome to Centmin Mod Community
Become a Member

Nginx Cloudflare: Why does one NGINX worker take all the load?

Discussion in 'Nginx and PHP-FPM news & discussions' started by pamamolf, Oct 24, 2017.

Tags:
Previous Thread Next Thread
Loading...
  1. Oct 24, 2017 #1
    pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:49 AM
    Nginx-1.25.x
    MariaDB 10.3.x
  2. Oct 24, 2017 #2
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Centmin Mod 123.09beta01 and higher already uses nginx reuseport to balance nginx worker load if Centmin Mod detects server's kernel has SO_REUSEPORT support (non-openvz based servers generally with 2.6.32+ higher kernels). Was added back in May 2015 !

    See
    upload_2017-10-24_15-40-53.png
    You can check via ss command whether reuseport is working. If centmin mod at initial install time detects your 2.6.32+ linux kernel for non-openvz systems supports SO_REUSEPORT it would add reuseport option to your main default hostname's listen directive in it's nginx vhost at /usr/local/nginx/conf/conf.d/virtual.conf.
    Code (Text):
    listen 80 default_server backlog=2048 reuseport fastopen=256;

    Here centmin mod detected system support for SO_REUSEPORT and TCP FastOpen so automatically added reuseport and fastopen options. You only need to list reuseport once per ip:port listen pairing so only once in one vhost at /usr/local/nginx/conf/conf.d/virtual.conf. If you have nginx vhosts listening on a 2nd dedicated IP instead of main server IP, then you manually add reuseport to that listen directive usually.

    Then output of below command should list multiple *:80 and if HTTPS is in use *:443 listings one for each nginx worker you have if reuseport is working
    Code (Text):
    ss -lnt | egrep -e ':80 |:443 '

    example on 8 cpu thread server with 4 cpu workers set in nginx.conf (worker_processes 4)
    Code (Text):
    ss -lnt | egrep -e ':80 |:443 '
LISTEN     0      2048         *:80                       *:*             
LISTEN     0      2048         *:80                       *:*             
LISTEN     0      2048         *:80                       *:*             
LISTEN     0      2048         *:80                       *:*             
LISTEN     0      2048         *:443                      *:*             
LISTEN     0      2048         *:443                      *:*             
LISTEN     0      2048         *:443                      *:*             
LISTEN     0      2048         *:443                      *:*

    4 sets of 80 and 4 sets of 443 listening pairs one for each nginx worker process
     
    Last edited: Oct 24, 2017
  3. Oct 31, 2017 #3
    Colin

    Colin Premium Member Premium Member

    191
    59
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +153
    Local Time:
    3:49 AM
    1.19.#
    MariaDB 10.1.#
    Interesting. I think I've just made some work for myself though as I did the same command to compare and I have 8 workers enabled, 7 are on 80 and one on 443.

    Problem is, the only site on the box is ssl. That said it seems to be fine, but perhaps not; hence the work for myself to discover why, or I'm missing something obvious :D
     
  4. Oct 31, 2017 #4
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Oh you may need to on one HTTPS based nginx vhost add to listen directive the reuseport + backlog directive - only need to do this for a single HTTPS nginx vhost
    Code (Text):
     listen 443 ssl http2 backlog=2048 reuseport;

    Centmin Mod only auto adds reuseport on non-HTTPS default main vhost config file at /usr/local/nginx/conf/conf.d/virtual.conf
     
  5. Oct 31, 2017 #5
    Colin

    Colin Premium Member Premium Member

    191
    59
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +153
    Local Time:
    3:49 AM
    1.19.#
    MariaDB 10.1.#
    Super. Will test, and report back :)
     
  6. Oct 31, 2017 #6
    pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:49 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Didn't know about that manual edit also :)

    Let's hope it will work well with https also :)

    George do you plan to add reuseport support out of the box for https also?
     
  7. Oct 31, 2017 #7
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    no plans as if can only be added for one nginx HTTPS vhost and not all of them so if you have multiple nginx HTTPS vhosts, only one needs reuseport. Maybe in future if/when i main main hostname's vhost.conf also support HTTPS by default.
     
  8. Oct 31, 2017 #8
    pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:49 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    On one of my servers i got this error:

    Code:
    On one of my servers i got this error:

[CODE]-- Unit nginx.service has begun starting up.
Oct 30 22:33:08 server.mydomain.com nginx[17311]: Starting nginx: nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/test.mydomain.com.ssl.conf:11
Oct 30 22:33:08 server.mydomain.com nginx[17311]: [FAILED]
Oct 30 22:33:08 server.mydomain.com systemd[1]: nginx.service: control process exited, code=exited status=1
Oct 30 22:33:08 server.mydomain.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
-- Subject: Unit nginx.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit nginx.service has failed.
--
-- The result is failed.
Oct 30 22:33:08 server.mydomain.com systemd[1]: Unit nginx.service entered failed state.
Oct 30 22:33:08 server.mydomain.com systemd[1]: nginx.service failed.
Oct 30 22:33:08 server.mydomain.com polkitd[676]: Unregistered Authentication Agent for unix-process:17305:68303076 (system bus name :1.37902, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
    But i can't find any duplicated lines on config line 11:

    Code:
    # redirect from www to non-www  forced SSL
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
# server {
#       listen   80;
#       server_name test.mydomain.com www.test.mydomain.com;
#       return 302 https://$server_name$request_uri;
# }

server {
  listen 443 ssl http2 backlog=2048 reuseport;
  server_name test.mydomain.com www.test.mydomain.com;

  ssl_dhparam /usr/local/nginx/conf/ssl/test.mydomain.com/dhparam.pem;
  ssl_certificate      /usr/local/nginx/conf/ssl/test.mydomain.com/test.mydomain.com.crt;
  ssl_certificate_key  /usr/local/nginx/conf/ssl/test.mydomain.com/test.mydomain.com.key;
  include /usr/local/nginx/conf/ssl_include.conf;

  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  #add_header X-Xss-Protection "1; mode=block" always;
  #add_header X-Content-Type-Options "nosniff" always;
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;
  #ssl_trusted_certificate /usr/local/nginx/conf/ssl/test.mydomain.com/test.mydomain.com-trusted.crt;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/test.mydomain.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/test.mydomain.com/log/error.log;

  #include /usr/local/nginx/conf/autoprotect/test.mydomain.com/autoprotect-test.mydomain.com.conf;
  root /home/nginx/domains/test.mydomain.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  include /usr/local/nginx/conf/cloudflare.conf;
  #include /usr/local/nginx/conf/503include-main.conf;

  location / {
  #include /usr/local/nginx/conf/503include-only.conf;

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Wordpress Permalinks example
  try_files $uri $uri/ /index.php?q=$uri&$args;

  }

  include /usr/local/nginx/conf/pre-staticfiles-local-test.mydomain.com.conf;
  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
 
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}
     
    Last edited: Oct 31, 2017
  9. Oct 31, 2017 #9
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    3:49 AM
    1.9.x
    10.1.x
    I have reuseport in one of my openvz vps's and its working. It's not only on non openvz system.
     
  10. Oct 31, 2017 #10
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    guess it depends on the host node kernel support :)

    output for command
    Code (Text):
    egrep -wrn 'backlog|reuseport' /usr/local/nginx/conf/conf.d/

    example
    Code (Text):
    egrep -wrn 'backlog|reuseport' /usr/local/nginx/conf/conf.d/
/usr/local/nginx/conf/conf.d/domain.com.ssl.conf:15:  listen 443 ssl http2 backlog=2048 reuseport;
/usr/local/nginx/conf/conf.d/virtual.conf:3:            listen   80 default_server backlog=2048 reuseport;
     
  11. Oct 31, 2017 #11
    pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:49 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Code:
    /usr/local/nginx/conf/conf.d/mydomain.com.ssl.conf:11:  listen 443 ssl http2 backlog=2048 reuseport;
/usr/local/nginx/conf/conf.d/test.mydomain.com.ssl.conf:11:  listen 443 ssl http2 backlog=2048 reuseport;
/usr/local/nginx/conf/conf.d/anotherdomain.com.ssl.conf:11:  listen 443 ssl http2 backlog=2048 reuseport;
/usr/local/nginx/conf/conf.d/virtual.conf:3:            listen   80 default_server backlog=2048 reuseport fastopen=256;
    It is a dedicated server :)
     
  12. Oct 31, 2017 #12
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    see you have more than one HTTPS nginx vhost with reuseport which is a no no - only one vhost is allowed to use the directive per ip:port pairing. So mainserverip:80 and mainserverip:443 only one nginx vhost for port 80 and one for port 443 on entire server needs reuseport/backlog directives. You have 3x HTTPS nginx vhosts with the directives.
     
  13. Feb 15, 2022 #13
    cloud9

    cloud9 Premium Member Premium Member

    431
    117
    43
    Oct 6, 2015
    England
    Ratings:
    +217
    Local Time:
    3:49 AM
    1.25.3
    10.6.x
    @eva2000 With respect to this - if you have more than one vhost does it matter which one you pick for reuseport ? or do you pick the busiest traffic site ?
     
  14. Feb 16, 2022 #14
    eva2000

    eva2000 Administrator Staff Member

    53,223
    12,116
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,654
    Local Time:
    12:49 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    No it doesn't matter. Centmin Mod by default will only add reuseport to the first Nginx vhost added via centmin.sh menu option 2, 22 or nv or addons/acmetool.sh commandline and auto detect on subsequent Nginx vhost creation if reuseport already exists.
     
Previous Thread Next Thread
Loading...

.