Get the most out of your Centmin Mod LEMP stack
Become a Member

Beta Branch error upon nv vhost creation - related reuseports

Discussion in 'Beta release code' started by MaxFein, Jul 30, 2018.

  1. MaxFein

    MaxFein New Member

    13
    2
    3
    Jul 9, 2018
    Ratings:
    +5
    Local Time:
    4:54 PM
    Please fill in any relevant information that applies to you:
    • CentOS Version:CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: cmm latest
    • PHP Version Installed: cmm latest
    • MariaDB MySQL Version Installed: 10,2 cmm latest
    • When was last time updated Centmin Mod code base ? : today
    • Persistent Config: yes - see link to my setup doc at bitbucket
    Code:
    nv -d tst0.xyz -s lelived
    ERROR nginx does not restart after vhost creation (before LE)

    ...now, I did so some slightly funky stuff in messing with the default virtual.conf (mostly, replaced it with virtual.ssl.conf) ...however, it seems like this is an issue with reuseport (when at other than :80)... removing reuseport from tst0.xyz.ssl.conf makes it work (oddly, removing reuseport from virtual.conf does not make it work).

    doing it all again with tst1.xyz results in same issue and is resolved by same action.

    so, at this point all new vhosts created with nv (when using https) seem to be breaking things --> removing the reuseport value seems to fix things...

    seems strange to me that this is happening... wondering if its related to something I did... before I start tearing things apart and trying to get lucky I'd thought I'd just ask here: any ideas why? solutions that don't require a manual touch every time? [etc.]?

    seemed best to me to just link to my CmM setup doc so that you can see what I've done... happy to answer Qs/provide output if you want a closer look at anything.

    maxfein / max_cmm / source / cmm_setup_v0.2.md — Bitbucket

    Cheers, Max

    (just a note, in case you notice in browser: tst1.xyz has /public as a symlink to tst0.xyz /public ...am testing config options for WP multisite... this seems a simple way to do it... haven't gotten further in testing yet though... ;)
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    reuseport is only meant to be set once per ip:port pairing and it applies reuseport to all nginx vhosts after setting. So only one nginx vhost can set reuseport ip:80 and and only one nginx vhost can set reuseport on ip:443.

    FYI, 123.09beta01 nginx https vhost generator now automatically checks if reuseport is used or not on ip:443 pairing and if not, will auto add reuseport directive to the currently generated nginx HTTPS vhost if no other current nginx HTTPS vhost has reuseport directive. Of course centmin.sh also checks if your linux kernel supports reuseport first before auto adding the directive.

    So best is to never try manually adding reuseport to Centmin Mod created nginx vhosts, as centmin mod will automatically determine if reuseport can be added. Though I should add checks for main hostname ip:443 reuseport listen directive usage too
     
  3. MaxFein

    MaxFein New Member

    13
    2
    3
    Jul 9, 2018
    Ratings:
    +5
    Local Time:
    4:54 PM
    yes

    I did not try to add reuseport

    Instead, I copied virtual.conf to make base virtual.ssl.conf
    Code:
    cp /usr/local/nginx/conf/conf.d/virtual.conf  /usr/local/nginx/conf/conf.d/virtual.ssl.conf
    and added dhparam then, when it was observed working in browser and at ssllabs I added server context to top of virtual.ssl.conf like
    Code:
    server {
       listen 80;
       server_name srvr1.tivism.com;
       return 301 https://srvr1.tivism.com$request_uri;
     }
    and changed existing server context from
    Code:
    server {
      listen 443 ssl http2;
    to
    Code:
    server {
      listen 443 default_server ssl http2 fastopen=256 backlog=2048 reuseport;
    and then
    Code:
    mv /usr/local/nginx/conf/conf.d/virtual.conf  /usr/local/nginx/conf/conf.d/virtual.conf-disabled
    Code:
    mv /usr/local/nginx/conf/conf.d/virtual.ssl.conf  /usr/local/nginx/conf/conf.d/virtual.conf
    Code:
    ngxrestart
    ...and it was fine (as expected)... did testing stuff with things in /html and all was cool...
    the issue was when I ran
    Code:
    nv -d tst0.xyz -s lelived
    ...observing the output, it seemed to create the vhost and self-signed certificate and then nginx failed to restart prior to running acme... googling brought me to stuff about reuseport and so I noticed that tst0.xyz.ssl.conf had been made like
    Code:
    server {
      listen 443 ssl http2 reuseport;
      server_name tst0.xyz www.tst0.xyz;
    and
    Code:
    journalctl -xe
    
    Jul 30 00:57:41 srvr1.tivism.com nginx[23737]: Stopping nginx: [  OK  ]
    Jul 30 00:57:41 srvr1.tivism.com systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server...
    -- Subject: Unit nginx.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit nginx.service has begun starting up.
    Jul 30 00:57:41 srvr1.tivism.com nginx[23751]: Starting nginx: nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/virtual.conf:8
    Jul 30 00:57:41 srvr1.tivism.com nginx[23751]: [FAILED]
    Jul 30 00:57:41 srvr1.tivism.com systemd[1]: nginx.service: control process exited, code=exited status=1
    Jul 30 00:57:41 srvr1.tivism.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    -- Subject: Unit nginx.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit nginx.service has failed.
    --
    -- The result is failed.
    ...so, when reuseport is removed from the auto generated tst0.xyz.ssl.conf then things work as expected.

    Just wanted to be clear about what I did...

    I really only want 443 to be default and seems that CmM really wants to find a file named virtual.conf ...so, this seemed a potentially easy way to go.

    I moved reuseport from a server context listening at :80 to a server context listening at :443 - and its all in the file virtual.conf

    Then, when using nv with -s lelived the vhost seems to be made with reuseports in an unexpected way that breaks things bcs reasons ;)

    ...so, seems like maybe this is both the why and the way forward:
    yeah?

    Cheers, Max

    [edit: also, not sure I selected the best cat for this post re unexpected behavior... maybe better for such in bug reports?]
     
    Last edited: Jul 30, 2018
  4. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah that is why as centmin mod nginx vhost routine doesn't check main hostname for reuseport usage right now :)

    for now fix is in virtual.ssl.conf remove reuseport in listen directive and let centmin nginx vhost creation routine add it to first https 443 port based vhost automatically
     
    • Informative Informative x 1
  5. MaxFein

    MaxFein New Member

    13
    2
    3
    Jul 9, 2018
    Ratings:
    +5
    Local Time:
    4:54 PM
    tried this already, seemed not to work... virtual.conf is like
    Code:
    server {
       listen 80;
       server_name srvr1.tivism.com;
       return 301 https://srvr1.tivism.com$request_uri;
     }
    
    server {
      listen 443 default_server ssl http2 fastopen=256 backlog=2048;
      server_name srvr1.tivism.com;
    and tst0.xyz.ssl.conf is like
    Code:
    server {
      listen 443 ssl http2 reuseport;
      server_name tst0.xyz www.tst0.xyz;
    and ngxrestart fails, with same mssg
    Code:
    Starting nginx: nginx: [emerg] duplicate listen options for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/virtual.conf:8
    ...same after reboot the server

    line 8 of current virtual.conf does not contain reuseport (I removed it from the file entirely, no comments or such)...? ...so this is weird, right?

    ...if I put reuseport back in :433 server in virtual.conf and remove reuseport from every other :443 server domain.tld.ssl.conf then everything works as expected.

    Did I miss something here?

    Cheers, Max
     
    Last edited: Jul 30, 2018
  6. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    strange indeed I can replicate the issue too if virtual.ssl.conf has reuseport only on port 443 listen directive = ok. But if a nginx vhost has reuseport and virtual.ssl.conf doesn't, it gives duplicate listen options error.

    will need to add a check to nginx vhost creation to check if main hostname in virtual.ssl.conf has reuseport too
     
    • Like Like x 1
  7. MaxFein

    MaxFein New Member

    13
    2
    3
    Jul 9, 2018
    Ratings:
    +5
    Local Time:
    4:54 PM
    awesome

    fyi, I might actually prefer to use this option at a vhost for a WP multisite rather than default server... not sure, need to read... thought I'd mention it though as a possible use case ;)

    Thanks very much, Max
     
  8. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    actually the current check already checks all vhosts including virtual.ssl.conf in listen /usr/local/nginx/conf/conf.d/ so nginx vhost creation routine should not have added reuseport if virtual.ssl.conf already had reuseport

    part of the check command
    Code (Text):
    grep -rn listen /usr/local/nginx/conf/conf.d/ | grep -v '#' | grep 443 | grep ' ssl' | grep ' http2' | grep reuseport 
    

    would return
    Code (Text):
    grep -rn listen /usr/local/nginx/conf/conf.d/ | grep -v '#' | grep 443 | grep ' ssl' | grep ' http2' | grep reuseport 
    /usr/local/nginx/conf/conf.d/virtual.ssl.conf:2:            listen 443 ssl http2 default_server backlog=2048 reuseport;
    

    if it returns reuseport, then nginx vhost creation skips adding reuseport
     
  9. MaxFein

    MaxFein New Member

    13
    2
    3
    Jul 9, 2018
    Ratings:
    +5
    Local Time:
    4:54 PM
    well, as you say, when reuseport is not used in either virtual.conf or virtual.ssl.conf and is used in tst0.xyz.ssl.conf
    Code:
    [16:48][[email protected] ~]# grep -rn listen /usr/local/nginx/conf/conf.d/ | grep -v '#' | grep 443 | grep ' ssl' | grep ' http2' | grep reuseport
    /usr/local/nginx/conf/conf.d/tst0.xyz.ssl.conf:15:  listen 443 ssl http2 reuseport;
    then
    Code:
    [16:51][[email protected] ~]# ngxrestart
    Restarting nginx (via systemctl):  Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
                                                               [FAILED]
    when reuseport is used in virtual.conf and is not used in any other vhost
    Code:
    [16:56][[email protected] conf.d]# grep -rn listen /usr/local/nginx/conf/conf.d/ | grep -v '#' | grep 443 | grep ' ssl' | grep ' http2' | grep reuseport
    /usr/local/nginx/conf/conf.d/virtual.conf:8:  listen 443 default_server ssl http2 fastopen=256 backlog=2048 reuseport;
    then
    Code:
    [16:56][[email protected] conf.d]# ngxrestart
    Restarting nginx (via systemctl):                          [  OK  ]
    and
    Code:
    nv -d tst2.xyz -s lelived
    succeeds without any issue

    ...however, when configured like this I swear it was not working before (ngx failed to restart, new vhosts created with duplicate reuseport option)... tested it six ways from Sunday... this morning, now its working... could change have been related to rebooting the whole system?

    ...in any case, at least when reuseport is in virtual.conf (and I'd guess it would also work if I was using virtual.ssl.conf) things are working as expected now - will be nice to have the ability to set option at a diff vhost (eg. tst0.xyz.ssl.conf) once you push an update.

    Cheers, Max
     
  10. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    there's no update to push as the code already working as expected to detect virtual.ssl.conf as well but yes there's a bug when have virtual.ssl.conf without reuseport but vhost has reuseport - need to figure that out

    you can check that all listening ports 80 and 443 have reuseport via command ss command listed at Nginx - Cloudflare: Why does one NGINX worker take all the load?
    Code (Text):
    ss -lnt | egrep -e ':80 |:443 '
    

    this means reuseport is working due to virtual.conf and virtual.ssl.conf having reuseport = socket sharding in use where one per cpu thread, so on 8 cpu thread server, should have 8x port 80 and 8x port 443 listing
    Code (Text):
    ss -lnt | egrep -e ':80 |:443 '
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:443                      *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*               
    LISTEN     0      2048         *:80                       *:*
    
     
    Last edited: Jul 31, 2018
  11. eva2000

    eva2000 Administrator Staff Member

    36,355
    7,981
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,290
    Local Time:
    9:54 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
..