Want to subscribe to topics you're interested in?
Become a Member

Beta Branch Nginx Upgrade - zero downtime mode

Discussion in 'Beta release code' started by eva2000, Jul 16, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:44 PM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod 123.09beta01's nginx upgrade routine (centmin.sh menu option 4) has added support for zero downtime mode on the fly nginx binary updates as outlined here. This routine was original added to a separate experimental branch of centmin mod 123.08 betas and now ported over to 123.09beta01.

    It's an optional feature which is disabled by default. To enable it add to your persistent config file at /etc/centminmod/custom_config.inc the variable.
    Code (Text):
    NGINX_ZERODT='y'
    

    • On centmin.sh menu option 4 recompile or upgrade/downgrades of Nginx with NGINX_ZERODT='y' enabled, you will see additional information to help verify zero downtime on the fly nginx binary updates are indeed working as outlined at http://nginx.org/en/docs/control.html#upgrade.
    • You'll see the previous nginx version's binary renamed and copied as /usr/local/sbin/nginx.old so you can revert to previous nginx version more quickly. Though you can also do that via centmin.sh menu option 4 recompiles.
    Existing nginx master process id (pid) = 4533 and the new updated nginx binary master pid = 9780
    Code (Text):
    ---------------------------------------------------------------------------
    nginx master id: 4533
    ---------------------------------------------------------------------------
    Active connections: 1
    server accepts handled requests
    256 256 257
    Reading: 0 Writing: 1 Waiting: 0
    ---------------------------------------------------------------------------
    kill -USR2 4533
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    4533     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    4536  4533 nginx     0.0 295708 ep_pol nginx: worker process
    4537  4533 nginx     0.0 295708 ep_pol nginx: worker process
    4538  4533 nginx     0.0 295708 ep_pol nginx: worker process
    4539  4533 nginx     0.0 324388 ep_pol nginx: worker process
    9780  4533 root      1.6 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    9799  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9801  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9802  9780 nginx     0.0 271124 ep_pol nginx: worker process
    
    -rwxr-xr-x 1 root root 3.4M Jul 15 15:42 /usr/local/sbin/nginx
    -rwxr-xr-x 1 root root 3.4M Jul 15 15:29 /usr/local/sbin/nginx.old
    ---------------------------------------------------------------------------
    Active connections: 1
    server accepts handled requests
    257 257 258
    Reading: 0 Writing: 1 Waiting: 0
    ---------------------------------------------------------------------------
    

    Old nginx master childs all exited leaving just the master pid = 4533 while new nginx master pid = 9780 and childs with PPID = 9780 are running side by side
    Code (Text):
    ---------------------------------------------------------------------------
    kill -WINCH 4533
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    4533     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    9780  4533 root      0.8 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    9799  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9801  9780 nginx     1.0 324388 ep_pol nginx: worker process
    9802  9780 nginx     0.0 271124 ep_pol nginx: worker process
    ---------------------------------------------------------------------------
    Active connections: 1
    server accepts handled requests
    3 3 3
    Reading: 0 Writing: 1 Waiting: 0
    ---------------------------------------------------------------------------
    

    End old master nginx pid = 4533 leaving just the new nginx master pid = 9780 and childs with PPID = 9780 are running
    Code (Text):
    ---------------------------------------------------------------------------
    kill -QUIT 4533
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    9780     1 root      0.5 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    9799  9780 nginx     0.0 295708 ep_pol nginx: worker process
    9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
    9801  9780 nginx     0.6 324388 ep_pol nginx: worker process
    9802  9780 nginx     0.0 271124 ep_pol nginx: worker process
    ---------------------------------------------------------------------------
    Active connections: 1
    server accepts handled requests
    5 5 5
    Reading: 0 Writing: 1 Waiting: 0
    ---------------------------------------------------------------------------
    

     
    Last edited: Jul 16, 2016
    • Like Like x 2
    • Winner Winner x 1
  2. tjk

    tjk Member

    74
    16
    8
    Jun 27, 2015
    Ratings:
    +25
    Local Time:
    5:44 AM
    Not that I need it for my sites, but this is way cool @eva2000 !
     
    • Like Like x 1
  3. pamamolf

    pamamolf Well-Known Member

    2,670
    240
    63
    May 31, 2014
    Ratings:
    +425
    Local Time:
    12:44 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I thought that this will replace the old upgrade routine at all and not as an option...or isn't stable yet?
     
  4. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:44 PM
    Nginx 1.13.x
    MariaDB 5.5
    Cheers, has always been on the books this feature for centmin mod eventually :)
    123.09beta01 so test it now and later on stable release NGINX_ZERODT='y' will be default in centmin.sh :D
     
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:44 PM
    Nginx 1.13.x
    MariaDB 5.5
    revised the NGINX_ZERODT='y' routine for the time between kill -WINCH and kill -QUIT commands to properly detect when the old nginx worker child processes with PPID = to the old nginx master PID eventually gracefully shutdown before issuing kill -QUIT. This can potentially extend and prolong the whole nginx upgrade routine in centmin.sh menu option 4 until the old nginx worker child processes finish. But this should probably allow old nginx processes to complete first.

    You'll have this extra output before kill -QUIT is issued
    Code (Text):
    ---------------------------------------------------------------------------
    waiting for old nginx worker processes to exit...
     checking... worker child PPID=24050 exists
     checking... worker child PPID=24050 exists
     checking... worker child PPID=24050 exists
    ---------------------------------------------------------------------------
    kill -QUIT 24050
    

    Full example
    Code (Text):
    ---------------------------------------------------------------------------
    nginx master id: 24050
    ---------------------------------------------------------------------------
    Active connections: 2
    server accepts handled requests
    871 871 868
    Reading: 0 Writing: 1 Waiting: 1
    ---------------------------------------------------------------------------
    kill -USR2 24050
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    1076 24050 root      1.6 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    1079  1076 nginx     0.0 271124 ep_pol nginx: worker process
    1080  1076 nginx     0.0 271124 ep_pol nginx: worker process
    1081  1076 nginx     2.0 324388 ep_pol nginx: worker process
    1082  1076 nginx     0.0 271124 ep_pol nginx: worker process
    24050     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    24051 24050 nginx     0.0 308000 ep_pol nginx: worker process
    24052 24050 nginx     0.0 295708 ep_pol nginx: worker process
    24053 24050 nginx     0.0 295708 ep_pol nginx: worker process
    24054 24050 nginx     0.0 308000 ep_pol nginx: worker process
    
    -rwxr-xr-x 1 root root 3.4M Jul 16 00:51 /usr/local/sbin/nginx
    -rwxr-xr-x 1 root root 3.4M Jul 15 15:42 /usr/local/sbin/nginx.old
    ---------------------------------------------------------------------------
    Active connections: 1
    server accepts handled requests
    3 3 3
    Reading: 0 Writing: 1 Waiting: 0
    

    Code (Text):
    ---------------------------------------------------------------------------
    kill -WINCH 24050
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    1076 24050 root      0.8 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    1079  1076 nginx     0.0 295708 ep_pol nginx: worker process
    1080  1076 nginx     0.0 295708 ep_pol nginx: worker process
    1081  1076 nginx     1.0 324388 ep_pol nginx: worker process
    1082  1076 nginx     0.0 295708 ep_pol nginx: worker process
    24050     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    ---------------------------------------------------------------------------
    Active connections: 2
    server accepts handled requests
    12 12 12
    Reading: 0 Writing: 1 Waiting: 1
    ---------------------------------------------------------------------------
    waiting for old nginx worker processes to exit...
     checking... worker child PPID=24050 exists
     checking... worker child PPID=24050 exists
     checking... worker child PPID=24050 exists
    ---------------------------------------------------------------------------
    kill -QUIT 24050
      PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
    1076     1 root      0.5 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    1079  1076 nginx     0.0 295708 ep_pol nginx: worker process
    1080  1076 nginx     0.1 295708 ep_pol nginx: worker process
    1081  1076 nginx     0.6 324388 ep_pol nginx: worker process
    1082  1076 nginx     0.0 295708 ep_pol nginx: worker process
    ---------------------------------------------------------------------------
    Active connections: 2
    server accepts handled requests
    15 15 15
    Reading: 0 Writing: 1 Waiting: 1
    ---------------------------------------------------------------------------