Beta Branch Nginx Upgrade - zero downtime mode

Centmin Mod 123.09beta01's nginx upgrade routine (centmin.sh menu option 4) has added support for zero downtime mode on the fly nginx binary updates as outlined here. This routine was original added to a separate experimental branch of centmin mod 123.08 betas and now ported over to 123.09beta01.

It's an optional feature which is disabled by default. To enable it add to your persistent config file at /etc/centminmod/custom_config.inc the variable.

Code (Text):

NGINX_ZERODT='y'

• On centmin.sh menu option 4 recompile or upgrade/downgrades of Nginx with NGINX_ZERODT='y' enabled, you will see additional information to help verify zero downtime on the fly nginx binary updates are indeed working as outlined at http://nginx.org/en/docs/control.html#upgrade.
• You'll see the previous nginx version's binary renamed and copied as /usr/local/sbin/nginx.old so you can revert to previous nginx version more quickly. Though you can also do that via centmin.sh menu option 4 recompiles.

Existing nginx master process id (pid) = 4533 and the new updated nginx binary master pid = 9780

Code (Text):

---------------------------------------------------------------------------
nginx master id: 4533
---------------------------------------------------------------------------
Active connections: 1
server accepts handled requests
256 256 257
Reading: 0 Writing: 1 Waiting: 0
---------------------------------------------------------------------------
kill -USR2 4533
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
4533     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
4536  4533 nginx     0.0 295708 ep_pol nginx: worker process
4537  4533 nginx     0.0 295708 ep_pol nginx: worker process
4538  4533 nginx     0.0 295708 ep_pol nginx: worker process
4539  4533 nginx     0.0 324388 ep_pol nginx: worker process
9780  4533 root      1.6 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9799  9780 nginx     0.0 271124 ep_pol nginx: worker process
9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
9801  9780 nginx     0.0 271124 ep_pol nginx: worker process
9802  9780 nginx     0.0 271124 ep_pol nginx: worker process

-rwxr-xr-x 1 root root 3.4M Jul 15 15:42 /usr/local/sbin/nginx
-rwxr-xr-x 1 root root 3.4M Jul 15 15:29 /usr/local/sbin/nginx.old
---------------------------------------------------------------------------
Active connections: 1
server accepts handled requests
257 257 258
Reading: 0 Writing: 1 Waiting: 0
---------------------------------------------------------------------------

Old nginx master childs all exited leaving just the master pid = 4533 while new nginx master pid = 9780 and childs with PPID = 9780 are running side by side

Code (Text):

---------------------------------------------------------------------------
kill -WINCH 4533
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
4533     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9780  4533 root      0.8 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9799  9780 nginx     0.0 271124 ep_pol nginx: worker process
9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
9801  9780 nginx     1.0 324388 ep_pol nginx: worker process
9802  9780 nginx     0.0 271124 ep_pol nginx: worker process
---------------------------------------------------------------------------
Active connections: 1
server accepts handled requests
3 3 3
Reading: 0 Writing: 1 Waiting: 0
---------------------------------------------------------------------------

End old master nginx pid = 4533 leaving just the new nginx master pid = 9780 and childs with PPID = 9780 are running

Code (Text):

---------------------------------------------------------------------------
kill -QUIT 4533
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
9780     1 root      0.5 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9799  9780 nginx     0.0 295708 ep_pol nginx: worker process
9800  9780 nginx     0.0 271124 ep_pol nginx: worker process
9801  9780 nginx     0.6 324388 ep_pol nginx: worker process
9802  9780 nginx     0.0 271124 ep_pol nginx: worker process
---------------------------------------------------------------------------
Active connections: 1
server accepts handled requests
5 5 5
Reading: 0 Writing: 1 Waiting: 0
---------------------------------------------------------------------------

---

 

Cheers, has always been on the books this feature for centmin mod eventually

123.09beta01 so test it now and later on stable release NGINX_ZERODT='y' will be default in centmin.sh

 

revised the NGINX_ZERODT='y' routine for the time between kill -WINCH and kill -QUIT commands to properly detect when the old nginx worker child processes with PPID = to the old nginx master PID eventually gracefully shutdown before issuing kill -QUIT. This can potentially extend and prolong the whole nginx upgrade routine in centmin.sh menu option 4 until the old nginx worker child processes finish. But this should probably allow old nginx processes to complete first.

You'll have this extra output before kill -QUIT is issued

Code (Text):

---------------------------------------------------------------------------
waiting for old nginx worker processes to exit...
 checking... worker child PPID=24050 exists
 checking... worker child PPID=24050 exists
 checking... worker child PPID=24050 exists
---------------------------------------------------------------------------
kill -QUIT 24050

Full example

Code (Text):

---------------------------------------------------------------------------
nginx master id: 24050
---------------------------------------------------------------------------
Active connections: 2
server accepts handled requests
871 871 868
Reading: 0 Writing: 1 Waiting: 1
---------------------------------------------------------------------------
kill -USR2 24050
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
1076 24050 root      1.6 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
1079  1076 nginx     0.0 271124 ep_pol nginx: worker process
1080  1076 nginx     0.0 271124 ep_pol nginx: worker process
1081  1076 nginx     2.0 324388 ep_pol nginx: worker process
1082  1076 nginx     0.0 271124 ep_pol nginx: worker process
24050     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
24051 24050 nginx     0.0 308000 ep_pol nginx: worker process
24052 24050 nginx     0.0 295708 ep_pol nginx: worker process
24053 24050 nginx     0.0 295708 ep_pol nginx: worker process
24054 24050 nginx     0.0 308000 ep_pol nginx: worker process

-rwxr-xr-x 1 root root 3.4M Jul 16 00:51 /usr/local/sbin/nginx
-rwxr-xr-x 1 root root 3.4M Jul 15 15:42 /usr/local/sbin/nginx.old
---------------------------------------------------------------------------
Active connections: 1
server accepts handled requests
3 3 3
Reading: 0 Writing: 1 Waiting: 0

Code (Text):

---------------------------------------------------------------------------
kill -WINCH 24050
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
1076 24050 root      0.8 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
1079  1076 nginx     0.0 295708 ep_pol nginx: worker process
1080  1076 nginx     0.0 295708 ep_pol nginx: worker process
1081  1076 nginx     1.0 324388 ep_pol nginx: worker process
1082  1076 nginx     0.0 295708 ep_pol nginx: worker process
24050     1 root      0.0 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
---------------------------------------------------------------------------
Active connections: 2
server accepts handled requests
12 12 12
Reading: 0 Writing: 1 Waiting: 1
---------------------------------------------------------------------------
waiting for old nginx worker processes to exit...
 checking... worker child PPID=24050 exists
 checking... worker child PPID=24050 exists
 checking... worker child PPID=24050 exists
---------------------------------------------------------------------------
kill -QUIT 24050
  PID  PPID USER     %CPU    VSZ WCHAN  COMMAND
1076     1 root      0.5 254736 sigsus nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
1079  1076 nginx     0.0 295708 ep_pol nginx: worker process
1080  1076 nginx     0.1 295708 ep_pol nginx: worker process
1081  1076 nginx     0.6 324388 ep_pol nginx: worker process
1082  1076 nginx     0.0 295708 ep_pol nginx: worker process
---------------------------------------------------------------------------
Active connections: 2
server accepts handled requests
15 15 15
Reading: 0 Writing: 1 Waiting: 1
---------------------------------------------------------------------------

 

I've tested it to work for myself, but no guarantee it works 100% for others, so leave it up to them to test it themselves and hopefully they report their feedback

 

nope - for that if you need 100% uptime, you need to use multiple servers in load balanced state each running php-fpm servers and a network shared/distributed common web site files storage configuration i.e. GlusterFS/NFS etc

 

The routine needs to monitor your nginx_status output - just for diagnostic purposes in nginx upgrade logs

Code (Text):

curl -s -4 localhost/nginx_status

Code (Text):

curl -s -4 localhost/nginx_status
Active connections: 1
server accepts handled requests
 7 7 7
Reading: 0 Writing: 1 Waiting: 0

 

It's handy to have for nginx statistics and if you use nginx related service monitoring i.e. nginx amplify, newrelic etc.

 

might just use 1.17.4 and remove nginx patches and see

also timeouts could be network related so unrelated to Nginx ?

 

are you behind Cloudflare ? if you are you don't really need HPACK nginx patch as Cloudflare can only communicate with origin servers over HTTP/1.1 and not HTTP/2 right now