Beta Branch add NGINX_IOURING_PATCH variable support in 123.09beta01

eva2000 · Aug 6, 2019

add NGINX_IOURING_PATCH variable support in 123.09beta01

- For Nginx 1.17.0+ and Linux Kernel 5.1 usage only when NGINX_IOURING_PATCH='y' is enabled via persistent config file /etc/centminmod/custom_config.inc setting prior to centmin.sh menu option 4 nginx compiles (disabled by default), apply patch to allow Nginx AIO https://nginx.org/en/docs/http/ngx_http_core_module.html#aio to use Linux Kernel 5.1's new io_uring interface to support more efficient buffered AIO with less system calls https://lwn.net/Articles/776703/ & https://hub.packtpub.com/linux-5-1-...-memory-new-patching-improvements-and-more-2/
- Obviously, if you remove/downgrade Linux Kernel from 5.1, need to set NGINX_IOURING_PATCH='n', and recompile Nginx via centmin.sh menu option 4 to remove the patch
- Once patched in /usr/local/nginx/conf/nginx.conf need to disable sendfiles and enable aio. The centmin.sh menu option 4 routine will attempt to do this automatically so you can verify if the following is set in nginx.conf when NGINX_IOURING_PATCH='y' enabled. When NGINX_IOURING_PATCH='n' set, centmin.sh menu option 4 routine will attempt to remove aio on and turn sendfile on again.
Code (Text):
sendfile off;
aio on;
Continue reading...

123.09beta01 branch

Branch: https://github.com/centminmod/centminmod/tree/123.09beta01

Commit History: https://github.com/centminmod/centminmod/commits/123.09beta01

eva2000 · Aug 6, 2019

Bit more reading on io_uring

https://developpaper.com/native-linux-asynchronous-file-operation-io_uring-experience/

older benchmarks done by io_uring author https://lore.kernel.org/linux-block/20190116175003.17880-1-axboe@kernel.dk/

Example on one of my Centmin Mod 123.09beta01 Nginx patched servers running Linux Kernel 5.2.5

with --with-file-aio specifically enabled for NGINX_IOURING_PATCH='y'

nginx -V
nginx version: nginx/1.17.2 (060819-062643-centos7-kvm)
built by gcc 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
built with OpenSSL 1.1.1c 28 May 2019
TLS SNI support enabled
configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=1 -fuse-ld=gold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto=1 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=060819-062643-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-file-aio --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../o
Click to expand...

Nginx configure stage log see
Code (Text):
checking for Linux io_uring support (liburing) ... found
Code (Text):
./configure --with-ld-opt="-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=1 -fuse-ld=gold" --with-cc-opt="-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto=1 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2" --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=060819-062643-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-file-aio --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1c --with-openssl-opt="enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold"
checking for OS
 + Linux 5.2.5-1.el7.elrepo.x86_64 x86_64
checking for C compiler ... found
 + using GNU C compiler
 + gcc version: 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
checking for gcc -pipe switch ... found
checking for --with-ld-opt="-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=1 -fuse-ld=gold" ... found
checking for -Wl,-E switch ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for gcc builtin 64 bit byteswap ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for EPOLLRDHUP ... found
checking for EPOLLEXCLUSIVE ... not found
checking for O_PATH ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for prctl(PR_SET_KEEPCAPS) ... found
checking for capabilities ... found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for nobody group ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... not found
checking for crypt() in libcrypt ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... not found
checking for dlopen() in libdl ... found
checking for sched_yield() ... found
checking for sched_setaffinity() ... found
checking for SO_SETFIB ... not found
checking for SO_REUSEPORT ... found
checking for SO_ACCEPTFILTER ... not found
checking for SO_BINDANY ... not found
checking for IP_TRANSPARENT ... found
checking for IP_BINDANY ... not found
checking for IP_BIND_ADDRESS_NO_PORT ... not found
checking for IP_RECVDSTADDR ... not found
checking for IP_SENDSRCADDR ... not found
checking for IP_PKTINFO ... found
checking for IPV6_RECVPKTINFO ... found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE ... found
checking for TCP_FASTOPEN ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for kqueue AIO support ... not found
checking for Linux io_uring support (liburing) ... found

steph40 · Aug 7, 2019

@eva2000 Wanted to try this but it is failing to compile and I get error 502 bad gateway. When I recompile without NGINX_IOURING_PATCH='y' everything works.

Here is my custom config

Code:

LETSENCRYPT_DETECT='y'
ENABLE_MARIADBTENTWOUPGRADE='y'
ENABLE_MARIADBTENTHREEUPGRADE='y'
NGXDYNAMIC_NGXPAGESPEED='y'
NGINX_PAGESPEED='y'
PHP_PGO='y'
SWITCH_PHPFPM_SYSTEMD='y'
NGINX_DYNAMICTLS='y'
MEMCACHED_TLS='y'
NGINX_IOURING_PATCH='y'

LIBRESSL_SWITCH='n'
CLANG='n'
DEVTOOLSETSIX='n'
DEVTOOLSETSEVEN='n'
DEVTOOLSETEIGHT='y'
NGINX_DEVTOOLSETGCC='y'
NGINX_FATLTO_OBJECTS='y'
NGINXOPENSSL_FATLTO_OBJECTS='y'

CLOUDFLARE_ZLIBPHP='y'
CLOUDFLARE_ZLIB='y'
NGINX_HPACK='y'

PHP_LIBZIP='y'
PHP_ARGON='y'

NGXDYNAMIC_BROTLI='y'
NGINX_LIBBROTLI='y'

NGINX_MODSECURITY_MAXMIND='N'
NGINX_MODSECURITY='N'

Here is the log

Code:

1017002
1017002
1017002
1017002
patch -p1 < /usr/local/src/centminmod/patches/nginx/nginx-1.17.2-fix-zero-size-buf.patch
patching file src/http/modules/ngx_http_gzip_filter_module.c
1017002
/svr-setup /svr-setup/nginx-1.17.2
Cloning into 'liburing'...
fatal: dumb http transport does not support --depth
inc/nginx_patch.inc: line 18: cd: liburing: No such file or directory
inc/nginx_patch.inc: line 19: ./configure: No such file or directory
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target `install'.  Stop.
/svr-setup/nginx-1.17.2
/svr-setup/nginx-1.17.2 /svr-setup/nginx-1.17.2
[1;33;40mpatching nginx for Linux 5.1 Kernel AIO io_ring https://github.com/hakasenyang/openssl-patch/pull/22
(B[m[1;33;40mhttps://lwn.net/Articles/776703/
(B[m[1;33;40mhttps://hub.packtpub.com/linux-5-1-out-with-io_uring-io-interface-persistent-memory-new-patching-improvements-and-more-2/
(B[m[1;33;40mpatch -p1 < /usr/local/src/centminmod/patches/nginx/nginx_io_uring.patch
(B[mpatching file auto/unix
patching file src/event/modules/ngx_epoll_module.c
patching file src/event/ngx_event.h
patching file src/os/unix/ngx_linux_aio_read.c
patching file src/os/unix/ngx_linux_config.h

patched nginx for Linux 5.1 Kernel AIO io_ring
/svr-setup/nginx-1.17.2
1017002
/usr/local/src/centminmod
15
18
/svr-setup/redis2-nginx-module-0.15/src
/svr-setup/memc-nginx-module-0.18/src
1017002

eva2000 · Aug 7, 2019

Just fixed the bug, you can run cmupdate to update 123.09beta01 and try again

steph40 · Aug 7, 2019

Thank you will do.

steph40 · Aug 7, 2019

@eva2000 First install after fix failed but after changing custom config to
Code:
NGINX_FATLTO_OBJECTS='n'
NGINXOPENSSL_FATLTO_OBJECTS='n'
Success

eva2000 · Aug 7, 2019

yes those 2 are best left disabled as noted in centmin.sh comments as well

fabianski · Aug 8, 2019

Did you update the kernel manually or did the hosting do it?

My vps in upcloud is still in version 3
Code (Text):
$ uname -r
3.10.0-957.21.3.el7.x86_64

steph40 · Aug 8, 2019

fabianski said: ↑
Did you update the kernel manually or did the hosting do it?

My vps in upcloud is still in version 3
Code (Text):
$ uname -r
3.10.0-957.21.3.el7.x86_64
Click to expand...
I did it manually after a quick google search, I was at 3.10 too. Now 5.1

steph40 · Aug 8, 2019

https://www.howtoforge.com/tutorial/how-to-upgrade-kernel-in-centos-7-server/

eva2000 · Aug 8, 2019

Yes elrepo kernel-ml packages is the way. But test on test server or VPS on same VPS provider you have live server and ensure you can properly reboot server on updated elrepo kernels on test server and all works, before trying on live server. Depending on how your server is configured and it's hardware, some 3rd party kernels like elrepo can prevent your server from rebooting properly i.e. dedicated servers with NVMe drives may have additional tasks - see https://community.centminmod.com/threads/grub-help.12867/ (and here)
eva2000 said: ↑
so sounds like for NVMe based servers, after a kernel update you need to rebuild grub before rebooting
Code (Text):
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
reboot
Click to expand...

steph40 · Aug 8, 2019

Good thing I don't have NVMe

eva2000 · Aug 8, 2019

It isn't just NVMe, but if you have dedicated servers with hardware raid controllers depending on how they're configured elrepo kernels and support them as well. I mainly used VPSes and only have dedicated servers with OVH right now and they're all fine for elrepo kernel-ml packages.

In fact as I prep Centmin Mod LEMP stack for listing on DigitalOcean's 1-Click App Marketplace, I also testing out build Centmin Mod LEMP stack preinstalled DO marketplace images with elrepo Linux 5.x kernels default out of the box too

fabianski · Aug 8, 2019

steph40 said: ↑

I did it manually after a quick google search, I was at 3.10 too. Now 5.1
Click to expand...

What is your hosting? Uplcloud too?

I will check with support to see if they do this for me

eva2000 said: ↑

It isn't just NVMe, but if you have dedicated servers with hardware raid controllers depending on how they're configured elrepo kernels and support them as well. I mainly used VPSes and only have dedicated servers with OVH right now and they're all fine for elrepo kernel-ml packages.

In fact as I prep Centmin Mod LEMP stack for listing on DigitalOcean's 1-Click App Marketplace, I also testing out build Centmin Mod LEMP stack preinstalled DO marketplace images with elrepo Linux 5.x kernels default out of the box too
Click to expand...

Are any extra steps required for upcloud?

eva2000 · Aug 8, 2019

No extra steps for most vps providers I have tested on digitalocean and upcloud. Haven't tested vultr lately for elrepo though I will soon as I am doing other tests with vultr soon.

steph40 · Aug 8, 2019

fabianski said: ↑

What is your hosting? Uplcloud too?
Click to expand...

No I am with ssdnodes

steph40 · Aug 19, 2019

@eva2000 Started a new vps with ssdnodes NVMe 6cores 24gb mem, upgraded to kernel 5.2.9-1.el7.elrepo.x86_64
My custom config is
Code (Text):
LETSENCRYPT_DETECT='y'
NGINX_HPACK='y'
NGINX_DYNAMICTLS='y'
NGINX_IORURING_PATCH='y'
PHP_PGO='y'
When I recompile nginx the NGINX_IORURING_PATCH='y' doesn't get applied and I don't see anything in the lod related to it. Is there an issue with the patch?

nginx compile aio - Pastebin.com

buik · Aug 19, 2019

Someone bench-marked Nginx upstream v.s. Nginx with NGINX_IOURING_PATCH?

upgrade81 · Sep 3, 2019

Hi, I have enabled and properly compiled nginx.
I have verified that sendfile is disabled, and AIO is enabled in nginx.conf.

Nevertheless, I find that on two different VMs, 2 different sites that some images on wordpress are not displayed, after a refresh (f5) I upload them.
I attach screenshots of edge (left) and chrome (right)

What could it be?

upgrade81 · Sep 3, 2019

enabling sendfile does it less.

Here are the errors from the chrome console.

some logs error.log

Code (Text):

2019/09/02 19:44:38 [crit] 24596#24596: *3357 aio read "/home/nginx/domains/xxxxxxxx.com/public/wp-content/uploads/2019/09/salvini-renzi-728x409.jpg" failed (14: Bad address) while sending response to client, client: 5.89.203.243, server: xxxxxxxx.com, request: "GET /wp-content/uploads/2019/09/salvini-renzi-728x409.jpg HTTP/2.0", host: "www.xxxxxx

Log in / Register

Forums

Welcome to Centmin Mod Community

Beta Branch add NGINX_IOURING_PATCH variable support in 123.09beta01

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

steph40 Member

eva2000 Administrator Staff Member

steph40 Member

steph40 Member

eva2000 Administrator Staff Member

fabianski Member

steph40 Member

steph40 Member

eva2000 Administrator Staff Member

steph40 Member

eva2000 Administrator Staff Member

fabianski Member

eva2000 Administrator Staff Member

steph40 Member

steph40 Member

buik “The best traveler is one without a camera.”

upgrade81 Member

Attached Files:

ApplicationFrameHost_gp9v7sxc1O.png

upgrade81 Member

Attached Files:

chrome_d828QrPLig.png

.

Log in / Register

Useful Searches

Welcome to Centmin Mod Community

Beta Branch add NGINX_IOURING_PATCH variable support in 123.09beta01

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

steph40 Member

eva2000 Administrator Staff Member

steph40 Member

steph40 Member

eva2000 Administrator Staff Member

fabianski Member

steph40 Member

steph40 Member

eva2000 Administrator Staff Member

steph40 Member

eva2000 Administrator Staff Member

fabianski Member

eva2000 Administrator Staff Member

steph40 Member

steph40 Member

buik “The best traveler is one without a camera.”

upgrade81 Member

Attached Files:

ApplicationFrameHost_gp9v7sxc1O.png

upgrade81 Member

Attached Files:

chrome_d828QrPLig.png

.