Discover Centmin Mod today
Register Now

Beta Branch add NGINX_IOURING_PATCH variable support in 123.09beta01

Discussion in 'Centmin Mod Github Commits' started by eva2000, Aug 6, 2019.

  1. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    add NGINX_IOURING_PATCH variable support in 123.09beta01

    - For Nginx 1.17.0+ and Linux Kernel 5.1 usage only when NGINX_IOURING_PATCH='y' is enabled via persistent config file /etc/centminmod/custom_config.inc setting prior to centmin.sh menu option 4 nginx compiles (disabled by default), apply patch to allow Nginx AIO https://nginx.org/en/docs/http/ngx_http_core_module.html#aio to use Linux Kernel 5.1's new io_uring interface to support more efficient buffered AIO with less system calls https://lwn.net/Articles/776703/ & https://hub.packtpub.com/linux-5-1-...-memory-new-patching-improvements-and-more-2/
    - Obviously, if you remove/downgrade Linux Kernel from 5.1, need to set NGINX_IOURING_PATCH='n', and recompile Nginx via centmin.sh menu option 4 to remove the patch
    - Once patched in /usr/local/nginx/conf/nginx.conf need to disable sendfiles and enable aio. The centmin.sh menu option 4 routine will attempt to do this automatically so you can verify if the following is set in nginx.conf when NGINX_IOURING_PATCH='y' enabled. When NGINX_IOURING_PATCH='n' set, centmin.sh menu option 4 routine will attempt to remove aio on and turn sendfile on again.

    Code (Text):
    sendfile off;
    aio on;


    Continue reading...

    123.09beta01 branch
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Bit more reading on io_uring
    Example on one of my Centmin Mod 123.09beta01 Nginx patched servers running Linux Kernel 5.2.5

    with --with-file-aio specifically enabled for NGINX_IOURING_PATCH='y'
    Nginx configure stage log see
    Code (Text):
    checking for Linux io_uring support (liburing) ... found

    Code (Text):
    ./configure --with-ld-opt="-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=1 -fuse-ld=gold" --with-cc-opt="-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto=1 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2" --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=060819-062643-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-file-aio --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1c --with-openssl-opt="enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold"
    checking for OS
     + Linux 5.2.5-1.el7.elrepo.x86_64 x86_64
    checking for C compiler ... found
     + using GNU C compiler
     + gcc version: 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
    checking for gcc -pipe switch ... found
    checking for --with-ld-opt="-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=1 -fuse-ld=gold" ... found
    checking for -Wl,-E switch ... found
    checking for gcc builtin atomic operations ... found
    checking for C99 variadic macros ... found
    checking for gcc variadic macros ... found
    checking for gcc builtin 64 bit byteswap ... found
    checking for unistd.h ... found
    checking for inttypes.h ... found
    checking for limits.h ... found
    checking for sys/filio.h ... not found
    checking for sys/param.h ... found
    checking for sys/mount.h ... found
    checking for sys/statvfs.h ... found
    checking for crypt.h ... found
    checking for Linux specific features
    checking for epoll ... found
    checking for EPOLLRDHUP ... found
    checking for EPOLLEXCLUSIVE ... not found
    checking for O_PATH ... found
    checking for sendfile() ... found
    checking for sendfile64() ... found
    checking for sys/prctl.h ... found
    checking for prctl(PR_SET_DUMPABLE) ... found
    checking for prctl(PR_SET_KEEPCAPS) ... found
    checking for capabilities ... found
    checking for crypt_r() ... found
    checking for sys/vfs.h ... found
    checking for nobody group ... found
    checking for poll() ... found
    checking for /dev/poll ... not found
    checking for kqueue ... not found
    checking for crypt() ... not found
    checking for crypt() in libcrypt ... found
    checking for F_READAHEAD ... not found
    checking for posix_fadvise() ... found
    checking for O_DIRECT ... found
    checking for F_NOCACHE ... not found
    checking for directio() ... not found
    checking for statfs() ... found
    checking for statvfs() ... found
    checking for dlopen() ... not found
    checking for dlopen() in libdl ... found
    checking for sched_yield() ... found
    checking for sched_setaffinity() ... found
    checking for SO_SETFIB ... not found
    checking for SO_REUSEPORT ... found
    checking for SO_ACCEPTFILTER ... not found
    checking for SO_BINDANY ... not found
    checking for IP_TRANSPARENT ... found
    checking for IP_BINDANY ... not found
    checking for IP_BIND_ADDRESS_NO_PORT ... not found
    checking for IP_RECVDSTADDR ... not found
    checking for IP_SENDSRCADDR ... not found
    checking for IP_PKTINFO ... found
    checking for IPV6_RECVPKTINFO ... found
    checking for TCP_DEFER_ACCEPT ... found
    checking for TCP_KEEPIDLE ... found
    checking for TCP_FASTOPEN ... found
    checking for TCP_INFO ... found
    checking for accept4() ... found
    checking for kqueue AIO support ... not found
    checking for Linux io_uring support (liburing) ... found
    
     
  3. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    @eva2000 Wanted to try this but it is failing to compile and I get error 502 bad gateway. When I recompile without NGINX_IOURING_PATCH='y' everything works.

    Here is my custom config
    Code:
    LETSENCRYPT_DETECT='y'
    ENABLE_MARIADBTENTWOUPGRADE='y'
    ENABLE_MARIADBTENTHREEUPGRADE='y'
    NGXDYNAMIC_NGXPAGESPEED='y'
    NGINX_PAGESPEED='y'
    PHP_PGO='y'
    SWITCH_PHPFPM_SYSTEMD='y'
    NGINX_DYNAMICTLS='y'
    MEMCACHED_TLS='y'
    NGINX_IOURING_PATCH='y'
    
    LIBRESSL_SWITCH='n'
    CLANG='n'
    DEVTOOLSETSIX='n'
    DEVTOOLSETSEVEN='n'
    DEVTOOLSETEIGHT='y'
    NGINX_DEVTOOLSETGCC='y'
    NGINX_FATLTO_OBJECTS='y'
    NGINXOPENSSL_FATLTO_OBJECTS='y'
    
    CLOUDFLARE_ZLIBPHP='y'
    CLOUDFLARE_ZLIB='y'
    NGINX_HPACK='y'
    
    PHP_LIBZIP='y'
    PHP_ARGON='y'
    
    NGXDYNAMIC_BROTLI='y'
    NGINX_LIBBROTLI='y'
    
    NGINX_MODSECURITY_MAXMIND='N'
    NGINX_MODSECURITY='N'
    
    Here is the log
    Code:
    1017002
    1017002
    1017002
    1017002
    patch -p1 < /usr/local/src/centminmod/patches/nginx/nginx-1.17.2-fix-zero-size-buf.patch
    patching file src/http/modules/ngx_http_gzip_filter_module.c
    1017002
    /svr-setup /svr-setup/nginx-1.17.2
    Cloning into 'liburing'...
    fatal: dumb http transport does not support --depth
    inc/nginx_patch.inc: line 18: cd: liburing: No such file or directory
    inc/nginx_patch.inc: line 19: ./configure: No such file or directory
    make: *** No targets specified and no makefile found.  Stop.
    make: *** No rule to make target `install'.  Stop.
    /svr-setup/nginx-1.17.2
    /svr-setup/nginx-1.17.2 /svr-setup/nginx-1.17.2
    [1;33;40mpatching nginx for Linux 5.1 Kernel AIO io_ring https://github.com/hakasenyang/openssl-patch/pull/22
    (B[m[1;33;40mhttps://lwn.net/Articles/776703/
    (B[m[1;33;40mhttps://hub.packtpub.com/linux-5-1-out-with-io_uring-io-interface-persistent-memory-new-patching-improvements-and-more-2/
    (B[m[1;33;40mpatch -p1 < /usr/local/src/centminmod/patches/nginx/nginx_io_uring.patch
    (B[mpatching file auto/unix
    patching file src/event/modules/ngx_epoll_module.c
    patching file src/event/ngx_event.h
    patching file src/os/unix/ngx_linux_aio_read.c
    patching file src/os/unix/ngx_linux_config.h
    
    patched nginx for Linux 5.1 Kernel AIO io_ring
    /svr-setup/nginx-1.17.2
    1017002
    /usr/local/src/centminmod
    15
    18
    /svr-setup/redis2-nginx-module-0.15/src
    /svr-setup/memc-nginx-module-0.18/src
    1017002
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Just fixed the bug, you can run cmupdate to update 123.09beta01 and try again
     
  5. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    Thank you will do.
     
  6. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    @eva2000 First install after fix failed but after changing custom config to
    Code:
    NGINX_FATLTO_OBJECTS='n'
    NGINXOPENSSL_FATLTO_OBJECTS='n'
    
    Success
     
  7. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yes those 2 are best left disabled as noted in centmin.sh comments as well
     
  8. fabianski

    fabianski Member

    102
    13
    18
    Feb 20, 2019
    Brazil
    Ratings:
    +35
    Local Time:
    10:44 PM
    Did you update the kernel manually or did the hosting do it?

    My vps in upcloud is still in version 3

    Code (Text):
    $ uname -r
    3.10.0-957.21.3.el7.x86_64
    
     
  9. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    I did it manually after a quick google search, I was at 3.10 too. Now 5.1
     
  10. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
  11. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yes elrepo kernel-ml packages is the way. But test on test server or VPS on same VPS provider you have live server and ensure you can properly reboot server on updated elrepo kernels on test server and all works, before trying on live server. Depending on how your server is configured and it's hardware, some 3rd party kernels like elrepo can prevent your server from rebooting properly i.e. dedicated servers with NVMe drives may have additional tasks - see https://community.centminmod.com/threads/grub-help.12867/ (and here)

     
    Last edited: Aug 8, 2019
  12. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    Good thing I don't have NVMe :p
     
    • Like Like x 1
  13. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    It isn't just NVMe, but if you have dedicated servers with hardware raid controllers depending on how they're configured elrepo kernels and support them as well. I mainly used VPSes and only have dedicated servers with OVH right now and they're all fine for elrepo kernel-ml packages.

    In fact as I prep Centmin Mod LEMP stack for listing on DigitalOcean's 1-Click App Marketplace, I also testing out build Centmin Mod LEMP stack preinstalled DO marketplace images with elrepo Linux 5.x kernels default out of the box too :)
     
    • Like Like x 1
  14. fabianski

    fabianski Member

    102
    13
    18
    Feb 20, 2019
    Brazil
    Ratings:
    +35
    Local Time:
    10:44 PM
    What is your hosting? Uplcloud too?

    I will check with support to see if they do this for me :D


    Are any extra steps required for upcloud?
     
  15. eva2000

    eva2000 Administrator Staff Member

    41,645
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    11:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    No extra steps for most vps providers I have tested on digitalocean and upcloud. Haven't tested vultr lately for elrepo though I will soon as I am doing other tests with vultr soon.
     
  16. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    No I am with ssdnodes
     
  17. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    9:44 PM
    1.1.5
    mariadb 10
    @eva2000 Started a new vps with ssdnodes NVMe 6cores 24gb mem, upgraded to kernel 5.2.9-1.el7.elrepo.x86_64
    My custom config is
    Code (Text):
    LETSENCRYPT_DETECT='y'
    NGINX_HPACK='y'
    NGINX_DYNAMICTLS='y'
    NGINX_IORURING_PATCH='y'
    PHP_PGO='y'
    


    When I recompile nginx the NGINX_IORURING_PATCH='y' doesn't get applied and I don't see anything in the lod related to it. Is there an issue with the patch?

    nginx compile aio - Pastebin.com
     
  18. buik

    buik Well-Known Member

    1,193
    318
    83
    Apr 29, 2016
    Ratings:
    +937
    Local Time:
    3:44 AM
    Someone bench-marked Nginx upstream v.s. Nginx with NGINX_IOURING_PATCH?
     
    • Agree Agree x 1
  19. upgrade81

    upgrade81 Premium Member Premium Member

    233
    14
    18
    Sep 5, 2016
    Switzerland
    Ratings:
    +21
    Local Time:
    3:44 AM
    1.17
    10.3
    Hi, I have enabled and properly compiled nginx.
    I have verified that sendfile is disabled, and AIO is enabled in nginx.conf.

    Nevertheless, I find that on two different VMs, 2 different sites that some images on wordpress are not displayed, after a refresh (f5) I upload them.
    I attach screenshots of edge (left) and chrome (right)


    What could it be?
     

    Attached Files:

  20. upgrade81

    upgrade81 Premium Member Premium Member

    233
    14
    18
    Sep 5, 2016
    Switzerland
    Ratings:
    +21
    Local Time:
    3:44 AM
    1.17
    10.3
    enabling sendfile does it less.

    Here are the errors from the chrome console.

    some logs error.log
    Code (Text):
    2019/09/02 19:44:38 [crit] 24596#24596: *3357 aio read "/home/nginx/domains/xxxxxxxx.com/public/wp-content/uploads/2019/09/salvini-renzi-728x409.jpg" failed (14: Bad address) while sending response to client, client: 5.89.203.243, server: xxxxxxxx.com, request: "GET /wp-content/uploads/2019/09/salvini-renzi-728x409.jpg HTTP/2.0", host: "www.xxxxxx
     

    Attached Files:

    • Informative Informative x 1