Join the community today
Become a Member

Beta Branch Centmin Mod Nginx 1.21.5+ PCRE2 Beta Testing

Discussion in 'Beta release code' started by eva2000, Dec 30, 2021.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x
    Nginx folks have released Nginx v1.21.5 with PCRE2 v10.x library support to replace the older PCRE v8.45 library support. For Centmin Mod 123.09beta01 and higher to support PCRE2, some major changes in Nginx and Nginx modules and dependencies' install routines are needed for the long term. These changes will need beta testing so this thread will outline how you can help beta test the changes before official release and commit to Centmin Mod 123.09beta01.

    All relevant threads on the forum will be tagged with nginx pcre2 tag, so you can follow any relevant threads posted on the forum.

    Nginx PCRE2 Beta Testing Steps



    The beta testing will be done via 2 steps for Nginx 1.21.5+ or higher.


    Step 1. Test existing PCRE v8.45 and all Nginx dependencies with the changed installation location from PREFIX = /usr/local to PREFIX = /usr/local/nginx-dep.

    Step 2. Test switching from existing PCRE v8.45 to newer PCRE2 v10.x supported in Nginx 1.21.5+. PCRE2 v10.x will also be installed at new PREFIX = /usr/local/nginx-dep location.

    Step 1 - Test Existing PCRE v8.45 New PREFIX Install Path Setup



    Apply update patch to local Centmin Mod 123.09beta01 using these commands or via pcre2patch command (read further below):
    Code (Text):
    # update local Centmin Mod code first
    cmupdate
    cd /usr/local/src/centminmod
    # remove inc/sync.inc file
    rm -f /usr/local/src/centminmod/inc/sync.inc
    # download patch
    wget -4 https://gist.github.com/centminmod/b7366d8b9a509f8d0da1f95914a167a7/raw/pcre2dirs.patch -O /usr/local/src/centminmod/pcre2dirs.patch
    # apply patch
    patch -p1 < pcre2dirs.patch
    # run centmin.sh menu option 4
    ./centmin.sh

    Or install the pre2patch script/command via command which will install a shell script at /usr/local/bin/pcre2patch with the above pcre2dirs.patch update commands already added.
    Code (Text):
    wget -4 https://gist.githubusercontent.com/centminmod/b7366d8b9a509f8d0da1f95914a167a7/raw/pcre2patch.sh -O /usr/local/bin/pcre2patch && chmod +x /usr/local/bin/pcre2patch

    So in SSH, running the command = pcre2patch will auto apply the pcre2dirs.patch to your system and auto pull the latest pcre2dirs.patch when the command is ran. Then you can run centmin.sh menu option 4.

    Verify working changed PREFIX directory locations by checking dynamic shared libraries loaded by Centmin Mod built Nginx binary. You can do this using below commands manually, or via the generated log output for these commands saved to /root/centminlogs/nginx_pcre_checks_XXXX.log where XXXX is date timestamped.

    The latest log would can be found using command below. Example log is at /root/centminlogs/nginx_pcre_checks_311221-002650.log
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)

    output of command
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)
    -rw-r--r-- 1 root root 3.5K Dec 31 00:28 /root/centminlogs/nginx_pcre_checks_311221-002650.log
    


    Manual command checks.
    Code (Text):
    ldd $(which nginx)

    Example output
    Code (Text):
    ldd $(which nginx)
            linux-vdso.so.1 =>  (0x00007ffcf3ffd000)
            libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f030202d000)
            libpcre.so.1 => /usr/local/nginx-dep/lib/libpcre.so.1 (0x00007f03023fa000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f0301e29000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0301c0d000)
            libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f03019d6000)
            libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f03017a4000)
            libatomic_ops.so.1 => /usr/local/nginx-dep/lib/libatomic_ops.so.1 (0x00007f03015a2000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f03011d4000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f0302261000)
            libfreebl3.so => /lib64/libfreebl3.so (0x00007f0300fd1000)

    looking for PCRE v8.45 and custom libatomic_ops 7.6.12 new PREFIX paths at /usr/local/nginx-dep/lib
    Code (Text):
    libpcre.so.1 => /usr/local/nginx-dep/lib/libpcre.so.1
    libatomic_ops.so.1 => /usr/local/nginx-dep/lib/libatomic_ops.so.1

    Check versions installed
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops

    If you have enabled optional Nginx Brotli (ngx_brotli) module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    

    If you have enabled optional ModSecurity Nginx module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion modsecurity

    If you have enabled optional Nginx Lua module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion luajit

    Example outputs
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre
    8.45

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops
    7.6.12

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    1.0.9
    
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    1.0.9
    

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion modsecurity
    3.0.6

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion luajit
    2.1.0-beta3
    

    Then let your new Nginx built server run for your traffic and see if there's any issues to report back.

    For example my Centmin Mod Blog's VPS server at Upcloud is running with Step 1 patched Centmin Mod Nginx 1.21.5 with PCRE at the new install PREFIX paths and the nginx_pcre_checks log is at /root/centminlogs/nginx_pcre_checks_311221-013616.log
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)
    -rw-r--r-- 1 root root 3.4K Dec 31 01:41 /root/centminlogs/nginx_pcre_checks_311221-013616.log
    

    contents of that log
    Code (Text):
    cat /root/centminlogs/nginx_pcre_checks_311221-013616.log
    
    nginx -V
    nginx version: nginx/1.21.5 (311221-013802-centos7-kvm-6a85c75-br-9aec15e)
    built by gcc 10.2.1 20210130 (Red Hat 10.2.1-11) (GCC)
    built with OpenSSL 1.1.1m  14 Dec 2021
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/nginx-dep/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/nginx-dep/lib -fuse-ld=gold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/nginx-dep/include -m64 -march=x86-64 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wno-pointer-sign -Wimplicit-fallthrough=0 -Wno-missing-profile -Wno-implicit-function-declaration -Wno-int-conversion -Wno-unused-result -Wno-unused-result -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=311221-013802-centos7-kvm-6a85c75-br-9aec15e --with-compat --without-pcre2 --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --add-dynamic-module=../njs/nginx --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5.1 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.62 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.19 --add-module=../srcache-nginx-module-0.32 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1m --add-module=../nginx-ssl-fingerprint --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold'
    
    nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    
    ldd /usr/local/sbin/nginx
            linux-vdso.so.1 =>  (0x00007ffdd5387000)
            libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f6230652000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f623044e000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6230232000)
            libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f622fffb000)
            libpcre.so.1 => /usr/local/nginx-dep/lib/libpcre.so.1 (0x00007f6230a24000)
            libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f622fdcb000)
            libatomic_ops.so.1 => /usr/local/nginx-dep/lib/libatomic_ops.so.1 (0x00007f622fbc9000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f622f7fb000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f6230886000)
            libfreebl3.so => /lib64/libfreebl3.so (0x00007f622f5f8000)
    
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre
    8.45
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops
    7.6.12
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    1.0.9
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    1.0.9
    


    Step 2 - Test PCRE2 v10.39 With New PREFIX Install Path Setup



    Once adequate Step 1 testing has been done, will move onto Step 2 :) I can not stress how important it is to test step 1 above because if Step 2 testing of PCRE2 for Nginx doesn't work, you'd need to fall back to the Step 1 configuration when I publicly release these updates. So if Step 1 isn't working properly and Step 2 doesn't work with PCRE2, then you won't have a working Nginx server!

    To switch to PCRE2 support for Nginx 1.21.5+, you need to update your persistent config file at /etc/centminmod/custom_config.inc (if it doesn't exist create it) and add the variable to enable Nginx PCRE2 support.
    Code (Text):
    NGINX_PCRE_TWO='y'

    Then run centmin.sh menu option 4 and update/recompile to Nginx version 1.21.5 or higher which support PCRE2.
    Verify working changed PREFIX directory locations by checking dynamic shared libraries loaded by Centmin Mod built Nginx binary. You can do this using below commands manually, or via the generated log output for these commands saved to /root/centminlogs/nginx_pcre_checks_XXXX.log where XXXX is date timestamped.

    The latest log would can be found using command below. Example log is at /root/centminlogs/nginx_pcre_checks_311221-002650.log
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)

    output of command
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)
    -rw-r--r-- 1 root root 3.5K Dec 31 00:28 /root/centminlogs/nginx_pcre_checks_311221-002650.log
    


    Manual command checks.
    Code (Text):
    ldd $(which nginx)

    Example output
    Code (Text):
    ldd $(which nginx)
            linux-vdso.so.1 =>  (0x00007fff0b668000)
            libpcre2-8.so.0 => /usr/local/nginx-dep/lib/libpcre2-8.so.0 (0x00007f7635488000)
            libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f76350c3000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f7634ebf000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7634ca3000)
            libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f7634a6c000)
            libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f763483a000)
            libatomic_ops.so.1 => /usr/local/nginx-dep/lib/libatomic_ops.so.1 (0x00007f7634638000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f763426a000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f76352f7000)
            libfreebl3.so => /lib64/libfreebl3.so (0x00007f7634067000)
    

    looking for PCRE v10.39+ and custom libatomic_ops 7.6.12 new PREFIX paths at /usr/local/nginx-dep/lib
    Code (Text):
    libpcre2-8.so.0 => /usr/local/nginx-dep/lib/libpcre2-8.so.0
    libatomic_ops.so.1 => /usr/local/nginx-dep/lib/libatomic_ops.so.1

    Check versions installed
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre2-8
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops

    If you have enabled optional Nginx Brotli (ngx_brotli) module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    

    If you have enabled optional ModSecurity Nginx module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion modsecurity

    If you have enabled optional Nginx Lua module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion luajit

    Example outputs
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre2-8
    10.39

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops
    7.6.12

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    1.0.9
    
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    1.0.9
    

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion modsecurity
    3.0.6

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion luajit
    2.1.0-beta3
    

    Then let your new Nginx built server run for your traffic and see if there's any issues to report back.


    Reverting Beta Patches



    To revert the applied beta patches, you can just run cmupdate command and that will reset your local Centmin Mod code back to the most current state for 123.09beta01 branch. Then run centmin.sh menu option 4 to recompile Nginx to revert your Nginx install configuration to pre-beta testing state.
     
    Last edited: Jan 2, 2022
  2. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x

    Feedback & Bug Reports



    For feedback or bug reports, you can start your own specific new thread in
    Beta release code subforum :)

    When you start a new thread, please provide the following info and output from these commands. You can do this using below commands manually, or via the generated log output for these commands saved to /root/centminlogs/nginx_pcre_checks_XXXX.log where XXXX is date timestamped after a centmin.sh menu option 4 Nginx run.

    The latest log would can be found using command below. Example log is at /root/centminlogs/nginx_pcre_checks_311221-002650.log
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)

    output of command
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_pcre_checks | tail -1)
    -rw-r--r-- 1 root root 3.5K Dec 31 00:28 /root/centminlogs/nginx_pcre_checks_311221-002650.log
    


    Manual command checks. Wrap nginx -v output in quote BBCODE so line breaks make it easier to view
    Code (Text):
    nginx -V

    Code (Text):
    nginx -t

    Code (Text):
    ldd $(which nginx)

    Wrap following commands' outputs in CODE or CODEB BBCODE tags for easier viewing

    For PCRE v8.45 step 1 tests
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre
    

    For PCRE2 v10.x step 2 tests
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libpcre2-8
    

    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion atomic_ops
    

    If you have enabled optional Nginx Brotli (ngx_brotli) module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlienc
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion libbrotlidec
    

    If you have enabled optional ModSecurity Nginx module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion modsecurity

    If you have enabled optional Nginx Lua module
    Code (Text):
    PKG_CONFIG_PATH='/usr/local/nginx-dep/lib/pkgconfig/' pkg-config --modversion luajit



    Nginx Compile Issues

    If the issue is that your Nginx recompile via centmin.sh menu option 4 failed to complete or errors out, then I'll need your Nginx upgrade log which is automatically created everytime you run centmin.sh menu option 4. You can find the latest Nginx upgrade log using command
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_upgrade | tail -1)

    example output shows latest generated Nginx upgrade log is at /root/centminlogs/centminmod_123.09beta01.b782_301221-073103_nginx_upgrade.log
    Code (Text):
    ls -lAh /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_upgrade | tail -1)
    -rw-r--r-- 1 root root 2.5M Dec 30 07:33 /root/centminlogs/centminmod_123.09beta01.b782_301221-073103_nginx_upgrade.log
    


    Sanitized Nginx Upgrade Log

    Then santize the Nginx upgrade log removing any identifying Nginx vhost site domain config files that maybe listed in the log.

    This command below is a single command and will remove all mentions of /usr/local/nginx/conf/conf.d from the Nginx upgrade log and create a sanitized log at /root/centminlogs/pcre2log.log
    Code (Text):
    egrep -v '/usr/local/nginx/conf/conf.d|Copyright (C)|This is free software|MERCHANTABILITY' /root/centminlogs/$(ls -rt /root/centminlogs/ | grep nginx_upgrade | tail -1) | sed -e "s|$HOSTNAME|hostname|g" > /root/centminlogs/pcre2log.log
    

    You can then optionally compress the pcre2log.log
    Code (Text):
    pigz -9k /root/centminlogs/pcre2log.log

    Compressed log will be at /root/centminlogs/pcre2log.log.gz which you can download it to share.
    Code (Text):
    ls -lah /root/centminlogs/pcre2log.log*
    -rw-r--r-- 1 root root 1.7M Dec 31 01:50 /root/centminlogs/pcre2log.log
    -rw-r--r-- 1 root root  86K Dec 31 01:50 /root/centminlogs/pcre2log.log.gz
    


    Using Dropbox To Share Logs



    You can then share the compressed pcre2log.log.gz file via a file sharing service or signing up for a Dropbox basic user free account dedicated for Centmin Mod use at Dropbox Personal Plans for Individual Use - Dropbox :)

    Dropbox allows you to in your settings enable email to Dropbox feature. So Dropbox will provide you a custom email address where you can send files to and they will automatically be uploaded to Dropbox account under Email Attachments folder.

    So once you enable email to Dropbox feature. You can send the the compressed pcre2log.log.gz log via email. You assign to email_to variable your provided Dropbox email YOUR-EMAIL@addtodropbox.com replacing YOUR-EMAIL with your specific email. And then run the 2nd command to email the log as an attachment
    Code (Text):
    email_to='YOUR-EMAIL@addtodropbox.com'
    echo "pcre2log $(date)" | mail -s "pcre2log $(date)" -a /root/centminlogs/pcre2log.log.gz $email_to
    

    If you don't receive the attachment in your Dropbox Email Attachment folder, make sure sending emails from server is working properly as per https://community.centminmod.com/th...ver-email-doesnt-end-up-in-spam-inboxes.6999/.

    dropbox-email-attachments-01.png dropbox-email-attachments-02.png

    Generate and copy the shared link and post to your thread the link.

    dropbox-email-attachments-03.png

    You can also create your own dbsend shell script to make sending files to Dropbox easier. See https://community.centminmod.com/threads/using-dropbox-to-share-server-files.22327/#post-91368

    For posting code or output from commands to keep the formatting, you might want to use CODE tags for code
    How to use forum BBCODE code tags :)
     
    Last edited: Dec 31, 2021
  3. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x

    Known Nginx PCRE2 Incompatibility Issues



    Currently, Nginx 1.21.5+ with PCRE2 isn't compatible with optional Nginx Lua and ModSecurity Nginx modules as at Dec 31, 2021. Both modules are disabled by default unless you specifically enable them via persistent config set variables in /etc/centminmod/custom_config.inc.

    Nginx Lua PCRE2 Incompatibilty

    Centmin Mod optionally enabled Nginx Lua module from OpenResty manages to get to end of centmin.sh menu option 4 Nginx compile run, but the resulting Nginx binary has an error and is unable to start the Nginx server with error on Nginx config check
    Code (Text):
    nginx -t
    nginx: [emerg] dlopen() "/usr/local/nginx/modules/ngx_http_lua_module.so" failed (/usr/local/nginx/modules/ngx_http_lua_module.so: undefined symbol: pcre_malloc) in /usr/local/nginx/conf/dynamic-modules.conf:9
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
    

    Nginx restart error
    Code (Text):
    ngxrestart
    Restarting nginx (via systemctl):  Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
                                                               [FAILED]
    

    and journatlctl log for Nginx shows the same error message as above Nginx config check
    Code (Text):
    journalctl -u nginx --no-pager | sed -e "s|$HOSTNAME|hostname|g"| tail -7
    
    Dec 30 23:48:38 hostname systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server...
    Dec 30 23:48:38 hostname nginx[4223]: Starting nginx: nginx: [emerg] dlopen() "/usr/local/nginx/modules/ngx_http_lua_module.so" failed (/usr/local/nginx/modules/ngx_http_lua_module.so: undefined symbol: pcre_malloc) in /usr/local/nginx/conf/dynamic-modules.conf:9
    Dec 30 23:48:38 hostname nginx[4223]: [FAILED]
    Dec 30 23:48:38 hostname systemd[1]: nginx.service: control process exited, code=exited status=1
    Dec 30 23:48:38 hostname systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    Dec 30 23:48:38 hostname systemd[1]: Unit nginx.service entered failed state.
    Dec 30 23:48:38 hostname systemd[1]: nginx.service failed.
    

    Issue tracker at module do not support with PCRE2 on nginx 1.21.5 · Issue #1984 · openresty/lua-nginx-module

    ModSecurity PCRE2 Incompatibility

    Centmin Mod optionally enabled ModSecurity Nginx module compiles (with NGINX_MODSECURITY='y') of Nginx 1.21.5 with PCRE2 enabled will result in errors like below during Nginx make stage.
    Code (Text):
    -L/usr/local/nginx-dep/lib -Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/nginx-dep/lib -lpcre2-8 -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/nginx-dep/lib:/usr/local/zlib-cf/lib:/usr/local/nginx-dep/lib -flto=8 -fuse-ld=gold \
    -shared
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function ‘ngx_http_modsecurity_pcre_malloc_init’:
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:78:9: error: ‘pcre_malloc’ undeclared (first use in this function); did you mean ‘old_pcre_malloc’?
       78 |     if (pcre_malloc != ngx_http_modsec_pcre_malloc) {
          |         ^~~~~~~~~~~
          |         old_pcre_malloc
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:78:9: note: each undeclared identifier is reported only once for each function it appears in
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:82:25: error: ‘pcre_free’ undeclared (first use in this function); did you mean ‘old_pcre_free’?
       82 |         old_pcre_free = pcre_free;
          |                         ^~~~~~~~~
          |                         old_pcre_free
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function ‘ngx_http_modsecurity_pcre_malloc_done’:
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:102:9: error: ‘pcre_malloc’ undeclared (first use in this function); did you mean ‘old_pcre_malloc’?
      102 |         pcre_malloc = old_pcre_malloc;
          |         ^~~~~~~~~~~
          |         old_pcre_malloc
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:103:9: error: ‘pcre_free’ undeclared (first use in this function); did you mean ‘old_pcre_free’?
      103 |         pcre_free = old_pcre_free;
          |         ^~~~~~~~~
          |         old_pcre_free
    make[1]: *** [objs/addon/src/ngx_http_modsecurity_module.o] Error 1
    make[1]: *** Waiting for unfinished jobs....
    make[1]: Leaving directory `/svr-setup/nginx-1.21.5'
    make: *** [build] Error 2
    

    Issue tracker at Module compilation error with NGINX 1.21.5 · Issue #261 · SpiderLabs/ModSecurity-nginx
     
  4. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x

    pcre2dirs.patch Updates



    Throughout development testing, the pcre2dirs.patch will be updated and updates will be listed below.
    • December 30-31, 2021 - Initial released pcre2dirs.patch
    • January 1, 2022 - Prep support option to allow auto sending of Nginx upgrade logs to Dropbox account via either Rclone or dbsend script
    • January 2, 2022 - Add additional debug diagnostic info when Nginx configure stage centmin.sh menu option 4 upgrade/recompiles fail and error out. Also made sure Dropbox sent logs are triggered when Nginx configure/make stages error out as well.
    • January 4, 2022 - Updated various routines and more improvements
    • January 6, 2022 - Updated patch with various improvements and fixes
    • January 7, 2022 - Updated patch with many fixes thanks to beta testing feedback from @cloud9 and also renamed Nginx upgrade logs with prefix of private- and public- to denote which logs you can share publicly. The public-nginx-pcre2log named log is sanitized version removing references to your Nginx vhost domain sites.
    To test the updated patches you do the same Step 1 above commands or use the pcre2patch command script (read further down):
    Code (Text):
    # update local Centmin Mod code first
    cmupdate
    cd /usr/local/src/centminmod
    # remove inc/sync.inc file
    rm -f /usr/local/src/centminmod/inc/sync.inc
    # download patch
    wget -4 https://gist.github.com/centminmod/b7366d8b9a509f8d0da1f95914a167a7/raw/pcre2dirs.patch -O /usr/local/src/centminmod/pcre2dirs.patch
    # apply patch
    patch -p1 < pcre2dirs.patch
    # run centmin.sh menu option 4
    ./centmin.sh
    

    Or install the pre2patch script/command via command which will install a shell script at /usr/local/bin/pcre2patch with the above pcre2dirs.patch update commands already added.
    Code (Text):
    wget -4 https://gist.githubusercontent.com/centminmod/b7366d8b9a509f8d0da1f95914a167a7/raw/pcre2patch.sh -O /usr/local/bin/pcre2patch && chmod +x /usr/local/bin/pcre2patch

    So in SSH, running the command = pcre2patch will auto apply the pcre2dirs.patch to your system and auto pull the latest pcre2dirs.patch when the command is ran. Then you can run centmin.sh menu option 4.

    Reverting Beta Patches



    To revert the applied beta patches, you can just run cmupdate command and that will reset your local Centmin Mod code back to the most current state for 123.09beta01 branch. Then run centmin.sh menu option 4 to recompile Nginx to revert your Nginx install configuration to pre-beta testing state.
     
    Last edited: Jan 7, 2022
  5. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x

    Using Rclone + Dropbox To Share Nginx Upgrade Logs



    Centmin Mod 123.09beta01 already has an addons/rclone.sh script to install and configure Rclone allowing you to sync server side files with multiple Cloud storage providers which includes Dropbox. We can use Rclone instead of above dbsend script to send Nginx upgrade logs including sanitized pcre2log.log to a Dropbox account - preferably a separate Dropbox account used only for this specific purposes.

    The latest pcre2dirs.patch updates includes updated code to allow centmin.sh menu option 4 Nginx compilation runs to optionally send the generated Nginx logs to Dropbox account.

    Steps to enable this include:

    Step 1. Setting persistent config file /etc/centminmod/custom_config.inc variables to enable Dropbox auto sending of log files and to enable addons/rclone.sh to install.

    Add to persistent config file /etc/centminmod/custom_config.inc (create it if it doesn't exist)
    Code (Text):
    RCLONE_ENABLE='y'
    DROPBOX_SEND='y'
    


    Step 2. Sign up for a new Dropbox account (my referral link) allows me to add 500MB storage quota for each referral. Then create your own Dropbox App Id at Login - Dropbox as per https://rclone.org/dropbox/#get-your-own-dropbox-app-id instructions but also Generated access token and set no expiration for token.

    dropbox-app-id-create-01.png
    dropbox-app-id-create-02.png


    Step 3. Install Rclone via addons/rclone.sh and configure it. The resulting created Rclone config file will be at /root/.config/rclone/rclone.conf

    Code (Text):
    /usr/local/src/centminmod/addons/rclone.sh install
    

    The addons/rclone.sh install command will install latest rclone binary to /usr/sbin/rclone and then immediately prompt you to configure your first Rclone remote (storage provider i.e. Dropbox). Pay attention to the instructions for configuration below. Centmin Mod will look for the Rclone remote named = dropbox1 so make sure you use that for remote name.
    1. select n to create new remote named = dropbox1
    2. select 11 for storage option = Dropbox
    3. hit enter to leave blank at client_id prompt & client_secret prompt
    4. enter y for advanced config
    5. for OAuth Access Token prompt, enter the generated token from DropBox App Id creation in step 2 above
    6. hit enter to leave bank for auth_url, token_url etc or enter your own configurations until you reach auto config prompt
    7. enter n for auto config
    8. at already have a token refresh prompt, enter n
    9. at prompt if the config is okay, enter y
    10. then q to quit
    Then you should have a working Rclone remote named dropbox1 that allows you to send files on your server to your desired Dropbox account. Whenever you run this patched tests for PCRE2 Nginx upgrades via centmin.sh menu option 4, you will now automatically send the logs to your Dropbox account. The one we are most focused on is the sanitized pcre2log.log compressed *.gz log.

    Example usage from centmin.sh menu option 4 where RCLONE_ENABLE='y' + DROPBOX_SEND='y' and /usr/sbin/rclone exists and rclone dropbox1 remote exists. At end of Nginx recompiles, there would be this output
    Code (Text):
    dropbox_generate 010122-042039
    
    Send Nginx Upgrade Logs To Dropbox Account
    
    /usr/sbin/rclone copy nginx_pcre_checks_010122-042039.log.gz dropbox1:
    
    /usr/sbin/rclone copy centminmod_123.09beta01.b782_010122-042039_nginx_upgrade.log.gz dropbox1:
    
    /usr/sbin/rclone copy pcre2log-010122-042039.log.gz dropbox1:

    And you'll see this uploaded to your Dropbox account. You just select the pcre2log with date timestamp and can select the 3 horizontal dots to far right of the desired file to select the share option and create and copy the share link.

    cm-dropbox-rclone-01.png

    cm-dropbox-rclone-share-01.png

    cm-dropbox-rclone-share-02.png

    Note the default limits for max file size are 48-150MB, so more than enough for sending compressed Nginx upgrade logs.

    You can check the remotes using listremotes command
    Code (Text):
    rclone listremotes
    dropbox1:
    

    Listing currently uploaded Dropbox files which will be in a directory called Apps/yourchosen_app_folder_name. For this example the App folder name = cmlogs
    Code (Text):
    rclone ls dropbox1:
        80053 centminmod_123.09beta01.b782_010122-050644_nginx_upgrade.log.gz
        80126 centminmod_123.09beta01.b782_010122-061645_nginx_upgrade.log.gz
        80168 centminmod_123.09beta01.b782_010122-062337_nginx_upgrade.log.gz
        80124 centminmod_123.09beta01.b782_010122-063007_nginx_upgrade.log.gz
        80144 centminmod_123.09beta01.b782_010122-063438_nginx_upgrade.log.gz
        80068 centminmod_123.09beta01.b782_010122-063949_nginx_upgrade.log.gz
         1275 nginx_pcre_checks_010122-050644.log.gz
         1275 nginx_pcre_checks_010122-061645.log.gz
         1275 nginx_pcre_checks_010122-062337.log.gz
         1276 nginx_pcre_checks_010122-063007.log.gz
         1277 nginx_pcre_checks_010122-063438.log.gz
         1275 nginx_pcre_checks_010122-063949.log.gz
        78549 pcre2log-010122-050644.log.gz
        78622 pcre2log-010122-061645.log.gz
        78635 pcre2log-010122-062337.log.gz
      1881218 pcre2log-010122-063007-mold.log
      1881217 pcre2log-010122-063438-mold.log
      1880958 pcre2log-010122-063949-mold.log
    


    Update: latest example of Dropbox auto sent logs on centmin.sh menu option 4 nginx upgrade runs with renamed private and public Nginx upgrade logs denoting that the public one is sanitized with removal of identifying Nginx vhost domain name references etc. Thus suited for public sharing.

    Code (Text):
    dropbox_generate 070122-001249
    
    Send Nginx Upgrade Logs To Dropbox Account
    
    /usr/sbin/rclone copy nginx_pcre_checks_070122-001249.log.gz dropbox1:
    
    /usr/sbin/rclone copy private-cm_123.09beta01.b782_070122-001249_nginx_upgrade.log.gz dropbox1:
    
    /usr/sbin/rclone copy public-nginx-pcre2log-070122-001249.log.gz dropbox1:
    
    /usr/sbin/rclone copy centminmod_123.09beta01.b782_070122-001249_pcre2_install.log.gz dropbox1:
    
    /usr/sbin/rclone copy nginx_autoconf.err.070122-001249.log.gz dropbox1:
    
    /usr/sbin/rclone copy nginx_pcre2_download_070122-001249.log.gz dropbox1:
    


    cm-dropbox-rclone-updated-01.png
     
    Last edited: Jan 7, 2022
  6. eva2000

    eva2000 Administrator Staff Member

    48,416
    11,099
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,274
    Local Time:
    12:32 AM
    Nginx 1.21.x
    MariaDB 10.x
    As at January 27, 2022 AEST time, this beta testing is closed now. The new code outlined above has now been merged into the main Centmin Mod 123.09beta01 branch via commit at https://community.centminmod.com/th...upport-related-updates-in-123-09beta01.22397/. This includes updated support to optionally automatically sent Nginx upgrade logs to a specified Dropbox account as outlined here https://community.centminmod.com/threads/sending-nginx-upgrade-logs-to-dropbox-via-rclone.22402/

    By default step 1 above config is in place so the same older PCRE v8.45 is installed with Nginx but changed installation location from PREFIX = /usr/local to PREFIX = /usr/local/nginx-dep for new installs. Existing users can update via SSH command = cmupdate.

    Existing Beta Test Users Update



    Existing Centmin Mod 123.09beta01 users who did take part in beta testing outlined here (above). Need one extra step to remove /usr/local/src/centminmod/inc/sync.inc before cmupdate can run. So for those users, run these 3 commands below:
    Code (Text):
    cd /usr/local/src/centminmod
    rm -f /usr/local/src/centminmod/inc/sync.inc
    cmupdate
    
     
Thread Status:
Not open for further replies.