Discover Centmin Mod today
Register Now

Beta Branch update inc/nginx_configure.inc fix cloudflare zlib usage

Discussion in 'Centmin Mod Github Commits' started by eva2000, Oct 25, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    update inc/nginx_configure.inc fix cloudflare zlib usage

    nginx compiled with cloudflare zlib may result in unexpected nginx master process (not to be confused with nginx worker process) being locked at 100% cpu load during centmin.sh menu option 4 recompile/upgrades and when you access server IP address during the nginx recompile/upgrade. This update seems to fix the problem.

    Upgrading Centmin Mod Code to Latest Version



    Getting Started Guide step 19 outlines also how to keep Centmin Mod code updated or how to switch version branches or you can run cmupdate command that was recently added.

    Centmin Mod LEMP stack's script code is constantly updated for improvements, bug fixes and security fixes so keeping the Centmin Mod code up to date is important.

    For 123.09beta01 and higher that means running SSH command = cmupdate and then re-enter /usr/local/src/centminmod and re-run centmin.sh menu.

    example SSH command = cmupdate run

    Code (Text):
    cmupdate
    Saved working directory and index state WIP on 123.09beta01: 85fae5e update nginx.conf in 123.09beta01
    HEAD is now at 85fae5e update nginx.conf in 123.09beta01
    Updating 85fae5e..4d3149e
    Fast-forward
     addons/ffmpeg.sh        |  2 +-
     centmin.sh              |  2 +-
     config/nginx/nginx.conf |  1 +
     inc/downloads.inc       |  2 +-
     inc/nginx_configure.inc |  2 +-
     inc/nginx_patch.inc     |  2 +-
     inc/openssl_install.inc | 20 +++++++++++---------
     7 files changed, 17 insertions(+), 14 deletions(-)
    


    For full details read the following links:
    Upgrading Centmin Mod involves 2 parts.
    1. Upgrading the actual Centmin Mod code outlined at Upgrade Centmin Mod. This is heart of Centmin Mod where the code is the engine that runs centmin.sh shell based menu and all the automation you're accustomed to. You can easily update within a Centmin Mod version branch or switch version branches via centmin.sh menu option 23 outlined here.
    2. Upgrade software that Centmin Mod installed or manages. For this part following outline at How to upgrade Centmin Mod software installed on your server.
    Continue reading...

    123.09beta01 branch
     
    Last edited: Oct 25, 2018
    • Like Like x 1
  2. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Is this the issue I encounter few weeks ago?
    When I run menu 4 to recompile nginx sites will not respond until it's done.
     
  3. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    That seems to be issue this commit fixes for @Matt. Though I haven't been able to reproduce it 100%, in my case normal nginx vhost domains work fine during nginx recompile (centmin.sh menu option 4). But only direct access to server IP address will lock nginx master process at 100% cpu usage and then normal nginx vhosts won't work until nginx recompile finishes. If I do not access direct server IP address during nginx recompile, nginx master process for me doesn't lock at 100% cpu usage. But does for @Matt and you it seems

    But this committed fix update, fixes both @Matt and my issues during nginx recompiler routine.
     
    • Informative Informative x 1
  4. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Yeah seems to fix for me now.
    Thanks!



    Edit: Seems to happen still but somewhere in between recompile process.
     
    Last edited: Oct 25, 2018
  5. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    hmm guess needs more investigating
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    @rdan if that doesn't work try nginx zero downtime on the fly binary upgrades via persistent config variable NGINX_ZERODT='y'. If after Clouldflare zlib commit fix and still have a moment during actual nginx restart where sites aren't available, it could be related to number of nginx workers you have.

    NGINX_ZERODT='y' Beta Branch - Nginx Upgrade - zero downtime mode
     
  7. Matt

    Matt Moderator Staff Member

    791
    351
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +534
    Local Time:
    12:06 PM
    1.5.15
    MariaDB 10.2
    This has fixed the issues I was experiencing with the master processes locking up at 100%
     
    • Informative Informative x 1
  8. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Previously it was 4 worker process, now I set it to 8 max CPU.
    Maybe I'll use auto?
     
  9. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    sweet.. you mean the updated commit fix or you mean using NGINX_ZERODT='y' ?
    auto will use cpu max threads. But have you tried NGINX_ZERODT='y' yet ?
     
  10. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    I disable Cloudflare Zlib on custom config for now, and no issues anymore on nginx recompile.
    With CF Zlib, Nginx recompile takes 500+ms and without CF Zlib takes 180ms only.
     
  11. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Not yet, maybe later on dev VPS not on my live server as it's peak time now.
     
  12. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    You mean seconds (s) not milliseconds (ms) ? hmmm something ain't right if that is in seconds and Cloudflare zlib based nginx recompile takes that long. But I also recall you're disabling alot of nginx modules via persistent config file right ? could be related - so you can try nginx recompile with an empty persistent config file /etc/centminmod/custom_config.inc too.

    Might need one of your nginx upgrade logs for for further troubleshooting

    To troubleshoot, you need to check the full nginx upgrade log at /root/centminlogs and instructions under Sharing logs and errors heading for using Pastebin.com or Gists to share a sanitised version of the contents of the nginx_upgrade.log log. You can see full details at How to troubleshoot Centmin Mod initial install issues

    also what's output of
    Code (Text):
    nginx -t

    when you run centmin.sh menu option 4 there's a nginx upgrade log timestamped at /root/centminlogs

    if you type this command it lists all logs in date ascending order so latest log at bottom
    Code (Text):
    ls -lArt /root/centminlogs

    so copy the entire contents of latest nginx_upgrade log to gist.github.com or pastebin.com

    you can use grep to filter the logs, i.e. look for nginx_upgrade in log name
    Code (Text):
    ls -lahrt /root/centminlogs/ | grep nginx_upgrade
    -rw-r--r--  1 root root 3.2M Oct 11 15:55 centminmod_1.2.3-eva2000.09.001_111016-155345_nginx_upgrade.log
    -rw-r--r--  1 root root 672K Oct 11 22:06 centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    So the last nginx upgrade log was named centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log and located at /root/centminlogs/centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    then use cat command to output the contents of that log

    clear your ssh window buffer/screen and type
    Code (Text):
    cat /root/centminlogs/centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    then select and copy and paste output to pastebin.com or gist.github.com file to share. If your SSH client's scroll buffer isn't large enough using cat might not output the entire log file contents, so you may need to download the log and use local text editor to open and copy and paste.

    so only need content of one specific log, in this case most recent nginx_upgrade.log log

    yeah be sure to get nginx_upgrade.logs for CF zlib enabled and disabled so can compare.
     
    • Like Like x 1
  13. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Sorry, yes seconds.
     
  14. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Yes, only Brotli is enabled for me.
     
  15. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    I tried again now, without specifying cloudflare zlib on/off on custom config, so using what centmin.sh is default and works fine.

    But why client_body_timeout always reset to 60s after recompile?
     
  16. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Interesting so something in your persistent config file is causing issues. So nginx compile time not slower either ?

    seems one of my auto tuned optimisation routine needs an update :)
     
  17. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    My custom_config is intact as before except for CLOUDFLARE_ZLIB='y' line removed.
    2 live server just tested now and seems to work fine.
     
  18. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:06 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    centmin mod nginx default is 5s too - no auto tune there

    what was persistent config contents before ?
     
  19. rdan

    rdan Premium Member Premium Member

    4,369
    1,053
    113
    May 25, 2014
    Ratings:
    +1,524
    Local Time:
    8:06 PM
    Mainline
    10.2
    Yeah my mistake, I'm looking at lingering_time :D.

    Same but with
    CLOUDFLARE_ZLIB='y'
    NGINX_DYNAMICTLS='y'

    Now this 2 line I already removed.
     
    • Informative Informative x 1
  20. R0rke

    R0rke Member

    124
    17
    18
    Jun 2, 2016
    Iran
    Ratings:
    +28
    Local Time:
    4:06 AM
    1.11.1
    10.1
    I experiencing this problem for a long time too, but I keep ignoring it!
     
    • Informative Informative x 1
..