Welcome to Centmin Mod Community
Register Now

Sysadmin Nginx 100% use Cpu

Discussion in 'System Administration' started by upgrade81, Apr 5, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    45,676
    10,371
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,094
    Local Time:
    5:38 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yeah the thing is to see about a different server to see if it isn't an issue related to your specific server seeing as all 3x VPS/VMs are on same host node itself. It could be many factors including mix of custom kernel used and your hardware and ngx_brotli. But segfaults reported are for ngx_brotli so definitely focus there first.

    One other combination would be seeing if centos default kernel 3.10 and ngx_brotli to rule out custom 4.17+ kernel and ngx_brotli with your specific server hardware/cpu.
     
  2. upgrade81

    upgrade81 Premium Member Premium Member

    266
    16
    18
    Sep 5, 2016
    Italy
    Ratings:
    +27
    Local Time:
    8:38 AM
    1.17
    10.3
    Two weeks ago one of the VMs completely went into a freeze state.

    Connecting with kvm (ssh was timeout) this is what we saw.

    grabilla.g41824.png

    Done at the provider, the day after he planned an intervention on that host hosting the vm, and updated kernel and librv.

    Nevertheless, nginx did not resolve. There have no longer been complete freezes.
    For now on the VM with brotli disabled, everything is OK.
    before singing victory, we wait 48 hours.

    I wonder if it was really brotli, because a simple compression can give so many problems?
     
  3. upgrade81

    upgrade81 Premium Member Premium Member

    266
    16
    18
    Sep 5, 2016
    Italy
    Ratings:
    +27
    Local Time:
    8:38 AM
    1.17
    10.3
    I confirm, disabling brotli, everything works great.
    50 hours of 100% continuous uptime.

    I understand that google does not touch the brotli code for years, but is this problem possible?
     
  4. eva2000

    eva2000 Administrator Staff Member

    45,676
    10,371
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,094
    Local Time:
    5:38 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Thanks for the update. Have you tried with CentOS stock linux 3.10 kernel + ngx_brotli enabled to rule out 4.17+ kernel ?

    Centmin Mod doesn't use outdated Google ngx_brotli at google/ngx_brotli. Instead Centmin Mod uses more up to date fork at eustas/ngx_brotli developed by one of the developers of Brotli itself at google/brotli

    Though forked version is a few versions behind brotli with 1.0.5, 1.0.6 and 1.0.7 pending brotli 1.0.6+ · Issue #24 · eustas/ngx_brotli
     
  5. upgrade81

    upgrade81 Premium Member Premium Member

    266
    16
    18
    Sep 5, 2016
    Italy
    Ratings:
    +27
    Local Time:
    8:38 AM
    1.17
    10.3
    Well ... there's definitely something wrong!
    New dedicated server, new data center, new clean installation of centos 7 with centminmod with the following configuration:

    Code:
    PHP_PGO = 'y'
    LETSENCRYPT_DETECT = 'y'
    NGINX_IPV = 'y'
    NGXDYNAMIC_BROTLI = 'y'
    NGINX_LIBBROTLI = 'y' # https://github.com/eustas/ngx_brotli
    NGINX_LIBBROTLISTATIC = 'y'
    NGXDYNAMIC_NGXPAGESPEED = 'n'
    NGINX_DEVTOOLSETGCC = 'y'
    GENERAL_DEVTOOLSETGCC = 'y'
    CLANG = 'n'
    LIBRESSL_SWITCH = 'n'
    NGINX_UPDATEMAINTENANCE = 'n'
    PHP_UPDATEMAINTENANCE = 'n'
    MARIADB_UPDATEMAINTENANCE = 'y'
    CUSTOMSERVERNAME = 'y'
    CUSTOMSERVERSTRING = 'nginx'
    DUALCERTS = 'y' #dual cert RSA + ECDSA
    NGINX_DYNAMICTLS = 'n'
    CLOUDFLARE_ZLIB = 'y'
    NGINX_HPACK = 'y'
    NGINX_ZLIBCUSTOM = 'y'
    -----------

    Code:
    cPUInfo;
    processor: 0
    vendor_id: GenuineIntel
    cpu family: 6
    model: 60
    model name: Intel (R) Xeon (R) CPU E3-1271 v3 @ 3.60GHz
    stepping: 3
    microcode: 0x25
    cpu MHz: 4000.122
    cache size: 8192 KB
    physical id: 0
    siblings: 8
    core id: 0
    cpu cores: 4
    apicid: 0
    initial apicid: 0
    fpu: yes
    fpu_exception: yes
    cpuid level: 13
    wp: yes
    flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx east tm2 SSSE3 sdbg FMA CX16 XTPR PDCM pcid sse4_1 sse4_2 X2apic movbe POPCNT tsc_deadline_timer aes xsave AVX f16c rdrand lahf_lm aBM epb ssbd IBRS ibpb stibp tpr_shadow vnmi FlexPriority ept vpid fsgsbase tsc_adjust BMI1 AVX2 SMEP bmi2 erms invpcid xsaveopt dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1d
    bogomips: 7200.45
    clflush size: 64
    cache_alignment: 64
    address sizes: 39 bits physical, 48 bits virtual
    power management:
    Code:
    uname -r
    3.10.0-957.1.3.el7.x86_64
    
    NGINX still goes 100% on 1 single core.
    I have a single site on this server that does very little traffic.
    there is a BUG, as big as a house, to be solved.
     
  6. eva2000

    eva2000 Administrator Staff Member

    45,676
    10,371
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,094
    Local Time:
    5:38 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    and nginx debug reveals ? and only when nginx brotli enabled or even when disabled ? if it's nginx brotli related then bug has to be resolved with ngx_brotli developers as Centmin Mod only uses what is provided. But it could be related to your site specifically and what is being served too as I have 100s of Centmin Mod servers running 123.09beta01 with ngx_brotli without this 100% cpu load issues.

    nginx debug mode logging would give more clues.

    and output for
    Code (Text):
    cminfo top
    


    and same troubleshooting tips discussed early in this thread starting from here
     
  7. eva2000

    eva2000 Administrator Staff Member

    45,676
    10,371
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,094
    Local Time:
    5:38 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    oh i noticed you enabled NGINX_LIBBROTLISTATIC = 'y' - you should keep that disabled which is default. Enabling this will only enable nginx brotli static support and disable dynamic support so that static files pre-compressed with brotli are served but on the fly brotli compression is disabled.

    proper way to enable Nginx brotli support at Nginx - How to use Brotli compression for Centmin Mod Nginx web servers is via persistent config file /etc/centminmod/custom_config.inc set variables below:
    Code (Text):
    NGXDYNAMIC_BROTLI='y'
    NGINX_LIBBROTLI='y'
    
     
  8. upgrade81

    upgrade81 Premium Member Premium Member

    266
    16
    18
    Sep 5, 2016
    Italy
    Ratings:
    +27
    Local Time:
    8:38 AM
    1.17
    10.3
    Hello, I tried even just with these two, it does not go anyway.
    The sites are always different than basic wordpress, at this time 1 only vhost configured with wordpress + redis.
     
  9. eva2000

    eva2000 Administrator Staff Member

    45,676
    10,371
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,094
    Local Time:
    5:38 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    did you recompile nginx via centmin.sh menu option 4 afterwards ?
     
  10. upgrade81

    upgrade81 Premium Member Premium Member

    266
    16
    18
    Sep 5, 2016
    Italy
    Ratings:
    +27
    Local Time:
    8:38 AM
    1.17
    10.3
    Sure!

    I'm doing further tests.