Welcome to Centmin Mod Community
Become a Member

Nginx PageSpeed Nginx Nginx uses all resources and then crashes when pagespeed is enabled

Discussion in 'Install & Upgrades or Pre-Install Questions' started by Volodymyr Lupu, Dec 12, 2017.

  1. Volodymyr Lupu

    Volodymyr Lupu New Member

    4
    1
    3
    Dec 12, 2017
    Ratings:
    +1
    Local Time:
    7:57 PM
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.13.7
    • PHP Version Installed: 7.1.12,
    • MariaDB MySQL Version Installed: 10.1.29
    • When was last time updated Centmin Mod code base ? : Just now
    • Persistent Config:
      Code:
      NGXDYNAMIC_BROTLI='y'
      NGINX_LIBBROTLI='y'
      CLANG='n'
      DEVTOOLSETSEVEN='y'
      NGINX_DEVTOOLSETGCC='y'
      LETSENCRYPT_DETECT='y'
      NGINX_PAGESPEED=y
      NGXDYNAMIC_NGXPAGESPEED='y'
      NGINX_HPACK='y'


    When enabled, even if with only:
    Code:
    pagespeed on;
    pagespeed FileCachePath "/var/cache/ngx_pagespeed/";
    (it is not even trying to optimize images, or even CSS/JS. It's simply on.)

    pagespeed will make Nginx use all the RAM and CPU, and then crash. Same happens if optimization options are enabled.
    [​IMG]
    (Image loads on the editor but not on the thread itself? Url : https://i.imgur.com/DaUG8CS.png)

    The VPS uses 4 E3-1270v6 cores(shared), 4gb ram, nvme drives.
    I will try to compile this without using GCC7 to see if anything changes. And then with only pagespeed, removing brotli, hpack, letsencrypt.
     
  2. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    could elaborate on what you mean crashes specifically providing examples of symptoms ?

    To troubleshoot Nginx and PHP-FPM issues you'd want to check the domain site's vhost access.log and error.log logs located within directory at /home/nginx/domains/yourdomain.com/logs. You can see a full overview at centminmod.com/configfiles.html

    FAQ item 19 has more info on all Centmin Mod relevant log files locations and how to use tail command to view a sample of the entries.

    Also post the contents of your site's nginx vhost http /usr/local/nginx/conf/conf.d/yourdomain.com.conf and/or /usr/local/nginx/conf/conf.d/yourdomain.com.ssl.conf in BBCODE CODE tags as outlined at How to use forum BBCODE code tags | Centmin Mod Community

    CPU load optimal is 1.0 per cpu thread so 8 cpu thread normal = 8x1 = 8.0. 16 cpu server normal is 16x1 = 16.0

    so cpu load of 8 for
    • 2 cpu server = 8/2 = 4.0 load average
    • 4 cpu server = 8/4 = 2.0 load average
    • 8 cpu server = 8/8 = 1.0 load average
    When you type top command, hit number 1 to break down per cpu stats

    Centmin Mod is provided as is, so troubleshooting load issues is left to end user to do. However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    If you're providing info on this forum, more info might be helpful
    1. What's your VPS/Server hardware specifications ? cpu type ? memory available ? disk space ?
    2. Who's your web host ? Different hosts have different limits for server resource usage and some are more restrictive that others so it could just be their resource usage policy you tripped which may or may not be restrictive. If restrictive, then real solution would be finding a better web host. If not restrictive, then it's finding out what caused high cpu load.
    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    For example what's output for these commands
    Code (Text):
    top -bn1
    

    Code (Text):
    ps aufxw
    

    Code (Text):
    sar -q
    

    Code (Text):
    sar -r
    

    Code (Text):
    sar -q -f /var/log/sa/sa$(date +%d -d yesterday)
    

    Code (Text):
    sar -r -f /var/log/sa/sa$(date +%d -d yesterday)
    

    for posting code you might want to use CODE tags for code How to use forum BBCODE code tags :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    for ngx_pagespeed did you follow instructions outlined at Nginx PageSpeed - CentminMod.com LEMP Nginx web stack for CentOS to turn it on after enabling ngx_pagepseed nginx module using SSH command
    Code (Text):
    pscontrol on
    

    and uncomment the 3 pagespeed include file lines in nginx vhost for the domain - so remove hash # in front of the 3 lines
    Code (Text):
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;

    output for command
    Code (Text):
    nginx -V
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  5. Volodymyr Lupu

    Volodymyr Lupu New Member

    4
    1
    3
    Dec 12, 2017
    Ratings:
    +1
    Local Time:
    7:57 PM
    I have been testing, and it seems Brotli is not playing well with PageSpeed.
     
  6. Volodymyr Lupu

    Volodymyr Lupu New Member

    4
    1
    3
    Dec 12, 2017
    Ratings:
    +1
    Local Time:
    7:57 PM
    Just confirmed it:

    LetsEncrypt + HPACK + GCC7 + :

    Brotli -> Fine
    PageSpeed -> Fine
    Brotli + PageSpeed -> No Bueno
     
  7. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    exact symptoms and issues ?

    ngx_pagespeed will serve static assets from pre compressed cache and that would be gzip, if you want ngx_pagespeed + brotli to work you may need to set in pagespeed.conf (via pscontrol edit command)
    Code (Text):
    pagespeed HttpCacheCompressionLevel 0;
    

    and restart nginx
    Code (Text):
    ngxrestart
    


    If you mean ngx_pagespeed + ngx_brotli is failing to compile/install via centmin.sh menu option 4. Then I have updated 123.09beta01 with a fix for GCC 6/7 compatibility Beta Branch - update ngx_brotli nginx module routine for GCC 6/7 compatibility
     
  8. rdan

    rdan Premium Member Premium Member

    4,255
    1,034
    113
    May 25, 2014
    Ratings:
    +1,486
    Local Time:
    2:57 AM
    Mainline
    10.2
    I have the same situation.
    When both are enabled, site doesn't load.
     
  9. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    interesting problem .. when was last time you recompiled nginx via centmin.sh menu option 4 ? was it before the latest nginx_broli updates on Dec 12th ~2 days ago Commits · centminmod/centminmod · GitHub ?
     
  10. rdan

    rdan Premium Member Premium Member

    4,255
    1,034
    113
    May 25, 2014
    Ratings:
    +1,486
    Local Time:
    2:57 AM
    Mainline
    10.2
    Recompile it today.
     
  11. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    what site web app in use ? xenforo ? cloudflare in front ? other persistent config file options ? output for
    Code (Text):
    nginx -V
    
     
  12. rdan

    rdan Premium Member Premium Member

    4,255
    1,034
    113
    May 25, 2014
    Ratings:
    +1,486
    Local Time:
    2:57 AM
    Mainline
    10.2
    yes XF.
    No.
    I got:
    Code:
    ENABLEMOTD='n'
    
    CUSTOMSERVERNAME='y'
    CUSTOMSERVERSTRING='nginx'
    
    AUTO_GITUPDATE='y'
    DISABLE_IPVSIX='y'
    
    NSD_DISABLED='y'
    MEMCACHED_DISABLED='y'
    PUREFTPD_DISABLED='y'
    
    NGXDYNAMIC_NGXPAGESPEED='y'
    
    NGINX_UPDATEMAINTENANCE='n'
    PHP_UPDATEMAINTENANCE='n'
    MARIADB_UPDATEMAINTENANCE='n'
    
    AUTOTUNE_CLIENTMAXBODY='n'
    NSD_INSTALL='n'              # Install NSD (DNS Server)
    NGXMODULE_ALTORDER='n'       # nginx configure module ordering alternative order
    NGINX_DEBUG='n'              # Enable & reinstall Nginx debug log nginx.org/en/docs/debugging_log.html & wiki.nginx.org/Debugging
    NGINX_HTTP2='y'              # Nginx http/2 patch https://community.centminmod.com/threads/4127/
    NGINX_MODSECURITY=n          # modsecurity module support https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#Installation_for_NGINX
    NGINX_REALIP='n'
    NGINX_RDNS='n'               # https://github.com/flant/nginx-http-rdns
    NGINX_NJS='n'                # nginScript https://www.nginx.com/blog/launching-nginscript-and-looking-ahead/
    NGINX_GEOIP='n'              # Nginx GEOIP module install
    NGINX_STUBSTATUS='n'         # http://nginx.org/en/docs/http/ngx_http_stub_status_module.html required for nginx statistics
    NGINX_SUB='n'                # http://nginx.org/en/docs/http/ngx_http_sub_module.html
    NGINX_ADDITION='n'           # http://nginx.org/en/docs/http/ngx_http_addition_module.html
    NGINX_IMAGEFILTER='n'        # http://nginx.org/en/docs/http/ngx_http_image_filter_module.html
    NGINX_PERL='n'               # http://nginx.org/en/docs/http/ngx_http_perl_module.html
    NGINX_XSLT='n'               # http://nginx.org/en/docs/http/ngx_http_xslt_module.html
    NGINX_CACHEPURGE='n'         # https://github.com/FRiCKLE/ngx_cache_purge/
    NGINX_ACCESSKEY='n'          #
    NGINX_HTTPCONCAT='n'         # https://github.com/alibaba/nginx-http-concat
    NGINX_THREADS='n'            # https://www.nginx.com/blog/thread-pools-boost-performance-9x/
    NGINX_STREAM='n'             # http://nginx.org/en/docs/stream/ngx_stream_core_module.html
    NGINX_STREAMGEOIP='n'        # nginx 1.11.3+ option http://hg.nginx.org/nginx/rev/558db057adaa
    NGINX_STREAMREALIP='n'       # nginx 1.11.4+ option http://hg.nginx.org/nginx/rev/9cac11efb205
    NGINX_STREAMSSLPREREAD='n'   # nginx 1.11.5+ option https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
    NGINX_RTMP='n'               # Nginx RTMP Module support https://github.com/arut/nginx-rtmp-module
    NGINX_FLV='n'                # http://nginx.org/en/docs/http/ngx_http_flv_module.html
    NGINX_MP4='n'                # Nginx MP4 Module http://nginx.org/en/docs/http/ngx_http_mp4_module.html
    NGINX_AUTHREQ='n'            # http://nginx.org/en/docs/http/ngx_http_auth_request_module.html
    NGINX_SECURELINK='n'         # http://nginx.org/en/docs/http/ngx_http_secure_link_module.html
    NGINX_FANCYINDEX='n'         # http://wiki.nginx.org/NgxFancyIndex
    NGINX_VHOSTSTATS='n'         # https://github.com/vozlt/nginx-module-vts
    NGINX_LIBBROTLI='y'          # https://github.com/google/ngx_brotli
    NGINX_LIBBROTLISTATIC='n'
    NGINX_PAGESPEED='y'          # Install ngx_pagespeed
    NGINX_PAGESPEEDGITMASTER='n' # Install ngx_pagespeed from official github master instead
    NGINX_PASSENGER='n'          # Install Phusion Passenger requires installing addons/passenger.sh before hand
    NGINX_WEBDAV='n'             # Nginx WebDAV and nginx-dav-ext-module
    NGINX_HTTPREDIS='n'          # Nginx redis http://wiki.nginx.org/HttpRedisModule
    ORESTY_HEADERSMORE='n'       # openresty headers more https://github.com/openresty/headers-more-nginx-module
    NGINX_STICKY='n'             # nginx sticky module https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng
    NGINX_UPSTREAMCHECK='n'      # nginx upstream check https://github.com/yaoweibin/nginx_upstream_check_module
    NGINX_OPENRESTY='n'          # Agentzh's openresty Nginx modules
    ORESTY_SETMISCGIT='n'        # use git master instead of version specific
    ORESTY_ECHOGIT='n'           # use git master instead of version specific
    
    LUAJIT_GITINSTALL='n'        # opt to install luajit 2.1 from dev branch http://repo.or.cz/w/luajit-2.0.git/shortlog/refs/heads/v2.1
    
    ORESTY_LUANGINX='n'             # enable or disable or ORESTY_LUA* nginx modules below
    NGX_LUAUPSTREAM='n'                # disable https://github.com/openresty/lua-upstream-nginx-module
    NGX_LUAUPSTREAMCACHE='n'           # disable https://github.com/cloudflare/lua-upstream-cache-nginx-module
    
    PHPGEOIP_ALWAYS='n'          # GeoIP php extension is always reinstalled on php recompiles
    PHPDEBUGMODE='n'             # --enable-debug PHP compile flag
    PHPFINFO='n'                 # Disable or Enable PHP File Info extension
    PHPPCNTL='n'                 # Disable or Enable PHP Process Control extension
    PHPINTL='n'                  # Disable or Enable PHP intl extension
    PHPRECODE=n                  # Disable or Enable PHP Recode extension
    PHPSNMP='n'                  # Disable or Enable PHP SNMP extension
    PHPIMAGICK='y'               # Disable or Enable PHP ImagicK extension
    PHPMAILPARSE='n'             # Disable or Enable PHP mailparse extension
    PHPIONCUBE='n'               # Disable or Enable Ioncube Loader via addons/ioncube.sh
    POSTGRESQL='n'               # set to =y to install PostgreSQL 9.4 server, devel packages and pdo-pgsql PHP extension
    
    NGINX_VHOSTSSL='n'           # enable centmin.sh menu 2 prompt to create self signed SSL vhost 2nd vhost conf
    NGINXBACKUP='n'
    
    ### Nginx SSL options
    OPENSSL_VERSION='1.0.2n'
    LIBRESSL_SWITCH='n'
    
    MEMCACHED_INSTALL='n'         # Install Memcached
    PHPREDIS='y'                # redis PHP extension install
    PHPMONGODB='n'              # MongoDB PHP extension install
    PHP_FTPEXT='n'              # ftp PHP extension
    PHP_MEMCACHE='n'            # memcache PHP extension
    PHP_MEMCACHED='n'           # memcached PHP extension
    PHP_VERSION='7.1.12'        # Use this version of PHP
    ZOPCACHEDFT='y'
    PHP_EXTRAOPTS=""
    
     
  13. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    looks good persistent config wise though make sure you didn't enable ngx_pagespeed to use memcached server for caching as you have memcached server disabled it seems.

    i just tested 123.09beta01 with centos 7.4 64bit and ngx_brotli + ngx_pagespeed and looks okay to me no crashes just for test https static html page at least

    brotli-pagespeed-141217-01.png
     
    • Like Like x 1
  14. noly

    noly Premium Member Premium Member

    93
    13
    8
    Jul 24, 2017
    Germany
    Ratings:
    +25
    Local Time:
    8:57 PM
    1.13.x
    10.x
    Debugged a little deeper and can confirm @Volodymyr Lupu 's catch of the combination of brotli, let's encrypt and pagespeed causes a huge loading delay which sometimes leads to a site crash.
    Adding..

    Code:
    pagespeed HttpCacheCompressionLevel 0;
    ...to pagespeed.conf didn’t help.

    Debugging:
    I have a https site, three more http and phpmyadmin open. Switching in between and reloading them causes at some point the server to go down. Log file show signal 11 and a few signal 9s. And trying to display..

    Code:
    _DOMAIN_/pagespeed_global_admin/message_history
    shows the error:

    Code:
    Writing to mod_pagespeed_message failed. Verify that MessageBufferSize is not set to 0 in pagespeed.conf.
    But there is no such setting in the configuration.

    And yes, i can also confirm that when brotli and pagespeed are both enabled and only http sites are loaded then everything seems fine. But the moment the https site comes to the game with a fresh reload the problems start.
    While debugging i saw that pagespeed handles PMA as well. Mentioned that in another thread already though without getting a response.

    Shouldn't PMA be by default excluded from pagespeed?

    I have commented this issue on ngx_pagespeed Github where a user catched the same "Worker process exited on signal 11"-error like i did on /usr/local/nginx/logs/error.log

    Btw. adding
    Code:
    pagespeed HttpCacheCompressionLevel 0;
    to pagespeed.conf didn't help.


    pagespeed off; vs. pagespeed unplugged; issue

    My pagespeed version is X-Page-Speed: 1.12.34.3-0 and when i turn pagespeed off with
    Code:
    pscontrol off
    then
    Code:
    pagespeed unplugged;
    is added to pagespeedadmin.conf.
    Now the https sites load normal but PMA still crashes. But when changing pagespeed unplugged; to
    Code:
    pagespeed off;
    in the pagespespeedadmin.conf (or adding directly to phpmyadmin_ssl.conf) then PMA and all sites load normal. Doesn't that confirm the idea of a by default pagespeed excluded PMA in centmin?

    Beware that if pagespeedadmin.conf is manually changed to
    Code:
    pagespeed off;
    then pscontrol on has no effect on the file anymore.

    One of the ngx_developer adviced to create a new issue on Github:

    Nginx, brotli, ssl combination crashes nginx · Issue #1518 · pagespeed/ngx_pagespeed · GitHub
     
    Last edited: Dec 21, 2017
  15. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    123.09beta01 should of default to ngx_pagespeed 1.13.35.1-beta now on latest 123.09beta01 update. Make sure you have updated 123.09beta01 code via centmin.sh menu option 23 submenu option 2, exit centmin.sh. Then re-run centmin.sh menu option 4 to recompile nginx with ngx_pagespeed 1.13.35.1-beta otherwise unplugged variable will crash nginx + ngx_pagespeed when version is less than 1.13.35.1-beta

    see 2nd post at Nginx PageSpeed - Nginx PageSpeed 1.13.35.1-beta Released
     
  16. noly

    noly Premium Member Premium Member

    93
    13
    8
    Jul 24, 2017
    Germany
    Ratings:
    +25
    Local Time:
    8:57 PM
    1.13.x
    10.x
    A few days ago you responded to my question on how to downgrade pagespeed back to the stable version that "123.09beta01 had an update to update pscontrol for 1.13.35.1 changes and to support downgrading back to previous version of ngx_pagespeed.".

    Am i understanding you right that changing back to the stable version will lead to crashes and is also the purpose of my above described issues?
     
  17. noly

    noly Premium Member Premium Member

    93
    13
    8
    Jul 24, 2017
    Germany
    Ratings:
    +25
    Local Time:
    8:57 PM
    1.13.x
    10.x
    I can confirm that i update centmin, yum, etc. on a daily basis and i don't post an issue if centmin.sh isn't uptodate nor before searching this forum and/or on google for a solution.

    Edit 2:
    Yes, updating to v.1.13.35.1-0 fixes the issue BUT isn't "pagespeed unplugged;" compatible since v.1.12.34.1 ?
     
    Last edited: Dec 21, 2017
  18. noly

    noly Premium Member Premium Member

    93
    13
    8
    Jul 24, 2017
    Germany
    Ratings:
    +25
    Local Time:
    8:57 PM
    1.13.x
    10.x
    Could you please also tell me if there is anything speaking against that?
     
  19. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Well straight from post #2
    it should by default but if you're using 1.12.34.1 and using pagespeed unplugged that won't work to turn it off by default. Updating to 1.13.35.1-beta should allow pagespeed to see that unplugged = pagespeed off for phpmyadmin
     
  20. eva2000

    eva2000 Administrator Staff Member

    35,995
    7,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,177
    Local Time:
    4:57 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah centmin mod 123.09beta01 has a downgrade support routine which i tested to work. But seems it isn't working in your case
     
..