Want to subscribe to topics you're interested in?
Become a Member

Nginx Dynamic Modules stopped working after update (1.11.09 to 1.11.10)

Discussion in 'Install & Upgrades or Pre-Install Questions' started by Simon Brown, Feb 21, 2017.

  1. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Hello,
    Sorry if this is posted in the wrong place...
    Centmin recently told me there was an update for Nginx from 1.11.09 to 1.11.10.
    I ran the update as instructed by the Centmin message and now none of the dynamic modules work. Nginx refuses to start-up.
    I get an error about the versions for the dynamic modules being wrong but I have no idea how to update them.

    nano /usr/local/nginx/conf/dynamic-modules.conf

    load_module "modules/ngx_http_image_filter_module.so";
    load_module "modules/ngx_http_fancyindex_module.so";
    load_module "modules/ngx_http_brotli_filter_module.so";
    load_module "modules/ngx_http_brotli_static_module.so";
    load_module "modules/ngx_stream_module.so";


    Here's the error message about the wrong version on a module:
    [​IMG]

    I've tried to comment out the dynamic modules one-by-one but the same version error just follows on to the next module.

    Can anyone offer any advice?

    • CentOS Version:CentOS 7 64bit
    • nginx version: 1.11.10
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.11.10
    • PHP Version Installed: 7.0.15
    • Persistent Config:
      NGXDYNAMIC_NGXPAGESPEED='n'
      NGINX_PAGESPEED='y'
      NGXDYNAMIC_BROTLI=y
      NGINX_LIBBROTLI=y
     
  2. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    what is output for
    Code (Text):
    nginx -V
    

    wrap in QUOTE bbcode tag

    and also what if you run centmin.sh menu option 4 with NGXDYNAMIC_NGXPAGESPEED='y' ?
     
  3. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    Also worth checking the nginx upgrade logs. To troubleshoot, you need to check the the nginx upgrade log at /root/centminlogs and instructions under Sharing logs and errors heading for using Pastebin.com or Gists to share a sanitised version of the contents of the nginx_upgrade.log log. You can see full details at How to troubleshoot Centmin Mod initial install issues

    also what's output of
    Code (Text):
    nginx -t

    when you run centmin.sh menu option 4 there's a nginx upgrade log timestamped at /root/centminlogs

    if you type this command it lists all logs in date ascending order so latest log at bottom
    Code (Text):
    ls -lArt /root/centminlogs

    so copy the entire contents of latest nginx_upgrade log to gist.github.com or pastebin.com

    you can use grep to filter the logs, i.e. look for nginx_upgrade in log name
    Code (Text):
    ls -lahrt /root/centminlogs/ | grep nginx_upgrade
    -rw-r--r--  1 root root 3.2M Oct 11 15:55 centminmod_1.2.3-eva2000.09.001_111016-155345_nginx_upgrade.log
    -rw-r--r--  1 root root 672K Oct 11 22:06 centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    So the last nginx upgrade log was named centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log and located at /root/centminlogs/centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    then use cat command to output the contents of that log

    clear your ssh window buffer/screen and type
    Code (Text):
    cat /root/centminlogs/centminmod_1.2.3-eva2000.09.001_111016-220515_nginx_upgrade.log

    then select and copy and paste output to pastebin.com or gist.github.com file to share. If your SSH client's scroll buffer isn't large enough using cat might not output the entire log file contents, so you may need to download the log and use local text editor to open and copy and paste.

    so only need content of one specific log, in this case most recent nginx_upgrade.log log
     
  4. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    nginx version: nginx/1.11.10
    built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
    built with LibreSSL 2.4.5
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-lrt -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -g -O3 -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_stub_status_module --with-http_secure_link_module --add-module=../nginx-module-vts --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-module=../ngx_pagespeed-1.12.34.2-beta --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.0 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0 --add-module=../set-misc-nginx-module-0.31 --add-module=../echo-nginx-module-0.60 --add-module=../redis2-nginx-module-0.13 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.17 --add-module=../srcache-nginx-module-0.31 --add-module=../headers-more-nginx-module-0.32 --with-pcre=../pcre-8.40 --with-pcre-jit --with-http_ssl_module --with-http_v2_module --with-openssl=../libressl-2.4.5
     
  5. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
  6. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

    It works because I've commented out the dynamic-modules.conf & brotli_inc.conf in the:
    nginx/conf/nginx.conf

    Here's the upgrade log for Nginx:
    gist:0e8c0b87f97e9ddd664a8ce5e118f63d ยท GitHub

    Thank you so much for your help :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    strange then looks good in nginx upgrade log

    try clearing out the dynamic modules in /usr/local/nginx/modules and then recompiling nginx via centmin.sh menu option 4
    Code (Text):
    cd /usr/local/nginx/modules
    rm -rf /usr/local/nginx/modules/*
    
     
  8. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Thanks for the tip but sadly it didn't work. I made sure the directory was empty after the did the 'rm'.
    I then ran option 4 and it all looked like it worked okay... I checked the modules directory and they had been reinstalled. When I ran 'nprestart' I get the same error as before:

    Feb 20 18:40:54 nginx[16762]: nginx: [emerg] module "/usr/local/nginx/modules/ngx_http_image_filter_module.so" version 1011010 instead of 1011009 in /usr/local/nginx/conf/dynamic-modules.conf:1

    Feb 20 18:40:54 nginx[16762]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed

    Feb 20 18:40:54 systemd[1]: nginx.service: control process exited, code=exited status=1

    Feb 20 18:40:54 systemd[1]: Failed to start The nginx HTTP and reverse proxy server.
     
  9. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    That's the weirdest thing I've seen for dynamic modules. First time I've seen this.

    You can temporarily for now disable image filter as a dynamic module and revert to using it as a static module via persistent config set variable
    Code (Text):
    NGXDYNAMIC_IMAGEFILTER='n'
    

    then recompile nginx via centmin.sh menu option 4
     
  10. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Always fun to be the first! :-/

    I'll give it a go now and let you know how it goes.
     
  11. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    you can also try clearing ccache compiler cache and see if that helps for a nginx recompile via centmin.sh menu option 4
    Code (Text):
    ccache -C
    
     
  12. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Okay, cleared the cache... One other thing is it's just the 'image filter'. Nginx is also having issues loading
    'brotli' dynamically.
     
  13. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    let's see what ccache clearing does first :)
     
  14. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Sadly clearing the cache didn't help... exactly the same issue... I'll try:
    NGXDYNAMIC_IMAGEFILTER='n'
     
  15. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    As I suspected, the error has moved to the next module in the chain:

    Code:
    module "/usr/local/nginx/modules/ngx_http_fancyindex_module.so" version 1011010 instead of 1011009 in /usr/local/nginx/conf/dynamic-modules.conf:1
     
  16. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    what happens if you stop nginx and start it instead of restart
    Code (Text):
    ngxstop
    ngxstart
    

    afterwards output for
    Code (Text):
    ps -afC nginx
    

    could be old nginx binary still running for some reason
     
  17. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Exactly the same issue:
    Code:
    module "/usr/local/nginx/modules/ngx_http_fancyindex_module.so" version 1011010 instead of 1011009 in /usr/local/nginx/conf/dynamic-modules.conf:1
    Code:
    UID        PID  PPID  C STIME TTY          TIME CMD
    root      2844 12654  0 19:33 pts/1    00:00:00 ps -afC nginx
    root      3002  2946  0 17:26 tty1     00:00:00 sudo -i
    root      3003  3002  0 17:26 tty1     00:00:00 -bash
    
     
  18. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    weird try downgrading to nginx 1.11.9 first with NGXDYNAMIC_IMAGEFILTER='n' commented out in persistent config so it's reenabled. Then make a copy of nginx 1.11.9 dynamic modules
    Code (Text):
    cp -a /usr/local/nginx/modules /usr/local/nginx/modules.copy
    

    then upgrade to nginx 1.11.10 and if you get errors, try copying the nginx 1.11.9 dynamic modules from /usr/local/nginx/modules.copy to /usr/local/nginx/modules directory and see if that works.

    Really strange stuff !
     
  19. Simon Brown

    Simon Brown Premium Member Premium Member

    22
    2
    3
    Feb 9, 2017
    Ratings:
    +5
    Local Time:
    5:05 AM
    1.11.9
    Just to keep you updated...

    I cleared the cache (ccache -C), cp the modules as suggested, rm the modules and then ran option 4 to downgrade to 1.11.9
    It worked this time. Last time I downgraded it didn't clear the error.
    I then tried to start Nginx again but the error was still there. I then 'rm' the modules and replaced them with the cp version. Then tried npstart and got exactly the same error.

    I've now downgraded to 1.11.9 again. At least it works :)
     
    • Informative Informative x 1
  20. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    3:05 PM
    Nginx 1.13.x
    MariaDB 5.5
    Thanks for the update, luckily Nginx 1.11.10 is just a regular non-security update so 1.11.9 is fine. I'll research this on my end and see if there's any other reports online for this as it's first time i seen this.