Discover Centmin Mod today
Register Now

Nginx [PATCH] Fix error when building a Nginx source RPM with custom Zlib - Nginx 1.13.7

Discussion in 'Nginx and PHP-FPM news & discussions' started by bassie, Dec 12, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    made some changes and renumber/versioned cloudflare zlib from 1.2.8 to 1.3.0 so it's library gets picked up first - seems to be alphanumerically the last version is picked if there's more than one version i.e. 1.2.11 over 1.2.8 or 1.3.0 over 1.2.8 so can keep previous version in place too
    Code (Text):
    ls -lahrt /usr/local/lib | grep libz
    -rwxr-xr-x   1 root root 115K Dec 13 03:54 libz.so.1.2.8
    -rw-r--r--   1 root root 143K Dec 13 05:48 libz.a
    -rwxr-xr-x   1 root root 111K Dec 13 05:48 libz.so.1.3.0
    lrwxrwxrwx   1 root root   13 Dec 13 05:48 libz.so.1 -> libz.so.1.3.0
    lrwxrwxrwx   1 root root   13 Dec 13 05:48 libz.so -> libz.so.1.3.0
    
    lsof | grep nginx | grep -v php-fpm | grep libz
    nginx     11733           root  mem       REG                9,2     113536   12089322 /usr/local/lib/libz.so.1.3.0
    nginx     11734          nginx  mem       REG                9,2     113536   12089322 /usr/local/lib/libz.so.1.3.0
    nginx     11735          nginx  mem       REG                9,2     113536   12089322 /usr/local/lib/libz.so.1.3.0
    nginx     11737          nginx  mem       REG                9,2     113536   12089322 /usr/local/lib/libz.so.1.3.0
    nginx     11738          nginx  mem       REG                9,2     113536   12089322 /usr/local/lib/libz.so.1.3.0
    

    still more testing though
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Interesting testing between standard zlib 1.2.11 and cloudflare zlib 1.2.8 renamed 1.3.0, encountered where nginx recompiles via centmin.sh menu option 4 will not pick up the correct zlib version if you test one version after the other out of the box. It seems the library directory containing libz.so.1.2.11 or libz.so.1.3.0 has 2 symlinks to them for libz.so and libz.so.1 but seems on subsequent recompiles symlink for libz.so.1 isn't updated and stuck on previous version symlink
    i.e. if i compiled with cloudflare zlib 1.2.8 renamed 1.3.0 and then recompile with zlib 1.2.11, the resulting directory would have
    Code (Text):
    ls -lahrt /usr/local/lib | grep libz
    -rwxr-xr-x   1 root root 111K Dec 13 07:14 libz.so.1.2.8
    -rwxr-xr-x   1 root root 111K Dec 13 07:35 libz.so.1.3.0
    -rw-r--r--   1 root root 144K Dec 13 07:38 libz.a
    -rwxr-xr-x   1 root root 115K Dec 13 07:38 libz.so.1.2.11
    lrwxrwxrwx   1 root root   14 Dec 13 07:40 libz.so.1 -> libz.so.1.3.0
    lrwxrwxrwx   1 root root   14 Dec 13 07:40 libz.so -> libz.so.1.2.11
    

    and nginx lsof output would report incorrectly zlib 1.3.0 in use when it should be 1.2.11

    so if i correct the libz.so.1 symlink on each zlib version recompiles, it is now reporting correct version
    Code (Text):
    ls -lahrt /usr/local/lib | grep libz
    -rwxr-xr-x   1 root root 111K Dec 13 07:14 libz.so.1.2.8
    -rwxr-xr-x   1 root root 111K Dec 13 07:35 libz.so.1.3.0
    -rw-r--r--   1 root root 144K Dec 13 07:38 libz.a
    -rwxr-xr-x   1 root root 115K Dec 13 07:38 libz.so.1.2.11
    lrwxrwxrwx   1 root root   14 Dec 13 07:40 libz.so.1 -> libz.so.1.2.11
    lrwxrwxrwx   1 root root   14 Dec 13 07:40 libz.so -> libz.so.1.2.11
    
     
    • Like Like x 1
  3. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  4. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    Probs for the Cloudflare Zlib addon.
    Hereby a few small enhancements as advice.

    - All modules are compiled with version number. Ie.
    --with-zlib=../zlib-1.2.11 and openssl=../openssl-1.1.0g

    Unfortunately not at Cloudflare Zlib (--with-zlib=../zlib-cloudflare).
    If a user has a problem, the user should report the currently used software according to the forum. Now the user can't. This gives a lot of unnecessary work for both user and mod to determine the currently used software in case of bug, errors et all.

    -- Please add --with-zlib=../zlib-cloudflare-1.3.0

    The configure script of Cloudflare Zlib writes about:
    "echo "CRC and SSE4.2 support is required" | tee -a configure.log
    leave 1"

    Please add a fail-back if SSE4.2 is not supported and the error messages is shown.

    -- if error then echo "CRC and SSE4.2 support is required, which your server cpu unfortunately, does not have. Choose for the default Zlib menu option and please continue" etc etc etc or something with the same purpose.
     
    Last edited: Dec 14, 2017
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    thanks made change to tack on a version number at end add CLOUDFLARE_ZLIBVER variable · centminmod/[email protected] · GitHub but strangest thing on make install part of zlib my SSH session always terminates at
    Code (Text):
    + make install
    cp libz.a /usr/local/lib
    chmod 644 /usr/local/lib/libz.a

    weirdest thing !

    make install debug log https://gist.github.com/centminmod/...7d2ef1f62902a5dccc4e2d/z-make-install-exit.md

    edit: seems ldconfig picks up /usr/local/lib path so means sshd service which uses zlib ends up using cloudflare zlib version and when you recompile nginx within same SSH session, the make install stage reaps/terminates the processes using that cloudflare zlib version = SSH session terminates. Workaround now is to set CLOUDFLARE_ZLIB='n', recompile nginx via centmin.sh menu option 4 to get zlib 1.2.11 version in place at /usr/local/lib path and then restart SSH service (service sshd restart) and then exit the existing SSH session before logging back into SSH to set CLOUDFLARE_ZLIB='y' and then recompile nginx via centmin.sh menu option 4. Not sure why this only happens for cloudflare zlib version and not my custom compiled zlib 1.2.11 standard version.
     
  6. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    What about if there is an update for Zlib on an existing Centminmod installation.
    For example Zlib 1.2.12? Same problem?

    Solution could be to use Cloudflare Zlib only for Nginx (with-zlib=) as it is intended by Cloudflare for Nginx or Cloudflare-Nginx and for the web server only. No reason to use it internally on a computer separate from web services.
     
    Last edited: Dec 14, 2017
  7. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    so far haven't ran into it.
    true.. probably best way forward

    more testing :)
     
  8. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    +1 for that solution.
    Same as currently been used for Brotli. Should be also the best solution for Cloudflare Zlib. As written before. For both. No reason to use it besides web services.
     
    Last edited: Dec 14, 2017
  9. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    brotli won't be used by anything else AFAIK.

    but zlib updated fix is going to need some testing as I would need to dynamically restart any services relying on zlib previously built the old way heh. Testing CentOS 7 systems looks fine. Need to test CentOS 6.
     
  10. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    You are still testing for the scenario: using Cloudflare Zlib for the whole system?

    As Brotli is used for Nginx alone.
     
  11. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Last edited: Dec 14, 2017
  12. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
  13. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    Looking at your replies and latest patch.
    You don't need to ./configure, make and make install Zlib Cloudflare if it is only for Nginx.
    Will do the job.
    And then proceed to configuring Nginx.
    For example:
    With make install at Cloudflare Zlib it will conflict with your current Zlib 1.2.11.
     
    • Agree Agree x 1
  14. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah the patch comments out the configure/make install stuff for both custom zlib 1.2.11 and cloudflare zlib. I left it commented out for now instead of total removal code wise :)
     
  15. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
    I found the code below in the PHP part of Centminmod.
    I think it would be a good idea to script this to for Cloudflare Zlib.
    I.e. Check support and enable or disable Cloudflare Zlib before building, if SSE4.2 is supported or not supported.
     
    • Agree Agree x 2
  16. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah good idea (y)
     
  17. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
  18. bassie

    bassie Active Member

    939
    220
    43
    Apr 29, 2016
    Ratings:
    +664
    Local Time:
    12:39 PM
  19. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    8:39 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  20. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    11:39 AM
    1.9.x
    10.1.x
    @eva2000 @bassie

    I compiled nginx with cloudflare zlib. But if i run the lsof command, it shows this:

    Code:
    lsof | grep nginx | grep libz
    
    amplify-a   483           nginx  mem       REG               8,17              52302646 /usr/lib64/libz.so.1.2.7 (path dev=0,81)
    amplify-a   483   484     nginx  mem       REG               8,17              52302646 /usr/lib64/libz.so.1.2.7 (path dev=0,81)
    amplify-a   483 19338     nginx  mem       REG               8,17              52302646 /usr/lib64/libz.so.1.2.7 (path dev=0,81)
    This means that if even we compile nginx with a custom zlib, it will still use the one that comes pre installed with Centos? I got confused now.

    PS: Ok, just noticed that those lines are the Nginx Amplify agent. Nginx doesn't even show :D
     
    Last edited: Feb 21, 2018
    • Funny Funny x 1
..