Join the community today
Become a Member

Nginx Loader.io Nginx Test

Discussion in 'Centmin Mod Insights' started by rdan, Feb 27, 2022.

  1. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    Server Specs:
    CMM on CentOS 7.9
    Vultr Dedicated Cloud
    $60/month$0.089/hour
    2 CPU
    8 GB Memory
    10 TB Bandwidth

    Loader.io Test specs:
    Clients 10k to 10k for 1 minute
    Gzip Encoding
    https and plain index.html test

    upload_2022-2-27_5-10-40.png


    Somehow it failed the test sometimes or with network error :-(


    Screen Shot 2022-02-27 at 5.12.02 AM.png Screen Shot 2022-02-27 at 5.13.19 AM.png
     
  2. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    Most of the time, loader.io aborted the test due to errors.

    But in comparison, a plain Ubuntu 20.04 Nginx install doesn't produce an error.
    On the same server specs and same testing.

    Curious what/where is wrong on CMM nginx?
    Help Eva :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    What's results like for Ubuntu tests ?
     
  4. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    Always no error/s.
    But a little bit higher average response time.
    But lower max response times.

    upload_2022-2-27_5-47-38.png
     
  5. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    If you still have access to both setups i.e. imaged/restorable, you can share the output for their respective Nginx full config outputs saved to nginx-full-config.txt files.
    Code (Text):
    nginx -T > nginx-full-config.txt

    You can then just post both outputs to Diffchecker to compare them and see what differs.

    Then do the same for
    Code (Text):
    sysctl -a > tcp-config.txt
     
  6. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
  7. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    still would help to have the outputs for those 2 commands to compare :)
     
  8. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    Ok I spun up a $60/month Vultr dedicated cloud VPS server to do same loader.io tests and saw the same network errors you posted. Found the issue and updated 123.09beta01 with potential fix which isn't enabled by default due to previous user reported failure to compile Nginx on initial Centmin Mod installs way back on Nginx 1.17!

    So you can try redoing a fresh Centmin Mod 123.09beta01 install but prior to initial install, set in the persistent config file /etc/centminmod/custom_config.inc the variable
    Code (Text):
    AUTOHARDTUNE_NGINXBACKLOG='y'

    as outlined originally from https://community.centminmod.com/th...og-defaults-in-123-09beta01.17620/#post-74529

    then re-test loader.io tests and see.

    This is what I got

    loader-io-10k-01.png
     
  9. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    That is very amazing result, woah!
    Thanks Eva.
    I'll create a new VPS later as I already deleted the server few hours ago.

    Again very great! :love:
     
  10. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    With AUTOHARDTUNE_NGINXBACKLOG='y' built Centmin Mod Nginx the current default backlog is raised from 511 to 16383 for HTTPS vhosts out of the box and you end up with topping out at 35K error free and 40K minor network errors.

    loader-io-35k-01.png loader-io-40k-01.png

    With nginx backlog manually set to 32768 in HTTPS nginx vhost listen directive (only can be set once in one nginx HTTPS listen directive to apply to all IP/Port (443) pairs.

    We max out loader.io free plan's 50K concurrent users :D Based on monitored stats, would be capable of 60-70K users if loader.io free plan allowed :D

    loader-io-40k-backlog32768-01.png loader-io-50k-backlog32768-01.png
     
  11. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
  12. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    Suggestion, from inc/nginx_configure.inc file.
    Maybe add also another sed command to force replace all custom/old values.

    Code:
    sed -i "s|backlog=.*|backlog=$SET_NGINXBACKLOG|" /usr/local/nginx/conf/conf.d/virtual.conf
    Just like the one from
    /usr/local/etc/php-fpm.conf
     
  13. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    Same server specs and test, but with OLS.

    OpenLiteSpeed is the BEAST!

    Brotli
    upload_2022-2-27_13-27-9.png

    Gzip
    upload_2022-2-27_13-28-52.png
     
  14. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    I'm on their legacy free plan :)
    Originally it was so it was set once, but yeah probably better to do that
    Yes Litespeed and OLS can be faster because they have an inbuilt small file memory based cache for files less than 4k in size + in built gzip static nginx equivalent feature so the precompress the static file for serving. You can replicate the latter in Centmin Mod Nginx by gzip compressing the index.html before testing ensuring both index.html and index.html.gz exist in same directory. That's why Centmin Mod WordPress installer with Cache-Enabler can be faster than PHP-FPM fastcgi_cache caching due to precompress full HTML page precompression and nginx gzip_static directive to serve precompressed static files if they're detected

    With pre-compressed index.html and Centmin Mod Nginx gzip_static improved average response times at 50,000 concurrent user loads

    loader-io-50k-backlog32767-gzipstatic-01.png

    Though these days with most of my sites behind Cloudflare CDN proxy, static file serving performance is at least offloaded to Cloudflare CDN edge servers, so throughput of origin Nginx backend isn't as important as latency response time (time Cloudflare edge server communicates with Nginx origin web server) :)
     
    Last edited: Feb 27, 2022
  15. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    index.html test
    Default CentOS 7.9 3.10.x kernel
    upload_2022-5-20_5-0-40.png
    Versus

    ELRepo 5.4.x with TCP_BBR
    upload_2022-5-20_5-0-59.png
     
  16. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    5.4.x ELRepo / TCP BBR / index.html
    upload_2022-5-20_5-20-48.png

    Versus with KTLS
    upload_2022-5-20_5-57-24.png



    5.4.x ELRepo / TCP BBR / index.js
    upload_2022-5-20_5-20-5.png
    Versus with KTLS
    upload_2022-5-20_6-7-56.png
     
  17. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    @rdan interesting numbers - pretty close throughput wise.
     
  18. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    5.4.x ELRepo / TCP BBR / index.html
    upload_2022-5-20_8-4-59.png

    Versus with IOURING
    upload_2022-5-20_8-9-46.png

    Something is off with IOURING, only the average response times is Advantage.
    But less Success response and Bandwidth recieved.
     
  19. rdan

    rdan Premium Member Premium Member

    5,328
    1,338
    113
    May 25, 2014
    Ratings:
    +2,066
    Local Time:
    7:22 AM
    Mainline
    10.2
    5.4.x ELRepo / TCP BBR / index.php
    upload_2022-5-20_8-21-17.png

    Versus with IOURING
    upload_2022-5-20_8-16-42.png

    IOURING seems a downgrade based on my test except for Fast response times BUT only for static files.
     
  20. eva2000

    eva2000 Administrator Staff Member

    49,280
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,572
    Local Time:
    9:22 AM
    Nginx 1.21.x
    MariaDB 10.x
    PHP will always have more overhead than static files :)