Get the most out of your Centmin Mod LEMP stack
Become a Member

WebPerf Google's BBR Algorithm for Speeding up Internet Traffic Gains Wider Adoption

Discussion in 'All Internet & Web Performance News' started by pamamolf, Jul 22, 2017.

  1. pamamolf

    pamamolf Well-Known Member

    2,539
    231
    63
    May 31, 2014
    Ratings:
    +394
    Local Time:
    7:59 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    [​IMG]
    Google's BBR algorithm for handling TCP traffic congestion has gained huge ground this week after Google announced integration with Google Cloud, a cloud hosting platform offered by Google to thousands of companies and which serves millions of websites on a daily basis.

    BBR stands for "Bottleneck Bandwidth and RTT (Round-Trip Time)," and is an algorithm for optimizing how network packets travel through servers in order to avoid jamming certain routes.

    Google started working on this algorithm as a replacement for Reno and CUBIC, two bandwidth control algorithms developed in the 80s. Work started a few years back, but in September 2016, Google submitted patches to the Linux kernel networking stack in order to add BBR support in modern Linux kernels.

    Earlier this month, Google engineers presented a technical spec to the IETF (Internet Engineering Task Force) describing BBR. Other details about the BBR algorithm are also available in two research papers, here and here.

    BBR will speed up Internet traffic, avoid congestions
    What the algorithm does is to define new ways for servers to handle TCP connections, the basic communication channel for exchanging data via the Internet.

    Google says older TCP congestion control systems lead to bottlenecks in Internet traffic because these algorithms were built around the idea of detecting a congestion after it happened, which would be too late to re-route some users.

    BBR was designed to prevent bottlenecks before they happen, say Google engineers, an important aspect in today's Internet landscape that handles a large number of connections which require real-time data transfers. Traffic congestions break these real-time connections and slow down Internet speeds.

    BBR roll-out less complicated than some people think
    Before submitting code changes to the Linux kernel last year, Google tested BBR for its internal network and with public services like YouTube and Google.com.

    Engineers were able to do so because implementing BBR requires applying changes to the server-side only, with no plugins or special code for existing client software such as browsers, desktop operating systems, smartphones, or local LAN equipment.

    This week, Google deployed BBR for the Google Cloud platform, home of services such as Spotify, Snapchat, Wix, WP Engine, and more. WP Engine says it enabled BBR for the 500,000 WordPress sites it hosts on its infrastructure.

    "BBR's throughput can reach as much as 2,700x higher than today's best loss-based congestion control," says Google, while "queueing delays can be 25x lower."

    Based on internal tests, Google says BBR improved YouTube network throughput by 4% on average globally and by more than 14% in some countries.
     
    • Informative Informative x 1
  2. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    Nice was reading about this the other day too :D

    please always cite the source of the news too Google's BBR Algorithm for Speeding up Internet Traffic Gains Wider Adoption :)

    and
    from Google Groups

     
    Last edited: Jul 23, 2017
  3. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    FYI, Linode VPS uses already have BBR support due to using 4.9 Linux kernels

    On non-linode / non linux kernel 4.9 servers cubic is default tcp_congestion_control set
    Code (Text):
    sysctl net.ipv4.tcp_available_congestion_control
    net.ipv4.tcp_available_congestion_control = cubic reno
    
    sysctl net.ipv4.tcp_congestion_control
    net.ipv4.tcp_congestion_control = cubic
    

    On Linode and linux kernel 4.9+ servers default is still cubic but there's TCP BBR congestion control support
    Code (Text):
    sysctl net.ipv4.tcp_available_congestion_control
    net.ipv4.tcp_available_congestion_control = cubic reno bbr bic westwood htcp
    
    sysctl net.ipv4.tcp_congestion_control
    net.ipv4.tcp_congestion_control = cubic
    
     
    • Informative Informative x 1
  4. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    centminmod.com's Linode Singapore CentOS 7.3 + Linux 4.9.x kernel cluster edge server has now enabled Google BBR tcp congestion control seeing as Linode provided 4.9.x kernel already supports BBR.
    Code (Text):
    uname -r
    4.9.36-x86_64-linode85
    

    Code (Text):
    sysctl net.ipv4.tcp_available_congestion_control
    net.ipv4.tcp_available_congestion_control = bbr cubic reno bic westwood htcp
    
    sysctl net.ipv4.tcp_congestion_control
    net.ipv4.tcp_congestion_control = bbr
    
    tc -s qdisc
    qdisc noqueue 0: dev lo root refcnt 2
     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
     backlog 0b 0p requeues 0
    qdisc fq 8001: dev eth0 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
     Sent 409011 bytes 635 pkt (dropped 0, overlimits 0 requeues 0)
     backlog 0b 0p requeues 0
      100 flows (96 inactive, 0 throttled)
      0 gc, 0 highprio, 22 throttled
    

    I saved a Linode backup snapshot before making the changes just to be safe :)
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    I spun up a $5 Linode VPS in Newark, New Jersey to test Linode block storage. So while I had the VPS, I enabled Google BBR congestion control switching from default cubic. Ran speedtest-cli and seems improved latency (ping) though throughput was varying. Or didn't make that much difference? Maybe needed to test speeds for larger distant servers instead of closest servers?

    Before

    upload_2017-7-23_16-31-0.png

    Code (Text):
    ./speedtest-cli --share --server 6030
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by fdcservers.net (New York City, NY) [14.26 km]: 7.5 ms
    Testing download speed................................................................................
    Download: 1464.35 Mbit/s
    Testing upload speed................................................................................................
    Upload: 1317.71 Mbit/s
    Share results: http://www.speedtest.net/result/6477884141.png

    Code (Text):
    ./speedtest-cli --share --server 6030
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by fdcservers.net (New York City, NY) [14.26 km]: 2.946 ms
    Testing download speed................................................................................
    Download: 1811.86 Mbit/s
    Testing upload speed................................................................................................
    Upload: 1429.16 Mbit/s
    Share results: http://www.speedtest.net/result/6477884908.png


    After

    upload_2017-7-23_16-31-24.png

    Code (Text):
    ./speedtest-cli --share --server 6030
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by fdcservers.net (New York City, NY) [14.26 km]: 4.257 ms
    Testing download speed................................................................................
    Download: 1481.81 Mbit/s
    Testing upload speed................................................................................................
    Upload: 1298.52 Mbit/s
    Share results: http://www.speedtest.net/result/6477879650.png

    Code (Text):
    ./speedtest-cli --share --server 6030
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by fdcservers.net (New York City, NY) [14.26 km]: 2.777 ms
    Testing download speed................................................................................
    Download: 1528.63 Mbit/s
    Testing upload speed................................................................................................
    Upload: 1417.94 Mbit/s
    Share results: http://www.speedtest.net/result/6477880526.png
     
    • Informative Informative x 1
  6. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    Further distance in California seems slightly better

    Before

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 69.891 ms
    Testing download speed................................................................................
    Download: 75.54 Mbit/s
    Testing upload speed................................................................................................
    Upload: 85.74 Mbit/s
    Share results: http://www.speedtest.net/result/6477903390.png
    

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 82.713 ms
    Testing download speed................................................................................
    Download: 145.67 Mbit/s
    Testing upload speed................................................................................................
    Upload: 131.58 Mbit/s
    Share results: http://www.speedtest.net/result/6477903919.png
    

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 70.16 ms
    Testing download speed................................................................................
    Download: 115.40 Mbit/s
    Testing upload speed................................................................................................
    Upload: 102.35 Mbit/s
    Share results: http://www.speedtest.net/result/6477904808.png
    


    After

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 68.061 ms
    Testing download speed................................................................................
    Download: 130.73 Mbit/s
    Testing upload speed................................................................................................
    Upload: 164.59 Mbit/s
    Share results: http://www.speedtest.net/result/6477907885.png
    

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 74.737 ms
    Testing download speed................................................................................
    Download: 124.60 Mbit/s
    Testing upload speed................................................................................................
    Upload: 130.78 Mbit/s
    Share results: http://www.speedtest.net/result/6477908686.png
    

    Code (Text):
    ./speedtest-cli --share --server 3864
    Retrieving speedtest.net configuration...
    Testing from Linode (45.xxx.xxx.xxx)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by California Internet Solutions (Los Angeles, CA) [3922.22 km]: 180.354 ms
    Testing download speed................................................................................
    Download: 131.66 Mbit/s
    Testing upload speed................................................................................................
    Upload: 140.50 Mbit/s
    Share results: http://www.speedtest.net/result/6477909362.png
    
     
    • Informative Informative x 1
  7. bassie

    bassie Active Member

    500
    106
    43
    Apr 29, 2016
    Ratings:
    +318
    Local Time:
    6:59 AM
    As indicated in the example with examples (picture above) as Google.com and Youtube.com
    This almost must be a good example of an innovation that renders only on the top of the top.

    Just like sites as Google, Facebook etc etc where your internet traffic environment is continuously running at the max. I.e. millions and millions of users at the same time. With data quantities you can't imagine.
     
    Last edited: Jul 23, 2017
    • Agree Agree x 1
  8. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    Indeed could be the case or might need further TCP tuning. The 2nd point mentioned at Google Groups might apply

    was testing with centmin mod auto set values of
    Code (Text):
    sysctl -a | egrep 'wmem|rmem'  
    net.core.rmem_default = 212992
    net.core.rmem_max = 16777216
    net.core.wmem_default = 212992
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 8192        87380   16777216
    net.ipv4.tcp_wmem = 8192        65536   16777216
    

    which isn't far off

    though articles on Google BBR are reporting overall around 4-14% improvement for some so maybe within normal variance?
     
  9. RoldanLT

    RoldanLT Well-Known Member

    3,831
    929
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,258
    Local Time:
    12:59 PM
    1.11
    10.2
    • Like Like x 1
  10. RoldanLT

    RoldanLT Well-Known Member

    3,831
    929
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,258
    Local Time:
    12:59 PM
    1.11
    10.2
  11. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    be careful with this especially on CentOS 6 as some systems don't support qdisc = fq so BBR won't work unless on kernel 4.13+. So you would need to know how to reverse the changes made by such a script too.
     
    • Informative Informative x 2
  12. bassie

    bassie Active Member

    500
    106
    43
    Apr 29, 2016
    Ratings:
    +318
    Local Time:
    6:59 AM
    Some end user of RHEL asked about if Red Hat could
    Backport BBR TCP congestion control to RHEL 7.
    Unfortunately you need an Red Hat subscription to view the topic.

    Could someone over here with an Red Hat subscription read the topic whether this will be addressed and backported?

    I know you could use an alternative kernel but upstream stable is
    is my preference.

    Backport of BBR TCP congestion control to RHEL 7 - Red Hat Customer Portal
     
  13. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    no subscription here :(

    Honestly don't see why RHEL/CentOS can't make newer software available via their Extras, Fasttrack repositories as an option that end users can choose to use if they wanted. Business and folks who require rock stable can just use as is. While folks who want more edge newer releases can use those repos. Or have a separate OS distro version, RHEL/CentOS Edge OS which would be based on RHEL 7.x but have newer packages by default. Ubuntu 17.10 for example is getting rebased to Linux 4.13 kernel, GCC 7 and Perl 5.26 out of the box !
     
  14. bassie

    bassie Active Member

    500
    106
    43
    Apr 29, 2016
    Ratings:
    +318
    Local Time:
    6:59 AM
    Of course it's annoying in case of BBR though RHEL is for mission critical and every extra code change could lead to instability. 100% availability is way more important than 0.5 maybe 1% at max improvement at the internet connection.

    For example your doc needs your profile right now, not tomorrow, not yesterday now, doesn't matter if it is 1% faster at download. Doc will not even notice it. He does notice it if it is not available.

    Youtube is a very different story and right there, BBR is extremely interesting.

    About Ubuntu 17.10. Ubuntu 17.10 is not an enterprise product.
    The Red Hat equivalent of Ubuntu 17.10: Fedora 26 also has BBR, GCC 7.1 etc etc.

    Finally. I'm not a licensing administrator so I'll ask a colleague to look at this message. Although I thought it would be faster with this request.
     
  15. bassie

    bassie Active Member

    500
    106
    43
    Apr 29, 2016
    Ratings:
    +318
    Local Time:
    6:59 AM
    About enabling BBR on CentOS with a stable CentOS kernel

    I am not a big fan of the elrepo.org kernel, its to edgy, straight from kernel.org, small test curve, small team etc. (I may have been biased, I have experienced a few crashed servers with elrepo.org kernels).
    Additionally, not everyone has Linode as provider to use their kernels.

    So searched some time and encountered the following:
    As a side-product to an Xen envoriment, it offers a well tested, BBR supporting, stable kernel. Backed by some notable teams mentioned above.

    Installation is simple:

    Result should be:
    net.ipv4.tcp_available_congestion_control = bbr cubic reno

    Result should be:
    bbr

    Result should be:
    tcp_bbr 16384 0

    Done.

    P.s. You are not installing an Xen environment but only the kernel.
    Which just like kernel.org upstream could offer KVM and Xen support.
    RHEL 7's default kernel can offer KVM only.
    (Red Hat removed Xen in RHEL).

    Test

    Before:

    After:

     
    Last edited: Aug 3, 2017
    • Informative Informative x 2
  16. RoldanLT

    RoldanLT Well-Known Member

    3,831
    929
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,258
    Local Time:
    12:59 PM
    1.11
    10.2
    @eva2000 Do you have any plans to auto enable this config for Linode Install?
     
  17. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    no plans right now, maybe later :)

    seems linode provided kernel might be missing BBR support itself or BBR is baked into the linode kernel itself as lsmod doesn't return anything

    If missing, you would need to revert linode custom 4.9.x kernels to centos distro kernel, then install 4.12+ kernel manually i.e. via elrepo kernel-ml packages. Reboot server and then apply BBR required changes

    Code (Text):
    uname -r
    4.12.6-1.el7.elrepo.x86_64
    

    Code (Text):
    sysctl net.ipv4.tcp_available_congestion_control
    net.ipv4.tcp_available_congestion_control = bbr cubic reno
    

    Code (Text):
    sysctl net.ipv4.tcp_congestion_control
    net.ipv4.tcp_congestion_control = bbr
    

    Code (Text):
    lsmod | grep bbr                                             
    tcp_bbr                20480  0
    

    Code (Text):
    tc -s qdisc
    qdisc noqueue 0: dev lo root refcnt 2
     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
     backlog 0b 0p requeues 0
    qdisc fq 8001: dev eth0 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
     Sent 686 bytes 5 pkt (dropped 0, overlimits 0 requeues 0)
     backlog 0b 0p requeues 0
      1 flows (0 inactive, 0 throttled)
      0 gc, 0 highprio, 0 throttled
    qdisc noqueue 0: dev docker0 root refcnt 2
     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
     backlog 0b 0p requeues 0 
     
    Last edited: Aug 13, 2017
    • Like Like x 1
  18. RoldanLT

    RoldanLT Well-Known Member

    3,831
    929
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,258
    Local Time:
    12:59 PM
    1.11
    10.2
    Yay, my recent Linode VPS.
    upload_2017-8-13_22-33-19.png
     
    • Like Like x 1
  19. eva2000

    eva2000 Administrator Staff Member

    29,053
    6,594
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,790
    Local Time:
    2:59 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah i think BBR is built into Linode 4.9.x kernels rather than loaded as a module.
     
  20. Tinkerd10

    Tinkerd10 New Member

    4
    1
    3
    Sep 28, 2016
    Ratings:
    +4
    Local Time:
    5:59 AM
    1.11
    5.5
    I also after reading about Google brr i wanted to install it on my production server
    Altho i have centos 6, so i followed some tutorial online and everything works fine
    at first at least

    But i was unable to save any data to database, it kept going time-out
    With upstream header already sent

    so i adjusted
    wait_timeout : 300
    max_allowed_packet: 500M

    But nothing changed
    i couldn't go deep with testing as it is a production server

    Os: centos 6.9
    Kernel-lm : vmlinuz-4.12.7-1.el6.elrepo.x86_64
    Kernel-lm : vmlinuz-4.12.5-1.el6.elrepo.x86_64
    Kernel : vmlinuz-2.6.32-696.6.3.el6.x86_64

    MariaDB : 10.1.26
    PHP 7.0.15
    nginx/1.10.3

    sysctl.conf:
    # Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 - Pastebin.com

    My.cnf
    [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock tmpdir=/home - Pastebin.com

    So i reverse it back to old kernel and everything worked fine.
     
    • Informative Informative x 2