Cloudflare A Tour Inside Cloudflare's G9 Servers

eva2000 · Oct 11, 2018

Cloudflare gives us a tour of some of their G9 servers sporting 192x custom made Intel Xeon Gold 24 core per cpu Skylake processors which power their network A Tour Inside Cloudflare's G9 Servers

We spend a significant amount of time specing and designing servers that makes up our network to meet our ever changing and growing demands. On regular intervals, we will take everything we've learned about our last generation of hardware and refresh each component with the next generation
Click to expand...

Since the last time we blogged about our G4 servers, we’ve iterated one generation each of the past 5 years. Our latest generation is now the G9 server. From a G4 server comprising 12 Intel Sandybridge CPU cores, our G9 server has 192 Intel Skylake CPU cores ready to handle today’s load across Cloudflare’s network. This server is QCT’s T42S-2U multi-node serverwhere we have 4 nodes per chassis, therefore each node has 48 cores. Maximizing compute density is the primary goal since rental colocation space and power are costly. This 2U4N chassis form factor has served us well for the past 3 generations, we’re revisiting this option once more.
Click to expand...

Hardware changes
CPU

Previously: 2x 12 core Intel Xeon Silver 4116 2.1Ghz 85W

Now: 2x 24 core Intel custom off-roadmap 1.9Ghz 150W

The performance of our infrastructure is heavily directed by how much compute we can squeeze in a given physical space and power. In essence, requests per second (RPS) per Watt is a critical metric that Qualcomm’s ARM64 46 core Falkor chip had a big advantage over Intel’s Skylake 4116. Embracing the value of optionality and market competition, we made some noise.

Intel proposed to co-innovate with us an off-roadmap 24-core Xeon Gold CPU specifically made for our workload offering considerable value in Performance per Watt. For this generation, we continue using Intel as system solutions are widely available while we’re working on realizing ARM64’s benefits to production. We expect this CPU to perform with better RPS per Watt right off the bat; increasing the RPS by 200% from doubling the amount of cores, and increasing the power consumption by 174% from increasing the CPUs TDP from 85W to 150W each.

Disk

Previously: 6x Micron 1100 512G SATA

Now: 6x Intel S4500 480G SATA

With all the requests we foresee for G9 to process, we need to tame down the outlying and long-tail latencies we have seen in our previous SSDs. Lowering p99 and p999 latency has been a serious endeavor. To help save milliseconds in disk response time for 0.01% or even 0.001% of all the traffic we see isn’t a joke!
Datacenter grade SSDs in Intel S4500 will proliferate our fleet. These disks come with better endurance to last over the expected service life of our servers and better performance consistency with lower p95+ latency.

Network

Previously: dual-port 10G Solarflare Flareon Ultra SFN8522

Now: dual-port 25G Mellanox ConnectX-4 Lx OCP

Our DDoS mitigation program is all done in userspace, so network adapter model can be anything on market as long as it supports XDP. We went with Mellanox for their solid reliability and their readily available 2x25G CX4 model. Upgrading to 25G intra-rack ethernet network is easy future-proofing since the 10G SFP+ ethernet port shares the same physical form factor as the 25G’s SFP28. Switch and NIC vendors offer models that can be configured as either 10G or 25G.

Memory

Previously: 192G (12x16G) DDR4 2400Mhz RDIMM

Now: 256G (8x32G) DDR4 2666Mhz RDIMM

Going from 192G (12x16G) to 256G (8x32G) made practical sense. The motherboard has 12 DIMM channels, which were all populated in the G8. We want to have the ability to upgrade just in case, as well as keeping memory configuration balanced and optimal bandwidth capacity. 8x32G works well leaving 4 channels open for future upgrades.
Click to expand...

In other words for every 3 G8 servers, just 2 of those G9's would take on the same workload. If one of our racks would normally have 9 G8 servers, than only 4 G9's are needed. Inversely, planning to turn up a cage of 10 G9 racks would be the same as if we did 15 G8 racks!

And we got big plans to cover our entire network with G9 servers, with most of them planned for the existing cities your site most likely uses. By 2019, you’ll benefit with increased bandwidth and lower wait times. And we’ll benefit in expanding and turning up datacenters quicker and more efficiently.
Click to expand...

2019 will see much faster Cloudflare network

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Cloudflare A Tour Inside Cloudflare's G9 Servers

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Cloudflare A Tour Inside Cloudflare's G9 Servers

eva2000 Administrator Staff Member

.