WebPerf Facebook's new front-end web server designs

eva2000 · Sep 22, 2016

Interesting Facebook article on their new front-end web server designs Facebook's new front-end server design delivers on performance without sucking up power | Engineering Blog | Facebook Code | Facebook

We have been using traditional two-socket servers for more than five server generations now at Facebook. Because our web servers are heavily compute-bound and don't require much memory capacity, the two-socket servers we had in production had several limitations. The server has a QPI link that connects the processors, which created a NUMA problem. It also requires an accompanying chipset that required more power. We kept pushing the performance (and hence power) envelope. Intel provided us with 95W, then 115W, and now 120W-130W CPUs to hit our performance targets. Given an 11kW rack level power budget, pushing the limits of CPU power was not scalable, and we were not seeing performance keep up with increased power.

Our current Haswell CPUs are produced based on Intel's 14 nm process, with the next process generation being 10 nm. Over a number of decades, Intel's performance improvements have steadily kept in line with Moore's Law — but more recently their “tick-tock” cycle encompassing a new architecture and process transition has slowed.

Making a server-class CPU is difficult and something that we take for granted. Process transitions are also challenging, especially at such minuscule dimensions. We knew that Intel was solving a hard problem, but since our software was evolving at a rapid pace in parallel, we wanted to take this problem on as well and look at our system design through a new lens.
Click to expand...

Since web servers represent a major portion of the front-end cluster, we decided to focus our optimization efforts on this component of our fleet. Three years ago we started working with Intel to define the details of a new processor called Broadwell-D, part of Intel's Xeon line of processors, that was better suited to data center workloads rather than enterprise. We minimized the CPU to exactly what we required. We took out the QPI links, which reduced costs for Intel and removed the NUMA problem for us, given that all servers would be one-socket-based. We designed for it to be a system-on-a-chip (SOC), which integrates the chipset, thus creating a simpler design. This single-socket CPU also has a lower thermal design power (TDP).

At the same time, we redesigned our server infrastructure to accommodate double the number of CPUs per rack within the same power infrastructure. Together, we designed a 4 CPU/sled server and 65W CPU that would land squarely in the sweet spot from a performance and power perspective. Facebook is limited by an 11kW rack power budget. We have 30 servers in a rack. At 120W per CPU, this allowed for 60 CPUs, while 65W would allow for 120 CPUs per rack within the same power infrastructure. While the math might look straightforward, it required a significant redesign of Intel's roadmap to develop this new part.

Modifying their roadmap is not an easy process given the complexity behind designing and manufacturing a processor. Intel calls this new line of products the Xeon-D line, and has made this line of parts plan of record for broader market adoption. We use Xeon-D and Broadwell-D (BW-D) interchangeably here.
Click to expand...

Nice so the Intel Broadwell-D processor came about through Facebook's work with Intel - good to know. Benchmarks I did of Intel Xeon D-1540 way back Benchmarks - Intel Xeon D-1540 benchmarks | Centmin Mod Community

Log in / Register

Forums

Get the most out of your Centmin Mod LEMP stack

WebPerf Facebook's new front-end web server designs

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Get the most out of your Centmin Mod LEMP stack

WebPerf Facebook's new front-end web server designs

eva2000 Administrator Staff Member

.