Benchmarks Optimizing and tuning WordPress and XenForo with NewRelic

I want document my journey of attempting to optimize my rather large WordPress/XenForo site (8.2k WP articles, 13m XF posts) after recently moving to a new server. (If this isn't the right forum for this post, please move it, @eva2000! )

Dedicated Server Specs

• AMD EPYC 7251 @ 2.1Ghz (with optimized power settings)
• 64GB DDR4 memory
• 500GB NVMe drive
• PHP 7.4
• MariaDB 10.4

New Relic ONE

I am using New Relic's PHP Application Performance Monitor (APM) to provide me with detailed performance data. It is affordable (I'm paying less than $5/month) and the data is invaluable when diagnosing a problem. The dashboard can be a bit overwhelming and I admit I am just barely scratching the service of understanding everything it does, but I would still feel like I was flying blind without it.

Using the PHP agent's API, you can segment the applications in your dashboard. (You can also segment reports per directory with the Nginx.conf file, but I was not able to get that to work for some reason — apparently it doesn't always work with Nginx/PHP-FPM/FastCGI sometimes.)

So, I just added the API code to my WordPress theme's functions.php file. This allows me to see how WordPress and XenForo are performing, individually.

As we can see, WordPress is significantly slower than XenForo 2.2 on my server.

XenForo (73ms average response time):

---

WordPress (391ms average response time):

Next Steps

My first plan of attack is to start caching with Redis. I assume the Centminmod Redis setup guide is still up-to-date?

I will begin with configuring XenForo 2.2's Redis caching. I am considering using Xon's Redis Cache for XenForo add-on, but I will admit that I am not entirely sure why it is necessary or so popular. According to Xon, it adds:

However, I know Centminmod's PHP-FPM config supports Redis out of the box, and I assume that means it is already using the "php-redis" extension, so I don't see an advantage there. I also don't plan on having multiple Redis instances, so I don't see how Redis Sentinel or slaves/scaling support will help me, either. I do use some of Xon's other add-ons so that would be good, I guess.

I just like to run as clean of an install as possible with as few add-ons as possible. I will post before/after NewRelic benchmarks once the XF Redis caching is in place.

Later, I will look at WordPress Redis caching. Of course I will eventually move to full-page caching of WordPress pages with Cloudflare Workers, but I want to look at how each step in the process improves (hopefully!) page generation time in NewRelic. I will probably not do any caching for XF guest page views because I am vain enough to want to maintain accurate thread view counts, but I digress...

 

Well, the first few hours of testing are done, and the results are interesting.

XenForo with Redis

I forgot I had indeed configured XF to use Memcached. I didn't expect Redis to make a difference, but according to NewRelic it looks like it did show a very minor improvement at least in the first few hours.

With Memcached, XF average response time was 73ms. After, 62ms. It also "feels" a bit more response, especially doing things like opening alerts and reacting to posts.




XenForo Guest Page Caching

This was really interesting. I could not find any real discussion or benchmarks around this so I decided to play around with guest page caching to see how it would affect TTFB.

I don't have graphs or hard data, but from looking at my own stats in Chrome developer tools it was very obvious: cached guest pages reduced TTFB by around 150ms.

Without guest page caching, threads with 30 posts typically load in about 350-400ms. Uncached page views also took this long. When the page was cached (as confirmed by the "X-XF-Cache-Status: HIT" header), TTFB was between 200-220ms. I even saw a few in the 140ms range.

This is significant because it helps me get my forum pages close to the ~200ms threshold that Google considers "good" for TTFB. I wish I didn't have to sacrifice thread view counts from guests, but maybe it's worth it. It's something I will have to consider.

WordPress Redis Caching

Something was definitely not right here.

I installed and configured the "Redis Object Cache" WP plugin as suggested here in the Centminmod Redis guide. The site immediately became sluggish and slow, with TTFB times regularly larger than 1 second.

NewRelic confirmed this:



I am not sure how to troubleshoot that any further. When others complained of slow performance in the plugin's support forum, the author suggested that Redis must be malfunctioning or misconfigured. That's certainly possible, but it's clearly working great with XenForo. I also have the Redis cache set to a fairly large size (2GB) and flushed it several times while debugging to see if that would fix it, but to no avail.

Any suggestions are welcome.

Even without Redis, I feel the WordPress pages are far too slow, so I need to dig in to any plugins or theme issues which might be causing some of these problems as well. I want to make sure everything is running good under the hood before turning on Cloudflare magic...

 

It makes me feel a bit better that you find it confusing as well!

It is good for spotting trends, though. Thanks for sharing your WP graphs so I have something to compare against.

The WordPress Themes & Plugins section has helped me identify one of the big problems I was having with WordPress' slow speeds. It revealed a rather simple plugin I was using called Internal Links Manager was adding approximately 175ms to my page loading times!

(The plugin in question is called "seo automated link building" in this graph.)



As you can see I disabled the plugin just before 11:00pm (the big spike before that is from the Redis plugin testing). Overall impact to WordPress app response time:



I'm disappointed that I won't be able to use that plugin — it really seems quite straightforward in terms of what it does (links certain words in posts for you automatically) — but my editors can still do that or I could write my own plugin to do it without all the overhead.

WooCommerce and the "schema-and-structured-data-for-wp" plugins are my next big concerns. The schema plugin is another one that should be simple; it must be quite poorly coded if that is really adding so much overhead to page creation time.

WooCommerce is a bit of an odd one. If we drill down into its numbers we can see that its response times are extremely quick, there are just a LOT of calls to it. This is concerning and unfortunate because WooCommerce pages are not heavily trafficked on my site, so that must mean a lot of its code is getting called on pages that have nothing to do with the products. It looks like it is possible to disable plugins based upon the page being visited, but that's something I am going to have to do more research into.

Oh wow, good to know! Thanks for sharing, I didn't realize you had already tested that.

How are you doing cache invalidation? This would be nice but some of our threads move really fast. It seems like it would be nearly impossible to find a reasonable cache expiration time for our forum pages.

 

Finally have a chance to work on optimizations a bit more.

I have been very curious to see just how much of an issue WooCommerce really is, and fortunately I think it's not as bad as it looks.

I disabled the WooCommerce plugin for 20 minutes on my site between 9:15pm and 9:35pm and here's what happened... not much!



There was also no significant difference in TTFB times.

The plugin-specific NewRelic report shows the impact as well (I know, it says 8 o'clock in this graph and I have no idea why... maybe a time zone bug on their end):



My interpretation of the top graph is that for every one minute, roughly three seconds of total page computation time is caused by WooCommerce. As for how much time that actually adds to each individual page, that's a very small number.

I have researched techniques such as crafting a custom "Must Use Plugin" to disable WooCommerce on non-store pages, but implementing it will be time consuming and maintaining it could be a real pain. Considering that I will be moving to full-page caching with CF Workers eventually, I don't think it's worth the time, effort, and overhead to eliminate a few extra milliseconds from non-cached pages.

 

@eva2000, how are you handling WordPress cache invalidation? What plugin are you using?

I can write my own CF Worker to cache my WordPress pages, but the cache invalidation is where it gets messy. The post URL, homepage, related tag pages, related category pages, and the author page all need to be purged any time a post is updated. I can write my own plugin too, but I'd rather not.

The existing plugins I have found all look like they try to do a bit too much hand-holding and try to do far more than what I need, so I'm curious how you chose to do this. (Sorry, I'm sure you have probably already documented this somewhere but I can't find it! )

 

Ah interesting, so that is the same plug-in to be used with Cloudflare WordPress APO, and you’re just using it without APO?

I suppose I could just cache posts and let the home page and categories remain as is, as SEO isn’t quite as important for those pages…

 

Ah, I thought it was all-in-one now. Good to know!

I continue to be frustrated and disappointed with my server's uncached page performance. I feel like my dedicated server is more than powerful enough for my site's traffic loads, but maybe I really do need to look into something with a higher clock speed to get where I want. Maybe Hivelocity's Black Friday deals will have something to consider.

As another attempt at optimization, I upgraded to PHP 8 last night. There was a slight improvement in TTFB times but if we look at the comparison graphs with NewRelic, we can see it brings virtually no improvement for either application compared to 7.4.

 

OK, hang on a minute...

After posting my last message, I started poking around and discovered that rebooting had reset my server's CPU power governor setting. To be frank, it's really dumb that such a severe power-saving mode is the default, and it's even more dumb that the setting change you make is lost on reboots. I have to figure out how to make that stick through reboots.

The good news is that increasing the CPU frequency provided a major jump in PHP 8 performance!



XenForo:

• PHP 7.4 daily average response time: 99.89ms
• PHP 8 daily average response time: 72.25ms
• Improvement: 27% faster

WordPress

• PHP 7.4 daily average response time: 325ms
• PHP 8 daily average response time: 238ms
• Improvement: 27% faster

This is speculation on my part, but if I had to guess, the performance gains from PHP 8 were almost exactly nullified by the slower CPU clock speed.

I guess this means I am seeing an additional ~200ms coming from network latency and Cloudflare. Is that normal for an un-cached page on a non-Argo routed Cloudflare site?

 

Thanks to @eva2000's heads-up on the Clouvider Black Friday sale, I decided to do the "nuclear option" and move to an entirely new provider and server with a much higher CPU clock speed.

Old dedicated server specs:

• AMD EPYC 7251 @ 2.1Ghz (with optimized power settings)
• 64GB DDR4 memory
• 500GB NVMe drive
• PHP 8.0
• MariaDB 10.4
• Location: Tampa, FL, USA

New server specs:

• Intel E-2276G (6 Cores, 12 Threads) @ 3.8Ghz, Turbo Boost at 4.9Ghz
• 64GB DDR4 ECC RAM
• 960 GB NVMe + 8TB HDD
• PHP 8.0
• MariaDB 10.4
• Location: Ashburn, VA, USA

The first thing I did was tune the CPU to run at its maximum speed of 4.9Ghz. At @eva2000's advice, I have been closely monitoring CPU temperatures using sensors and they generally stay around 25-30°C under production load, and I have seen single cores go up to 50°C during CPU heavy operations like database dumps or restores. Those numbers appear to be in an acceptable range of the CPU so I'm comfortable running it that speed.

Code (Text):

Every 2.0s: sensors                                                                                                                    Tue Dec 14 06:39:17 2021

acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +119.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +30.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:        +29.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:        +30.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:        +28.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:        +28.0°C  (high = +80.0°C, crit = +100.0°C)
Core 4:        +29.0°C  (high = +80.0°C, crit = +100.0°C)
Core 5:        +27.0°C  (high = +80.0°C, crit = +100.0°C)

power_meter-acpi-0
Adapter: ACPI interface
power1:       49.00 W  (interval =   1.00 s)

Now, time to look at the benchmarks in NewRelic comparing the last 24 hours of performance with last week with the old server. This is all the same software versions and the same website:

WordPress Performance




WordPress average page response times have improved from 229ms to 99ms. That's 56.7% faster. Keep in mind this is a fairly heavy WordPress install with 31 active plugins, including WooCommerce.

TTFB performance also improved significantly. Of course most of this improvement will be due to the faster application response time, but some of this is also derived from the improved connectivity in Ashburn, VA, especially in Europe.


XenForo Performance




Really stunning performance from XenForo here (16 active plugins). It was already fast but now has an average response time of less than 30ms over the course of 24 hours. That's a 65% improvement over the slower processor.



All TTFB tests were performed with the KeyCDN TTFB performance test.

Google Analytics has also detected an improvement in speed (please notice, Google!):



(Server migration was completed on the evening of Dec. 10 in this chart.)

Needless to say, I am really happy with these results. The site feels faster with the "eyeball test" as well — things like the XenForo Alerts menu and reactions to post now feel nearly instant, compared to the slight delay that I experienced before.

I fully admit to becoming disillusioned with CPU clock speeds over the past several years, but for single-threaded PHP performance, this whole experience has reminded me just how important it is.

I'm happy with XenForo performance for now, but I will continue to chip away at WordPress with full-page caching using a custom CloudFlare Worker. (I know the functionality is there in Workers, I'm just not sure if they'll let me actually do it on the Pro plan... )

 

Nice, I'm curious to hear how your experience with reloading goes. I just set up the backups last night and so far I like Acronis. It's expensive (I'll be backing up 850GB for now) but the full-system restore just feels like a smart insurance policy to have, in addition to my other backups, of course.

LOL, no, my forum members only notice when things go wrong!

I'm eager to start working on my custom CF Worker script but I have decided to look at WordPress full-page caching first following your guide at Wordpress Cache Enabler Advanced Full Page Caching Guide - Centmin Mod Blog so that each CF POP can quickly load each article.

 

Time for an update.

As mentioned in my last post, I'm going to focus on WordPress performance now.

At eva's recommendation I have installed and configured KeyCDN Cache Enabler. I really liked its low-profile, single-purpose, utilitarian nature compared to most of the bloated do-everything WP plugins that plague the ecosystem.

However, I did get off to a bit of a rocky start as the plugin somehow crashed all of PHP when my editor published a post for the first time after I installed the plugin. I documented my experience here in the support forum; apparently others had a similar experience. Fortunately (though oddly...), it has not happened again.

The results were very impressive, with a few surprises.

For me, personally, it cut my TTFB times for cached pages in half from ~250ms to ~120ms, which is awesome. The cached pages feel super-fast on my own devices. But when I tested using the KeyCDN Performance Test (the same one I used to get the TTFB times in the charts above), there was virtually no difference in TTFB for any location. Maybe 10-20ms saved here or there, but nothing like what I was seeing from my own location. I can't explain what's going on there.

The improvement does show up on Google Analytics' Average Server Response Time Report, though, when I filter it to just show WordPress pages. It really shows how much progress I have made. As a reminder, I migrated to the new server on Dec. 10 and implemented KeyCDN cache enabler on Dec. 15:



Cache Enabler reduced the the response time from ~0.45s to ~0.35s in this report.

Interestingly, the caching actually makes WordPress look slower in my NewRelic reports. This kind of makes sense, because I implemented Cache Enabler's custom Nginx configuration rules so PHP is never touched on cached requests. That means PHP is serving fewer requests, so slower pages have a greater influence over the averages recorded by NewRelic.

I have cropped the report so you can see the significant drop in throughput that was recorded once Cache Enabler was installed.



Up next: a custom CloudFlare Worker for edge caching WordPress pages...

 

Cheers! I enjoy sharing it. It does take a lot of time to put together, but it is useful to look back and see how the work has paid off, and more importantly, it is a great reference to have in the future to see what I did and what my logic was when making decisions. One thing I started doing a few years ago is making my own notes as I do maintenance and migration, and it has made everything a lot easier. As I've mentioned before, I am a full-time site owner and administrator — not a full-time sysadmin. I can't immerse myself in the subject like most of you here, so it's a constant learning process, and writing out what I've done helps.

Maybe a dedicated forum encouraging CMM users to create threads documenting their maintenance and optimization journeys would be useful here. I would be very interested in seeing more benchmarks from other users, too.

Yeah, pretty close. I'm about 400 miles and 42ms away from the server in Ashburn. But there is no real TTFB improvement according to KeyCDN from any location, even other close POPs like NYC. My theory is that KeyCDN's requests include query strings or cookies which bypass their own plugin. I should take a closer look at their incoming requests.

Just look at the difference between a request to a plain HTML on my site compared to a cached article:

Plain HTML Page:



Cached Article:


As you can see, there's a clear ~50ms overhead there to account for PHP processing. I don't think that's what "real" users are seeing on cached pages, as I am getting 60-100ms TTFB times on articles myself, but I can't be quite sure.

Other TTFB tests like SpeedVitals also show a similar difference between the two types of pages. Strange.

Do you know if I will run into any problems writing a Worker with this functionality on just the CF Pro plan? I've tried to do some research but there doesn't seem to be a clear consensus on whether or not it actually works.

Before I start on the worker, though, I just realized how big my WordPress uploaded images are... I am serving some way-too-large images and have neglected optimizing them since I ditched my old image optimization plugin (long story). It's also past time I start serving WebP.

Right now I am considering Optimus since I saw it mentioned in KeyCDN's Cache Enabler plugin. It's also by KeyCDN, which I like, and the flat-rate pricing is also very attractive compared to some of the other image optimization plugins out there. I will need to do a bit more research before making a decision.