/ Register

Join the community today
Register Now

High TTFB on "cold" start on low traffic site

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Shirkit, Sep 1, 2022.

Previous Thread Next Thread
Loading...
  1. Sep 1, 2022 #1
    Shirkit

    Shirkit New Member

    20
    4
    3
    Feb 6, 2019
    Ratings:
    +7
    Local Time:
    3:14 PM
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7.9.2009 64bit
    • Centmin Mod Version Installed: 124.00 stable (but this droplet is 2 years old)
    • Nginx Version Installed: 1.21.6
    • PHP Version Installed: 7.4.30 with Zend OPcache
    • MariaDB MySQL Version Installed: 10.3.35
    • When was last time updated Centmin Mod code base ? : About a month ago
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      MARCH_TARGETNATIVE='n'
PHPFINFO='y'
LETSENCRYPT_DETECT='y'
#NGXDYNAMIC_NGXPAGESPEED='y'
#NGINX_PAGESPEED='y'

    Hello,

    So I've been on a journey to improve our site and my clients Wordpress sites that are hosted on a small DigitalOcean Droplet U$12/mo (1vCPU 2GB Ram SSD).

    I have a decent loading speed, with a SpeedIndex of 2.6 and LCP of 2.5 on average. There is still room for improvement, but I'll be lying if I thought that this was bad for an ecommerce website.

    What is happening is that sometimes the TTFB goes through the roof, and I can't explain what exactly is happening. I've disabled the cronjob that clears the cache everyday, since this actually hurts my site since I have really low traffic.

    I have a cminfo top log here: PHP Warning: PHP Startup: Unable to load dynamic library 'imagick.so' (tried: / - Pastebin.com

    "Cold start" - 4G USA - WebPageTest Performance Test Results

    [​IMG]

    1 minute later - 3G Fast USA - WebPageTest Performance Test Results

    [​IMG]

    As you can see in the second test, now the TTFB is what I'd expect even with it being a slower network (3G Fast on the later vs the 4G on "Cold start"). I noticed also my analytics tag firing the config event, but only half of those ended up with my paging firing the DOM Ready, which I figured is that the user leaves the page before it actually displays.

    I noticed that php-fpm peaks CPU usage as when I reload the page on Firefox with cache disabled, the CPU usage goes to 100%. Wasn't the cache supposed to help minimize this?

    [​IMG]

    I tried setting up a Cloudflare based approach to bypass my server cache following eva2000 guide but I thought it ended up being slower than with only my host working. I've saved results and I can share later.

    I'm using Autopmize with Cache Enabler. The only thing I think could be messing things here is my Webp Express webp image generation plugin but since the images are already generated this shouldn't be an issue for this case.

    I thought of setting up Redis in front to cache all the generated cache pages, but with an SSD I honestly don't think my gains would be measurable enough to notice, assuming I have Cache Enabler properly working.

    Has anyone faced similar issues before? Maybe is something with the caching? I'm really lost here to what this could be. Any suggestions? Have I overlook or done something wrong?
     
    Last edited: Sep 1, 2022
  2. Sep 1, 2022 #2
    eva2000

    eva2000 Administrator Staff Member

    54,363
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    4:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    It sounds like typical Wordpress and/or MySQL database apps where slow MySQL backend response leads to PHP-FPM processes waiting for a response from MySQL and queueing up and in turn that leads to Nginx waiting on PHP-FPM to respond.

    How was your Wordpress site created originally?
    1. via centmin.sh menu option 22 Wordpress auto installer with Cache Enabler selected for full HTML page caching? Or
    2. was your Wordpress site created on non-Centmin Mod server and moved over to Centmin Mod and had Wordpress Nginx vhost manually re-created on Centmin Mod with Cache Enabler manually installed again?
    Centmin Mod WordPress install via centmin.sh menu option uses a force downgraded Cache Enabler 1.4.9 version due to various issues with Cache Enabler 1.5.1+ new cache routines https://community.centminmod.com/th...hanges-breaking-wordpress-sites-search.20546/ so would differ from you manually installing Cache Enabler and manually re-creating your Nginx vhost.

    However, as you aren't using Cloudflare or a CDN then it's expected to be slower as TTFB is relative to your server's location and test location or client /visitor location so further away geographically regardless of caching on origin Centmin Mod server side. But you're WPT 4G cold start vs WPT 3G warmed up origin cache tests show that Cache Enabler seems to be working with faster TTFB due to Cache Enabler origin based cached response.

    It should be depends on the page or URL too. Are you reloading the page as a logged in Wordpress user with Wordpress related cookies?

    I see you have shopping cart/commerce site so if you have Wordpress specific cookies for logged in users, they will bypass Cache Enabler to prevent messing up logged in user sessions as such performance of your site is dependent on non-cached origin server performance for it's configuration and/or hardware.

    My day job is basically to do server optimization consultation for clients for these very issues as logged in user based server/site performance is more about having an optimal configuration for software/hardware as guest full HTML page caching won't help for those users.

    For Cloudflare and TTFB see the guide I wrote on official Cloudflare community forums at https://community.cloudflare.com/t/improving-time-to-first-byte-ttfb-with-cloudflare/390367/1. If you want fast TTFB for most geographical users/guest non-logged in, you need to use a CDN and properly configure it. For logged in users, it's all about your settings and hardware as I mentioned in the official Cloudflare community forum link.
     
  3. Sep 1, 2022 #3
    eva2000

    eva2000 Administrator Staff Member

    54,363
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    4:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    One thing obvious from your cminfo top output you only have 1 CPU thread and already maxing out your CPU from the sar filtered times reported where you exceed 1cpu x 1.0 = 1.0 CPU load average at 1 min interval and not at 5 min interval so the variance in your TTFB response times would be normal as sometimes you have maxed out CPU and other times the CPU is not so heavily loaded which seems to be the case most of the time with low CPU utilisation.

    Peaked at 2.52/1 CPU = 2.52 load at 1:05:01 PM which means 252% CPU load
    Code (Text):
    ------------------------------------------------------------------
Filter sar -q for times cpu load avg (1min) hit/exceeded cpu threads max

12:55:01 AM         2       162      1.04      0.51      0.33         0
01:05:01 PM         3       169      2.52      0.93      0.38         1
03:10:01 PM         4       161      1.29      0.48      0.32         0
04:15:01 PM         5       166      1.11      0.73      0.50         0

    CPU utilisation for past 7 days from your cminfo top output shows light CPU usage but max 26% CPU steal also means your DigitalOcean VPS host node is overloaded and other VPS users are stealing CPU time
    Code (Text):
    ------------------------------------------------------------------
 CPU Utilisation % Last 7 days (1 CPU Threads):
------------------------------------------------------------------
%CPU  min:  %user:  0.62   %nice:  0.00  %system:  0.30  %iowait:  0.01  %steal:  0.07   %idle:  28.97
%CPU  avg:  %user:  4.50   %nice:  0.01  %system:  1.15  %iowait:  0.13  %steal:  1.64   %idle:  92.58
%CPU  max:  %user:  59.52  %nice:  1.52  %system:  7.16  %iowait:  3.65  %steal:  26.40  %idle:  98.89
%CPU  50%:  %user:  3.33   %nice:  0.00  %system:  0.99  %iowait:  0.05  %steal:  1.28   %idle:  93.99
%CPU  75%:  %user:  4.51   %nice:  0.00  %system:  1.32  %iowait:  0.09  %steal:  2.40   %idle:  95.62
%CPU  90%:  %user:  6.71   %nice:  0.00  %system:  1.66  %iowait:  0.26  %steal:  3.32   %idle:  96.74
%CPU  95%:  %user:  10.21  %nice:  0.00  %system:  2.15  %iowait:  0.42  %steal:  4.04   %idle:  97.24
%CPU  99%:  %user:  32.82  %nice:  0.01  %system:  5.44  %iowait:  1.74  %steal:  6.77   %idle:  98.11

    seem that was on August 25th from cminfo top output
    Code (Text):
    ------------------------------------------------------------------
 CPU Utilisation % Daily Last 7 days (1 CPU Threads):
------------------------------------------------------------------

Aug 25 2022 %CPU
%CPU  min:  %user:  1.37   %nice:  0.00  %system:  0.49  %iowait:  0.02  %steal:  0.08   %idle:  33.41
%CPU  avg:  %user:  4.50   %nice:  0.00  %system:  1.40  %iowait:  0.16  %steal:  2.79   %idle:  91.15
%CPU  max:  %user:  39.17  %nice:  1.32  %system:  7.16  %iowait:  3.65  %steal:  26.40  %idle:  97.90
%CPU  95%:  %user:  7.51   %nice:  0.00  %system:  2.11  %iowait:  0.50  %steal:  6.06   %idle:  96.36

Aug 24 2022 %CPU
%CPU  min:  %user:  1.13   %nice:  0.00  %system:  0.62  %iowait:  0.02  %steal:  0.49  %idle:  58.16
%CPU  avg:  %user:  4.24   %nice:  0.01  %system:  1.35  %iowait:  0.16  %steal:  2.25  %idle:  91.99
%CPU  max:  %user:  36.75  %nice:  1.40  %system:  6.56  %iowait:  2.91  %steal:  8.66  %idle:  96.72
%CPU  95%:  %user:  8.83   %nice:  0.00  %system:  2.01  %iowait:  0.67  %steal:  3.96  %idle:  95.97


    Also, DigitalOcean is still one of the slowest VPS providers I have tested https://community.centminmod.com/th...talocean-vs-linode-vs-vultr-vs-hetzner.17742/. Even recent tests are still comparatively slower.
     
  4. Sep 1, 2022 #4
    Shirkit

    Shirkit New Member

    20
    4
    3
    Feb 6, 2019
    Ratings:
    +7
    Local Time:
    3:14 PM
    Hello @eva2000 ,

    First I'd like to thank you for taking your time to look at my topic. Indeed there are a lot of things that combined are probably adding up to this. What I do know is that for guest visitors I still see the spike in utilization, which is weird since it should be fully cached.

    This site in specific I have no ideia how it was originally created. It's been so long it's running antd I've migrated it before this, so it could be anything

    I'm not using a CDN since I noticed a slow down after setting up using your guide (the one setting up with Wordpress) on CF Free Plan on US (droplet on US and Testing in the US).

    I don't think I can do much to improve DB speed on my specific case. Maybe upgrading MariaDB to a newer version? Maybe cleaning up the database, which I've done in the last year and noticeed a big difference for logged in users.

    The thing I said is that 4G was slower than 3G on the tests, and on both of them the cache was already there created.

    I tested as a guest since most of our traffic and conversions comes from guest and first time visitors.

    Logged-in visitors for my tracking their retation is high currently, so even serving slower pages, it appears to be fine on the analytics (not perfect but it's our reality). Search results are always cached as well, so only cart/checkout pages are actually not properly cached, which is fine for the measuring we've been doing.

    My Next steps:

    What I think I'll do is take a look at your benchmarks for VPS and choose a new VPS for running it, and creating everything with latest CentminMod beta. I'll grab a 2 vCPU VPS to improve the peak usage and logged-in user experience. But for sure I'll move out from DigitalOcean. Maybe Upcloud NYC or Vultr, or just move into a SA server (the downside is that all that I've tested have lower performance but lower ping to my SA clients)

    Maybe I'll split into two VPS so I can satisfy my US and SA customers better. Or maybe try attaching CF only to SA clients while using a US VPS.

    One last question: do you know how to reduce these timings? I suspect this is more related to the testing platform (webpagetest.org) than anything else. Google.com has a 535ms total connection time so I'm 20% slower than Google.com on these metrics (my VPS is at NYC and WebPageTest is set from Virginia).

    DNS Lookup: 281 ms
    Initial Connection: 182 ms
    SSL Negotiation: 188 ms
     
  5. Sep 1, 2022 #5
    eva2000

    eva2000 Administrator Staff Member

    54,363
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    4:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yes, that's what clients pay me to do for them :)

    Slow is relative as 3G and 4G connections by nature are slow with anywhere between 150-300ms round trip times (RTT)/response times and download speeds as slow as 1.6Mbps which is 160-180KB/s. So the slow 3G and 4G test environments will not allow you to go faster than the network (speed of light fiber connections).

    For WPT 4G profile RTT is 170ms so a single initial connection or SSL negotiation of 188ms is within variance for 4G profile's RTT.
     
    Last edited: Sep 1, 2022
  6. Sep 2, 2022 #6
    tininho

    tininho Active Member

    182
    44
    28
    May 22, 2019
    eu
    Ratings:
    +135
    Local Time:
    8:14 PM
    If I was you, I would

    • Backup your site with All-in-One WP Migration – WordPress plugin | WordPress.org or similar (restore it with the this/same tool after succesful WP install on Centminmod).
    • Install a fresh setup of Centminmod 130.00beta01 and go with PHP 7.4 for now, unless you are sure all your plugins are compatible with PHP 8.1 (check your error log).
    • Get your WP optimized custom_config.cfg file from here centmininit/custom_config.inc or read through this forum to make your own.
    • Choose now what cache option you want to use (dont change it after WP installation), in my opinion, for Woo, go either with Centminmod's Fastcgi (free) or save yourself from trouble, and go with WP Rocket (paid), configuration guide here.
    • If you chose Autoptimize, make sure your settings are correct and read this. WP Rocket automatically optimizes your assets (again, read a guide to get the settings right)
    • Make sure Cloudflare's Rocket Loader is disabled and all your CF cache settings and page rules are inline with Woocommerce.
    • After you are sure your site works correctly, you can further optimize it like this.
    This can be done a lot quicker than to troubleshoot a site which you were not building from the start. Get back to us if your site is still slow on TTFB. Good luck
     
    Last edited: Sep 2, 2022
  7. Sep 2, 2022 #7
    Shirkit

    Shirkit New Member

    20
    4
    3
    Feb 6, 2019
    Ratings:
    +7
    Local Time:
    3:14 PM
    Thank you for your answer! What a great, concise and incredible suggestion! I really appreciated that!
    • I'll go try to go with PHP 8.1 if posible
    • Gonna spin ip a new VPS, prolly thinking of Vultr NVMe 2vCPU 4GB Ram machine and attaching a Cloudflare CDN on top on a domain basis to serve SA countries (I just can't afford two VPS atm)
    • Autopmize is already working fine, what I think I'll study is if I should or should not aggregrate JS/CS, as sometimes it's just not that good.
    I've been following CentminMod and have read many articles and topics written by eva thus far, and I can say his work has taught me a lot.

    Do you think WP Rocket caching solution is better (in some metric other than initial setup costs) than Centmin's menu 22 caching (Cache Enabler plus it's modifications)?

    Thank you for your input!
     
  8. Sep 2, 2022 #8
    rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    2:14 AM
    Mainline
    10.2
    Do you have an extra $5 to spend?
    Try Cloudflare APO.

    For logged users, install and configure Redis Data caching.
     
  9. Sep 2, 2022 #9
    Shirkit

    Shirkit New Member

    20
    4
    3
    Feb 6, 2019
    Ratings:
    +7
    Local Time:
    3:14 PM
    Thank you very much for your input!

    I'll take a look at Cloudflare APO, as this can solve guest non-cookie users on different continents.

    I've never setup Redis so far, you mean setup it up as a replacement for Cache Enabler or just as object cache? If it is the latter, you mean following this guide https://community.centminmod.com/th...ss-redis-object-cache-plugin-installer.20418/
     
  10. Sep 2, 2022 #10
    tininho

    tininho Active Member

    182
    44
    28
    May 22, 2019
    eu
    Ratings:
    +135
    Local Time:
    8:14 PM
    Definitely WP Rocket with Woocommerce over anything else, you can bypass php-fpm even on Woo for logged-in users. Cloudflare APO is not needed on this setup (you should try though, maybe it helps on your case). And yes, Get NVME and 4Gb server if you can.

    You can use Cloudflare as CDN by making a page rule to cache your /uploads folder, its often not useful to cache logged-in users at this level. If you have the traffic where you need Cloudflare for Woo, you should upgrade to Cloudflare Business anyway (bypass cookie).

    You definitely should make a copy of your site and try setups like rdan's with free open-source tools on a subdomain.

    I would say having fastcgi/WPRocket site working first would minimize your stress. Then try APO, Redis Object cache and so on. Learn and test, its fun :)
     
  11. Sep 2, 2022 #11
    rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    2:14 AM
    Mainline
    10.2
    What I'm using are for several WP sites...

    For Guest Full Page Caching (choose only 1):
    - Cloudflare APO (the best and easier to setup)
    - Cloudflare Bypass Cookie on Page Rule
    - WP Super Cache
    - Cache Enabler
    - Nginx FastCGI_Cache

    For Logged user Data Caching (choose only 1):
    - Memcached
    - Redis

    Disable unnecessary PHP and Nginx extensions via custom_config.

    That is great.
     
  12. Sep 2, 2022 #12
    tininho

    tininho Active Member

    182
    44
    28
    May 22, 2019
    eu
    Ratings:
    +135
    Local Time:
    8:14 PM
    I did only quick scan on your site, but I am not convinced it's working to the max :) The thing is you need to Inline Critical CSS Above-the-Fold, and this needs to be done separately for each site, which is quite a lot on Woocommerce site..

    So with Autoptimize you can do it like this, as we discussed with @eva2000: Wordpress - Which plugins to use for better performance for Wordpress? | Page 2 | Centmin Mod Community Support Forums

    Once again, with WP Rocket it's done for you, automatically.
     
  13. Sep 2, 2022 #13
    Shirkit

    Shirkit New Member

    20
    4
    3
    Feb 6, 2019
    Ratings:
    +7
    Local Time:
    3:14 PM
    WP Rocket is a bit outside of my reach as it's costs are considerable when currency is converted. If this was all for a single site it would be fine, but since this VPS hosts a few different low traffic sites, it gets expensive for them all. Same goes with Cloudflare Business, it's just outside of my reach. For a single domain I'll take a look at implementing WP Rocket or Cloudflare APO as they have the same cost.

    Logged-in users caching for all 11 websites is not a concern, just for one of them.

    I'll try and pick one of them, probably gonna stick around with Cache Enabler as it is the cheapest and easiest to setup considering running costs. From everything I've read that @eva2000 has written I think his centmin.sh 22 with Cache Enabler does a really good job to full page Guest Caching, at the same level of Nginx FastCGI cache.

    Still, WP Rocket and Cloudlflare solutions are looking better and better the more I look.

    I'll take a look at Data caching with Memcached to start things off.

    It's not working at the maximum, critical CSS is something that I indeed did not touch as there's not a good solution for it without throwing additional $$$ on a website basis. Critical CSS on the website would be more doable if I could set it up per page basis with Autopmize with additional rules. It would be great if it was that way.

    And removing unused CSS is something I really really want, but it's a non-trivial task on most Wordpress installations, as necessary plugins throw in CSS in excess and purging them while keeping everything update can only be done by two tools I've seen thus far, WP Rocket being one of them.

    Again, thank you everyone for your inputs and taking your time to answer this thread!

    Edit: as I've taken a look at Autopmize it seems that I can set up on a page basis! I'll add this to the backlog of tasks to do! Just perfect!
     
    Last edited: Sep 2, 2022
Previous Thread Next Thread
Loading...

.