Learn about Centmin Mod LEMP Stack today
Become a Member

Nginx PageSpeed PageSpeed Serf fetch failure rate extremely high

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Jon Snow, May 28, 2019.

  1. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: Latest
    • PHP Version Installed: Latest 7.1
    • MariaDB MySQL Version Installed: Latest
    • When was last time updated Centmin Mod code base ? : Yesterday
    Custom config:

    Code (Text):
    NGINX_PAGESPEED='y'
    NGXDYNAMIC_NGXPAGESPEED='y'
    LETSENCRYPT_DETECT='y'
    SECOND_IP=0.0.0.0.0
    MARCH_TARGETNATIVE='n'
    DISABLE_IPVSIX='y'


    I'm seeing this in the nginx error log:

    Code (Text):
    Slow ReadFile operation on file /var/ngx_pagespeed_cache/v3/domain.com/https,3A/,2Fdomain.                com/forum/data/avatars/s/12/12577.jpg,3F1519161390,: 218.861ms; configure SlowFileLatencyUs to change threshold
    
    2019/05/27 13:41:33 [error] 3190#3380: [ngx_pagespeed 1.13.35.2-0] PageSpeed Serf fetch failure rate extremely high; only 0 of 1737 recent fetches fully successful; is                 fetching working?
    2019/05/27 14:11:35 [error] 3190#3380: [ngx_pagespeed 1.13.35.2-0] PageSpeed Serf fetch failure rate extremely high; only 0 of 1260 recent fetches fully successful; is                 fetching working?
    2019/05/27 14:12:23 [alert] 3191#3191: *647106 open socket #14 left in connection 60
    2019/05/27 14:12:23 [alert] 3191#3191: *654013 open socket #38 left in connection 61
    2019/05/27 14:12:23 [alert] 3191#3191: aborting
    2019/05/27 14:42:53 [error] 13120#13142: [ngx_pagespeed 1.13.35.2-0] PageSpeed Serf fetch failure rate extremely high; only 0 of 1273 recent fetches fully successful;                 is fetching working?
    2019/05/27 14:47:55 [error] 13119#13145: [ngx_pagespeed 1.13.35.2-0] Slow ReadFile operation on file /var/ngx_pagespeed_cache/v3/domain.com/https,3A/,2Fdomain.com/forum/data/avatars/s/0/122.jpg,3F1397640560,: 72.242ms; configure SlowFileLatencyUs to change threshold
    
     
  2. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    see Frequently Asked Questions
    did you properly configure ngx_pagespeed's pagespeed.conf as it's not enough to just enable ngx_pagespeed in the HTTPS first world these days - see below Xenforo Nginx PageSpeed - This forum's pagespeed.conf thread discussion for how to properly setup for HTTPS

    FYI, centmin mod 123.09beta01's default pagespeed.conf config file already has
    Code (Text):
      # https://www.modpagespeed.com/doc/https_support
        pagespeed SslCertDirectory /etc/pki/tls/certs;
        pagespeed SslCertFile /etc/pki/tls/cert.pem;
    

    so you probably need to setup appropriate LoadFromFile lines PageSpeed Authorizing and Mapping Domains and combined with LoadFromFileRuleMatch from my example at https://community.centminmod.com/threads/this-forums-pagespeed-conf.62/

    Code (Text):
    pagespeed LoadFromFile "http://community.centminmod.com" "/home/nginx/domains/community.centminmod.com/public";
    pagespeed LoadFromFile "https://community.centminmod.com" "/home/nginx/domains/community.centminmod.com/public";
    
    pagespeed LoadFromFileRuleMatch disallow .*;
    pagespeed LoadFromFileRuleMatch allow \.css$;
    pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
    pagespeed LoadFromFileRuleMatch allow \.png$;
    pagespeed LoadFromFileRuleMatch allow \.gif$;
    pagespeed LoadFromFileRuleMatch allow \.js$;
    
     
  3. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    I don't think I saw any of:
    Code (Text):
        pagespeed SslCertDirectory /etc/pki/tls/certs;
        pagespeed SslCertFile /etc/pki/tls/cert.pem;

    So I added it to the top with the rest right at the bottom.

    I started seeing the below in the error log:
    Code (Text):
    2019/05/28 12:51:05 [alert] 13120#13120: *205057 open socket #28 left in connection 9
    2019/05/28 12:51:05 [alert] 13120#13120: *203202 open socket #34 left in connection 33
    2019/05/28 12:51:05 [alert] 13120#13120: *205058 open socket #29 left in connection 43
    2019/05/28 12:51:05 [alert] 13120#13120: *203204 open socket #37 left in connection 54
    2019/05/28 12:51:05 [alert] 13120#13120: aborting

    This slowed down the whole site and I couldn't type anything via SSH terminal for like 30 seconds.
     
    style="display:inline-block;min-width:400px;max-width:970px;width:95%;height:90px" data-ad-client="ca-pub-6669518204467592" data-ad-slot="4024536743" data-ad-format="auto">
  4. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Also do I even need the non-HTTP line if I only use HTTPS?

    Code (Text):
    pagespeed LoadFromFile "http://community.centminmod.com" "/home/nginx/domains/community.centminmod.com/public";
    pagespeed LoadFromFile "https://community.centminmod.com" "/home/nginx/domains/community.centminmod.com/public";
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    doesn't hurt having it

    depends how old your pagespeed.conf is, it should be half way down the file

    next time it slows down, try disabling pagespeed temporarily to rule it out as it could be network issues if you can't connect to SSH as SSH has nothing to do with nginx/ngx_pagespeed.

    to disable ngx_pagespeed
    Code (Text):
    pscontrol off

    to renable ngx_pagespeed
    Code (Text):
    pscontrol on
     
  6. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    I brought it up with Linode before and they said it might be an internal config issue because of a 502 bad gateway error I mentioned with it. They also claimed to have looked for networking / connectivity issues but I don't think they found any.

    Disabled pagespeed like you recommended above and the issue is still there although it seems like it's A LOT better than before (no more 30+ second pauses via SSH). It's more like a 3-8 second wait.

    This started happening about 5-8 days ago (edit: I think? But it wasn't much days like a whole 30+ days).

    I've searched the nginx, php fpm and mariadb error logs but nothing showed up besides what I've mentioned above.
     
    Last edited: May 28, 2019
  7. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    The pauses are random (like one can happen in 2 mins then the other in maybe 15 mins) btw.
     
  8. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Pagespeed is disabled from command line and I just got a 502 bad gateway page from my browser.
     
  9. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  10. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Wouldn't there be any hints in an error log?
     
  11. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Code (Text):
    pool:                 www
    process manager:      ondemand
    start time:           28/May/2019:14:36:43 +0000
    start since:          925
    accepted conn:        2200
    listen queue:         0
    max listen queue:     25
    listen queue len:     511
    idle processes:       17
    active processes:     7
    total processes:      24
    max active processes: 39
    max children reached: 0
    slow requests:        0

    This looks OK to me if idle processes is fine.
    max active processes did not reach pm.max_children.
     
  12. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yes if it was related to php-fpm you'd also see in php-fpm status output if max children where reached which you don't see

    do you mean initial ssh log in ? that could be related to broken IPv6 network connectivity

    Could you provide some of the answers and output to the questions and commands asked at https://community.centminmod.com/threads/troubleshoot-slow-ssh-logins.12124/
     
  13. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Not initial. That's fine. I kept the SSH session active and waited for my site to load slowly. I noticed when the site loads slowly, I'm unable to type anything in Putty. When the site was fine, I'd see some of what I typed in Putty. So it might just be lag when the site loads slowly.

    Edit: IPv6 is disabled too and the IPv6 IPs from Linode's DNS manager are not there.
     
  14. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    @eva2000 Issue was Linode again. I messaged them around when it first started happening and they said nothing was wrong. Got a different support guy and I ran his commands. People on the server were using up resources.
     
    • Informative Informative x 1
  15. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    check your cpu steal percentages indicating overloaded VPS server Linode - Thank goodness for glances - cpu steal

    sar command to check cpu utilisation and cpu steal %
    Code (Text):
    sar -u | sed -e "s|$(hostname)|hostname|"
    
     
  16. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Yea already did that. It was around 50%
     
    • Informative Informative x 1
  17. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    that's extremely high cpu steal percentage which as you now know means other VPS neighbours on your VPS host node at Linode are heavily using cpu and server resources and thus affecting your VPS's performance. You'd need to ask Linode to move your VPS to a VPS node that is not overcrowded and busy.
     
  18. Jon Snow

    Jon Snow Active Member

    425
    64
    28
    Jun 30, 2017
    Ratings:
    +102
    Local Time:
    9:33 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    I didn't have to ask. They did it for me already after I sent them the results. Most of my issues have been because of bad neighbors on Linode.. heh.
     
    • Like Like x 1
  19. eva2000

    eva2000 Administrator Staff Member

    40,624
    9,016
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,883
    Local Time:
    10:33 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Good to hear :D
     
    • Like Like x 1