/ Register

Join the community today
Register Now

Sysadmin Nginx 100% use Cpu

Discussion in 'System Administration' started by upgrade81, Apr 5, 2018.

Previous Thread Next Thread
Loading...
Page 2 of 4
  1. Apr 8, 2018 #21
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Upgrading servers just to handle good bots isn't ideal as there's no guarantee the good search bots crawling your site means you get indexed in a search rank that provides any visitors to you anyway. Rate limiting (not blocking) good bots won't affect your search ranking unlike blocking the bot.

    Also there's fake bots which disguise themselves by changing their user agent strings to pretend to be good bots too.

    Also if they're legit bingbots, using bing webmaster tools for your site (like google webmaster tools), you can control bingbot's crawl rate Crawl Control - Bing Webmaster Tools. But if they're fake bingbots, such crawl rate control won't work and need rate limiting in place.

     
  2. Apr 8, 2018 #22
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    do you mean the rate limit to be included in robots.txt?


    You think it's the fault of brotli in the last 30 of / log / message brotli is ok.

    Do you think that a mixed configuration of static files created with brotli.sh and dynamic can create problems?
     
  3. Apr 8, 2018 #23
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    maybe you'd have to diagnose that unfortunately

    no in Bing Webmaster Tools which you should sign up to like Google Webmaster Tools and add your domain/site profile for SEO analysis and crawl rate controls i.e. for my forums

    Bing Webmaster Tools

    upload_2018-4-8_4-54-3.png

    upload_2018-4-8_4-58-20.png
     
  4. Apr 8, 2018 #24
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    Ahhhh you mean this about Bing and Google. okok

    However, from the data that I entered on the forum does not show anything unusual?
     
  5. Apr 8, 2018 #25
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    however, it seems ridiculous to me that a Crawler can throw me down Nginx that stays fixed at 100% until I restart it.

    It is not a matter of 30 minutes 1 hour, if not the restart remains so, perennial!
     
  6. Apr 8, 2018 #26
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    heavy crawling from bingbot and google image bot crawlers seems to be most likely - that can be normal if legit you just need to change the crawl rates + rate limiting
    can also be fake bots too
     
  7. Apr 8, 2018 #27
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    FYI, from my Sucuri stats 67% of all blocked attacks are from bad bots !

    upload_2018-4-8_5-6-37.png
    upload_2018-4-8_5-8-48.png
     
  8. Apr 8, 2018 #28
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    I understand.

    Solution beyond that of the rate limit?
    - Fai2Ban for bad-bots?
    - An external Waf Example: Cloudflare? Although I would avoid it because it slows down the time to download the pages.
    - Put a reverse proxy in front of nginx? like Haproxy?
     
  9. Apr 9, 2018 #29
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    fail2ban + rate limiting is good. But external WAF like Cloudflare probably better for some attacks. But Cloudflare shouldn't slow down page download times when setup correctly, especially these days as Cloudflare has launched so many datacenters to cover every part of the world.
     
  10. Apr 9, 2018 #30
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    Can you explain better how to set up Cloudflare?

    This example is with Cloudflare in Italy.

    grabilla.g44976.png
     
  11. Apr 9, 2018 #31
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
  12. Oct 24, 2018 #32
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    12:05 AM
    1.5.15
    MariaDB 10.2
    So I've seen this first hand now on his server. Nginx sits at 100% cpu usage on a single core. It's still processing traffic. Netstat shows around 900 connections to port 443.

    A restart of nginx clears it down and it returns to normal.

    upload_2018-10-23_21-23-52.png

    Not seeing anything out of the ordinary with the config files.
     
  13. Oct 24, 2018 #33
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    if you can not solve it, I see it very hard.:(:pompous:
     
  14. Oct 24, 2018 #34
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    You have 4 cpu threads, so using one fully shouldn't be a problem though ? or was it causing problems/performance issues ?

    what does ngxtop and logs say those connections are from and where they're directed to request/url wise Sysadmin - Nginx 100% use Cpu ?
     
  15. Oct 24, 2018 #35
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    12:05 AM
    1.5.15
    MariaDB 10.2
    Sites were loading fine as far as I could see.

    As I've just mentioned in a PM though, this is happening on 3 VPs hes built, so wonder if it's something related to a build option with nginx?

    I've personally never seen this on any server ive built, especially where a restart drops the CPU back down, but there is no actual reduction in traffic.
     
  16. Oct 24, 2018 #36
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Oh these are @upgrade81's servers not yours ? Using latest Nginx 1.15.5 ? CentOS 6 or 7 ? KVM/OpenVZ/Xen ?

    You've PM'd me about this before ? Don't have any recollection heh. His Wordpress installs if installed via centmin.sh menu option 22 do they have additional rate limiting and workaround for Wordpress - Wordpress DOS Attack Flaw Security CVE-2018-6389 ?
     
  17. Oct 24, 2018 #37
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    12:05 AM
    1.5.15
    MariaDB 10.2
    Correct, not mine, hes just asked me to take a look.

    Only have access to the 1, but don't know how they were installed and the sites setup.
     
  18. Oct 24, 2018 #38
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    Hi @eva2000 ,
    exact is my VM, I used option 22 with Redis.
    then I migrated the site into this vhost. the site has a seniority of 12 13 years.

    it's been months since I have this problem is not related to CVE-2018-6389

    One thread is used because the default nginx is with:
    worker_connections 50000;

    if you lower the workers to 4092, it also uses the other cpu cores.

    therefore the first thread is saturated but the others remain almost free.
    I think it's for this, but this is not the problem, nginx load salt until the site is offline and I get notified by uptime monitor.

    It's really heartbreaking, Matt has detected a spike of connections to 443

    Code:
    netstat -an | grep ": 443" | wc -l
958
    online with analytics there were between 350 and 400 real users.
     
  19. Oct 24, 2018 #39
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    9:05 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Shouldn't matter as long as nginx and sites run fine with there's other nginx workers to handle the load. sounds like some of sort of slow DDOS like resource consuming attack like traffic patterns ???

    same IPs ? you should inspect the traffic patterns to see where it's coming from and where it's being directed
     
  20. Oct 24, 2018 #40
    upgrade81

    upgrade81 Member

    296
    17
    18
    Sep 5, 2016
    CH
    Ratings:
    +30
    Local Time:
    1:05 AM
    1.17
    10.3
    as far as I can see on the access.log the traffic goes to different articles all pages with 200 very few answers 301.

    different ip, many national and reserved for mobile devices.

    for another @Matt he will answer you, it is he who is looking well.
     
Previous Thread Next Thread
Loading...
Page 2 of 4

.