Join the community today
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. deltahf

    deltahf Premium Member Premium Member

    587
    265
    63
    Jun 8, 2014
    Ratings:
    +489
    Local Time:
    5:09 PM
    Interesting point, I had not thought about that. Are Googlebot imposters not an issue as well?

     
  2. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah that MAYBE an issue too. I have much more extensive nginx block/rate limiting method for google and bingbot imposters. But can use more resources as it involves using nginx and geoip nginx module to analyse and monitor the entire ip address/ASN ranges for both google and bing/microsoft to allow only known ips belong to them through and rate limit or block the imposter bots which are coming from ips not known to be owned by google/bing/microsoft.
     
  3. negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    12:09 AM
    1.9.10
    10.1.11
    I applied these configurations nginx.con and virtual hosts confs both but when i test the results:

    PHP:
    siege --c10 -r1 -"GetWeb" http://www.domain.com
    ** SIEGE 3.1.0
    ** Preparing 10 concurrent users for battle.
    The server is now under siege...
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.28 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.31 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.34 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.35 secs:    5395 bytes ==> GET  /
    done.

    Transactions:                  10 hits
    Availability
    :              100.00 %
    Elapsed time:                0.35 secs
    Data transferred
    :            0.05 MB
    Response time
    :                0.29 secs
    Transaction rate
    :           28.57 trans/sec
    Throughput
    :                0.15 MB/sec
    Concurrency
    :                8.23
    Successful transactions
    :           0
    Failed transactions
    :               0
    Longest transaction
    :            0.35
    Shortest transaction
    :            0.00
    It gives 520 Errors, not 444? I ' m using cloudflare, it may related with that?

    BTW: "Cliqzbot", "MegaIndex.ru" and "Yahoo! Slurp" are coming one of the top list user_agents for me.
    I will add the yahoo bot to whitelisted as 1 and "Cliqzbot" and "MegaIndex.ru" will be 3 of course.
     
    Last edited: Mar 7, 2016
  4. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah sounds like 520 is a cloudflare error status code Error 520: Web server is returning an unknown error – CloudFlare Support

    nginx 444 gives empty response so matchings what triggers cloudflare 520 so normal
     
  5. negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    12:09 AM
    1.9.10
    10.1.11
    Added these bots;

    "~*Pixray" 3;
    "~*MegaIndex.ru" 3;
    "~*Cliqzbot" 3;

    # bots whitelisted
    "~*Googlebot" 1;
    "~*bingbot" 1;
    "~*Yahoo!\ Slurp" 1;
     
  6. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    nice i updated 1st post with megaindex.ru and cliqzbot block and added yahoo to whitelist
     
  7. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    so how's everyones experiments with the 1st post outlined bad bot rate limiting/blocking configurations ?

    more tests please so can polish it for inclusion into 123.09beta01 config setups :D
     
  8. deltahf

    deltahf Premium Member Premium Member

    587
    265
    63
    Jun 8, 2014
    Ratings:
    +489
    Local Time:
    5:09 PM
    It's been working great for me. (y)
     
  9. pamamolf

    pamamolf Premium Member Premium Member

    4,084
    428
    83
    May 31, 2014
    Ratings:
    +834
    Local Time:
    12:09 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Working great for me also :)
     
  10. Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    5:09 PM
    What about passive aggressive bots?
     
  11. negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    12:09 AM
    1.9.10
    10.1.11
    I have no negative performance issue yet. So it looks good.

    Btw I moved the YandexBot to whitelist as "1" i think it is important searchbot after google, bing, yahoo so we shouldn't filter it.
     
  12. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    great
    thanks for confirmation
    define passive aggressive bots ?
    indeed the forums suddenly got more Russian vistors after yandex visited it this past week. I still have it set to 2 for rate limiting though
     
  13. Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    5:09 PM
    I can't, it was just a joke. Little bot humor.
     
  14. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    good one :LOL:
     
  15. pamamolf

    pamamolf Premium Member Premium Member

    4,084
    428
    83
    May 31, 2014
    Ratings:
    +834
    Local Time:
    12:09 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    I keep also yandex limited !
     
  16. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  17. Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    5:09 PM
  18. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Last edited: Apr 11, 2016
  19. eva2000

    eva2000 Administrator Staff Member

    54,647
    12,230
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,799
    Local Time:
    8:09 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  20. Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    10:09 PM
    1.9.x
    10.1.x
    I just received a massive flood(really massive), hitting random links at my site and with the following User Agent: "Pcore-HTTP/v0.24.5"

    I blocked it with a 444 response code and it resolved the issue.

    Does anyone know this user agent?
     
    Last edited: Aug 9, 2016