/ Register

Want to subscribe to topics you're interested in?
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

Tags:
Previous Thread Next Thread
Loading...
Page 2 of 8
  1. Mar 6, 2016 #21
    deltahf

    deltahf Premium Member Premium Member

    534
    237
    43
    Jun 8, 2014
    Ratings:
    +431
    Local Time:
    2:04 PM
    Interesting point, I had not thought about that. Are Googlebot imposters not an issue as well?
     
  2. Mar 6, 2016 #22
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    Yeah that MAYBE an issue too. I have much more extensive nginx block/rate limiting method for google and bingbot imposters. But can use more resources as it involves using nginx and geoip nginx module to analyse and monitor the entire ip address/ASN ranges for both google and bing/microsoft to allow only known ips belong to them through and rate limit or block the imposter bots which are coming from ips not known to be owned by google/bing/microsoft.
     
  3. Mar 7, 2016 #23
    negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    9:04 PM
    1.9.10
    10.1.11
    I applied these configurations nginx.con and virtual hosts confs both but when i test the results:

    PHP:
    siege -b -c10 -r1 -A "GetWeb" http://www.domain.com
    ** SIEGE 3.1.0
    ** Preparing 10 concurrent users for battle.
    The server is now under siege...
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.28 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.31 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.34 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.35 secs:    5395 bytes ==> GET  /
    done.

    Transactions:                  10 hits
    Availability    :              100.00 %
    Elapsed time:                0.35 secs
    Data transferred    :            0.05 MB
    Response time    :                0.29 secs
    Transaction rate    :           28.57 trans/sec
    Throughput    :                0.15 MB/sec
    Concurrency    :                8.23
    Successful transactions    :           0
    Failed transactions    :               0
    Longest transaction    :            0.35
    Shortest transaction    :            0.00
    It gives 520 Errors, not 444? I ' m using cloudflare, it may related with that?

    BTW: "Cliqzbot", "MegaIndex.ru" and "Yahoo! Slurp" are coming one of the top list user_agents for me.
    I will add the yahoo bot to whitelisted as 1 and "Cliqzbot" and "MegaIndex.ru" will be 3 of course.
     
    Last edited: Mar 7, 2016
  4. Mar 7, 2016 #24
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    yeah sounds like 520 is a cloudflare error status code Error 520: Web server is returning an unknown error – CloudFlare Support

    nginx 444 gives empty response so matchings what triggers cloudflare 520 so normal
     
  5. Mar 7, 2016 #25
    negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    9:04 PM
    1.9.10
    10.1.11
    Added these bots;

    "~*Pixray" 3;
    "~*MegaIndex.ru" 3;
    "~*Cliqzbot" 3;
    # bots whitelisted
    "~*Googlebot" 1;
    "~*bingbot" 1;
    "~*Yahoo!\ Slurp" 1; ​
     
  6. Mar 8, 2016 #26
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    nice i updated 1st post with megaindex.ru and cliqzbot block and added yahoo to whitelist
     
  7. Mar 28, 2016 #27
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    so how's everyones experiments with the 1st post outlined bad bot rate limiting/blocking configurations ?

    more tests please so can polish it for inclusion into 123.09beta01 config setups :D
     
  8. Mar 28, 2016 #28
    deltahf

    deltahf Premium Member Premium Member

    534
    237
    43
    Jun 8, 2014
    Ratings:
    +431
    Local Time:
    2:04 PM
    It's been working great for me. (y)
     
  9. Mar 28, 2016 #29
    pamamolf

    pamamolf Premium Member Premium Member

    3,991
    416
    83
    May 31, 2014
    Ratings:
    +806
    Local Time:
    9:04 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Working great for me also :)
     
  10. Mar 28, 2016 #30
    Jimmy

    Jimmy Premium Member Premium Member

    1,761
    381
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +967
    Local Time:
    2:04 PM
    1.17.x
    MariaDB 10.3.x
    What about passive aggressive bots?
     
  11. Mar 28, 2016 #31
    negative

    negative Active Member

    415
    50
    28
    Apr 11, 2015
    Ratings:
    +98
    Local Time:
    9:04 PM
    1.9.10
    10.1.11
    I have no negative performance issue yet. So it looks good.

    Btw I moved the YandexBot to whitelist as "1" i think it is important searchbot after google, bing, yahoo so we shouldn't filter it.
     
  12. Mar 28, 2016 #32
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    great
    thanks for confirmation
    define passive aggressive bots ?
    indeed the forums suddenly got more Russian vistors after yandex visited it this past week. I still have it set to 2 for rate limiting though
     
  13. Mar 29, 2016 #33
    Jimmy

    Jimmy Premium Member Premium Member

    1,761
    381
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +967
    Local Time:
    2:04 PM
    1.17.x
    MariaDB 10.3.x
    I can't, it was just a joke. Little bot humor.
     
  14. Mar 29, 2016 #34
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    good one :LOL:
     
  15. Mar 29, 2016 #35
    pamamolf

    pamamolf Premium Member Premium Member

    3,991
    416
    83
    May 31, 2014
    Ratings:
    +806
    Local Time:
    9:04 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I keep also yandex limited !
     
  16. Apr 7, 2016 #36
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
  17. Apr 9, 2016 #37
    Jimmy

    Jimmy Premium Member Premium Member

    1,761
    381
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +967
    Local Time:
    2:04 PM
    1.17.x
    MariaDB 10.3.x
  18. Apr 11, 2016 #38
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
    Last edited: Apr 11, 2016
  19. Apr 16, 2016 #39
    eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:04 AM
    Nginx 1.21.x
    MariaDB 10.x
  20. Aug 9, 2016 #40
    Revenge

    Revenge Active Member

    467
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    7:04 PM
    1.9.x
    10.1.x
    I just received a massive flood(really massive), hitting random links at my site and with the following User Agent: "Pcore-HTTP/v0.24.5"

    I blocked it with a 444 response code and it resolved the issue.

    Does anyone know this user agent?
     
    Last edited: Aug 9, 2016
Previous Thread Next Thread
Loading...
Page 2 of 8

.