Join the community today
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. deltahf

    deltahf Premium Member Premium Member

    340
    155
    43
    Jun 8, 2014
    Ratings:
    +247
    Local Time:
    3:21 AM
    Interesting point, I had not thought about that. Are Googlebot imposters not an issue as well?
     
  2. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah that MAYBE an issue too. I have much more extensive nginx block/rate limiting method for google and bingbot imposters. But can use more resources as it involves using nginx and geoip nginx module to analyse and monitor the entire ip address/ASN ranges for both google and bing/microsoft to allow only known ips belong to them through and rate limit or block the imposter bots which are coming from ips not known to be owned by google/bing/microsoft.
     
  3. negative

    negative Active Member

    358
    40
    28
    Apr 11, 2015
    Ratings:
    +82
    Local Time:
    10:21 AM
    1.9.10
    10.1.11
    I applied these configurations nginx.con and virtual hosts confs both but when i test the results:

    PHP:
    siege --c10 -r1 -"GetWeb" http://www.domain.com
    ** SIEGE 3.1.0
    ** Preparing 10 concurrent users for battle.
    The server is now under siege...
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.26 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.27 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.28 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.31 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.34 secs:    5395 bytes ==> GET  /
    HTTP/1.1 520   0.35 secs:    5395 bytes ==> GET  /
    done.

    Transactions:                  10 hits
    Availability
    :              100.00 %
    Elapsed time:                0.35 secs
    Data transferred
    :            0.05 MB
    Response time
    :                0.29 secs
    Transaction rate
    :           28.57 trans/sec
    Throughput
    :                0.15 MB/sec
    Concurrency
    :                8.23
    Successful transactions
    :           0
    Failed transactions
    :               0
    Longest transaction
    :            0.35
    Shortest transaction
    :            0.00
    It gives 520 Errors, not 444? I ' m using cloudflare, it may related with that?

    BTW: "Cliqzbot", "MegaIndex.ru" and "Yahoo! Slurp" are coming one of the top list user_agents for me.
    I will add the yahoo bot to whitelisted as 1 and "Cliqzbot" and "MegaIndex.ru" will be 3 of course.
     
    Last edited: Mar 7, 2016
  4. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yeah sounds like 520 is a cloudflare error status code Error 520: Web server is returning an unknown error – CloudFlare Support

    nginx 444 gives empty response so matchings what triggers cloudflare 520 so normal
     
  5. negative

    negative Active Member

    358
    40
    28
    Apr 11, 2015
    Ratings:
    +82
    Local Time:
    10:21 AM
    1.9.10
    10.1.11
    Added these bots;

    "~*Pixray" 3;
    "~*MegaIndex.ru" 3;
    "~*Cliqzbot" 3;

    # bots whitelisted
    "~*Googlebot" 1;
    "~*bingbot" 1;
    "~*Yahoo!\ Slurp" 1;
     
    • Informative Informative x 3
  6. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    nice i updated 1st post with megaindex.ru and cliqzbot block and added yahoo to whitelist
     
    • Like Like x 1
  7. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    so how's everyones experiments with the 1st post outlined bad bot rate limiting/blocking configurations ?

    more tests please so can polish it for inclusion into 123.09beta01 config setups :D
     
    • Like Like x 1
  8. deltahf

    deltahf Premium Member Premium Member

    340
    155
    43
    Jun 8, 2014
    Ratings:
    +247
    Local Time:
    3:21 AM
    It's been working great for me. (y)
     
    • Like Like x 1
  9. pamamolf

    pamamolf Premium Member Premium Member

    3,575
    344
    83
    May 31, 2014
    Ratings:
    +664
    Local Time:
    10:21 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Working great for me also :)
     
  10. Jimmy

    Jimmy Well-Known Member

    1,646
    353
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +884
    Local Time:
    3:21 AM
    1.17.x
    MariaDB 10.3.x
    What about passive aggressive bots?
     
  11. negative

    negative Active Member

    358
    40
    28
    Apr 11, 2015
    Ratings:
    +82
    Local Time:
    10:21 AM
    1.9.10
    10.1.11
    I have no negative performance issue yet. So it looks good.

    Btw I moved the YandexBot to whitelist as "1" i think it is important searchbot after google, bing, yahoo so we shouldn't filter it.
     
    • Informative Informative x 1
  12. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    great
    thanks for confirmation
    define passive aggressive bots ?
    indeed the forums suddenly got more Russian vistors after yandex visited it this past week. I still have it set to 2 for rate limiting though
     
  13. Jimmy

    Jimmy Well-Known Member

    1,646
    353
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +884
    Local Time:
    3:21 AM
    1.17.x
    MariaDB 10.3.x
    I can't, it was just a joke. Little bot humor.
     
    • Funny Funny x 3
  14. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    good one :LOL:
     
  15. pamamolf

    pamamolf Premium Member Premium Member

    3,575
    344
    83
    May 31, 2014
    Ratings:
    +664
    Local Time:
    10:21 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    I keep also yandex limited !
     
  16. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    • Like Like x 3
  17. Jimmy

    Jimmy Well-Known Member

    1,646
    353
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +884
    Local Time:
    3:21 AM
    1.17.x
    MariaDB 10.3.x
    • Like Like x 3
  18. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Last edited: Apr 11, 2016
    • Informative Informative x 1
  19. eva2000

    eva2000 Administrator Staff Member

    42,312
    9,557
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,724
    Local Time:
    6:21 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  20. Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    8:21 AM
    1.9.x
    10.1.x
    I just received a massive flood(really massive), hitting random links at my site and with the following User Agent: "Pcore-HTTP/v0.24.5"

    I blocked it with a 444 response code and it resolved the issue.

    Does anyone know this user agent?
     
    Last edited: Aug 9, 2016