Learn about Centmin Mod LEMP Stack today
Register Now

robots.txt

Discussion in 'System Administration' started by pamamolf, Oct 11, 2014.

  1. pamamolf

    pamamolf Well-Known Member

    4,022
    421
    83
    May 31, 2014
    Ratings:
    +816
    Local Time:
    1:20 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    I think it will be good to have a topic for robots.txt (many bots ignore so maybe do the job in firewall? ) and post some ip's and info to protect from hitting the server as crazy?

    # Baidu:
    deny from 91.197.129.0/24
    deny from 185.10.104.0/24
    deny from 188.129.143.0/24
    deny from 113.119.37.227
    deny from 123.125.71.0/24

    # Ahrefs:
    deny from 37.58.100.0/24
    deny from 209.222.8.0/24
    deny from 173.199.115.0/24
    deny from 173.199.116.0/24
    deny from 173.199.120.0/24

    # Trendiction:
    deny from 144.76.23.0/24

    # Random
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 61.135.192.0/18
    deny from 77.88.0.0/18
    deny from 77.91.224.0/24
    deny from 87.250.224.0/19
    deny from 92.241.182.0/24
    deny from 93.158.128.0/18
    deny from 95.108.128.0/17
    deny from 119.63.192.0/21
    deny from 123.125.64.0/18
    deny from 178.154.128.0/17
    deny from 180.76.0.0/16
    deny from 182.118.0.0/16
    deny from 193.47.80.0/24
    deny from 213.180.192.0/19
    deny from 220.181.0.0/18
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 220.181.0.0/16


    What do you think?

    Thanks

     
  2. eva2000

    eva2000 Administrator Staff Member

    50,455
    11,659
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,078
    Local Time:
    8:20 AM
    Nginx 1.25.x
    MariaDB 10.x
  3. pamamolf

    pamamolf Well-Known Member

    4,022
    421
    83
    May 31, 2014
    Ratings:
    +816
    Local Time:
    1:20 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Yes i know but i create the topic as maybe someone try to add some bad ip's :)