Join the community today
Register Now

robots.txt

Discussion in 'System Administration' started by pamamolf, Oct 11, 2014.

  1. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    7:36 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I think it will be good to have a topic for robots.txt (many bots ignore so maybe do the job in firewall? ) and post some ip's and info to protect from hitting the server as crazy?

    # Baidu:
    deny from 91.197.129.0/24
    deny from 185.10.104.0/24
    deny from 188.129.143.0/24
    deny from 113.119.37.227
    deny from 123.125.71.0/24

    # Ahrefs:
    deny from 37.58.100.0/24
    deny from 209.222.8.0/24
    deny from 173.199.115.0/24
    deny from 173.199.116.0/24
    deny from 173.199.120.0/24

    # Trendiction:
    deny from 144.76.23.0/24

    # Random
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 61.135.192.0/18
    deny from 77.88.0.0/18
    deny from 77.91.224.0/24
    deny from 87.250.224.0/19
    deny from 92.241.182.0/24
    deny from 93.158.128.0/18
    deny from 95.108.128.0/17
    deny from 119.63.192.0/21
    deny from 123.125.64.0/18
    deny from 178.154.128.0/17
    deny from 180.76.0.0/16
    deny from 182.118.0.0/16
    deny from 193.47.80.0/24
    deny from 213.180.192.0/19
    deny from 220.181.0.0/18
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 220.181.0.0/16


    What do you think?

    Thanks
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    2:36 AM
    Nginx 1.13.x
    MariaDB 5.5
  3. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    7:36 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Yes i know but i create the topic as maybe someone try to add some bad ip's :)