Join the community today
Register Now

robots.txt

Discussion in 'System Administration' started by pamamolf, Oct 11, 2014.

  1. pamamolf

    pamamolf Premium Member Premium Member

    3,167
    304
    83
    May 31, 2014
    Ratings:
    +551
    Local Time:
    10:36 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I think it will be good to have a topic for robots.txt (many bots ignore so maybe do the job in firewall? ) and post some ip's and info to protect from hitting the server as crazy?

    # Baidu:
    deny from 91.197.129.0/24
    deny from 185.10.104.0/24
    deny from 188.129.143.0/24
    deny from 113.119.37.227
    deny from 123.125.71.0/24

    # Ahrefs:
    deny from 37.58.100.0/24
    deny from 209.222.8.0/24
    deny from 173.199.115.0/24
    deny from 173.199.116.0/24
    deny from 173.199.120.0/24

    # Trendiction:
    deny from 144.76.23.0/24

    # Random
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 61.135.192.0/18
    deny from 77.88.0.0/18
    deny from 77.91.224.0/24
    deny from 87.250.224.0/19
    deny from 92.241.182.0/24
    deny from 93.158.128.0/18
    deny from 95.108.128.0/17
    deny from 119.63.192.0/21
    deny from 123.125.64.0/18
    deny from 178.154.128.0/17
    deny from 180.76.0.0/16
    deny from 182.118.0.0/16
    deny from 193.47.80.0/24
    deny from 213.180.192.0/19
    deny from 220.181.0.0/18
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 220.181.0.0/16


    What do you think?

    Thanks
     
  2. eva2000

    eva2000 Administrator Staff Member

    38,023
    8,349
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,862
    Local Time:
    6:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  3. pamamolf

    pamamolf Premium Member Premium Member

    3,167
    304
    83
    May 31, 2014
    Ratings:
    +551
    Local Time:
    10:36 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Yes i know but i create the topic as maybe someone try to add some bad ip's :)
     
..