Learn about Centmin Mod LEMP Stack today
Become a Member

robots.txt

Discussion in 'System Administration' started by pamamolf, Oct 11, 2014.

  1. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    12:16 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    I think it will be good to have a topic for robots.txt (many bots ignore so maybe do the job in firewall? ) and post some ip's and info to protect from hitting the server as crazy?

    # Baidu:
    deny from 91.197.129.0/24
    deny from 185.10.104.0/24
    deny from 188.129.143.0/24
    deny from 113.119.37.227
    deny from 123.125.71.0/24

    # Ahrefs:
    deny from 37.58.100.0/24
    deny from 209.222.8.0/24
    deny from 173.199.115.0/24
    deny from 173.199.116.0/24
    deny from 173.199.120.0/24

    # Trendiction:
    deny from 144.76.23.0/24

    # Random
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 61.135.192.0/18
    deny from 77.88.0.0/18
    deny from 77.91.224.0/24
    deny from 87.250.224.0/19
    deny from 92.241.182.0/24
    deny from 93.158.128.0/18
    deny from 95.108.128.0/17
    deny from 119.63.192.0/21
    deny from 123.125.64.0/18
    deny from 178.154.128.0/17
    deny from 180.76.0.0/16
    deny from 182.118.0.0/16
    deny from 193.47.80.0/24
    deny from 213.180.192.0/19
    deny from 220.181.0.0/18
    deny from 27.153.128.0/17
    deny from 27.152.0.0/13
    deny from 121.204.0.0/14
    deny from 120.40.0.0/14
    deny from 117.24.0.0/13
    deny from 222.76.0.0/14
    deny from 220.249.160.0/19
    deny from 120.32.0.0/13
    deny from 110.80.0.0/13
    deny from 220.160.0.0/15
    deny from 220.162.0.0/16
    deny from 220.181.0.0/16


    What do you think?


    Thanks
     
  2. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,739
    Local Time:
    8:16 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. pamamolf

    pamamolf Premium Member Premium Member

    4,074
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    12:16 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Yes i know but i create the topic as maybe someone try to add some bad ip's :)
     
  4. MaximilianKohler

    MaximilianKohler Member

    196
    5
    18
    Jun 23, 2023
    Ratings:
    +28
    Local Time:
    2:16 PM
    How would one use a robots.txt with centmin mod? IE: this recent Xenforo discussion recommended using one. I'm not sure where that file should be placed or what should be in it.
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,739
    Local Time:
    8:16 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    robots.txt is always placed in web root so domain.com/robots.txt at /home/nginx/domains/domain.com/public/robots.txt
     
  6. MaximilianKohler

    MaximilianKohler Member

    196
    5
    18
    Jun 23, 2023
    Ratings:
    +28
    Local Time:
    2:16 PM
    You listed both of these:
    The 2nd one is correct?
     
  7. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,739
    Local Time:
    8:16 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    domain.com/robots.txt referred to https://domain.com/robots.txt