Get the most out of your Centmin Mod LEMP stack
Become a Member

Sysadmin Blocking robots.txt

Discussion in 'System Administration' started by inthecloudblog, Aug 12, 2016.

  1. inthecloudblog

    inthecloudblog Active Member

    194
    36
    28
    Jan 26, 2016
    Ratings:
    +82
    Local Time:
    10:08 AM
    1.4.6
    Hi . I'm not sure if this is the proper category.
    I want to block robots.txt until one of the webs is done.. any simple way to achieve such thing?

    EDIT:

    just read

    Blocking Certain Robots | Centmin Mod Community

    still remains unclear to me how to make my web unreacheable to yahoo,google, bing,etc..
     
    Last edited: Aug 12, 2016
  2. eva2000

    eva2000 Administrator Staff Member

    30,961
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    11:08 PM
    Nginx 1.13.x
    MariaDB 5.5
    • Friendly Friendly x 1
  3. inthecloudblog

    inthecloudblog Active Member

    194
    36
    28
    Jan 26, 2016
    Ratings:
    +82
    Local Time:
    10:08 AM
    1.4.6
    I will re read about the robots stuff.. ( I was reading while installing centminmod)..

    I was waiting for centminmod to be installed and after it had been done I entered :
    nano /usr/local/nginx/conf/nginx.conf

    I found ( I use CF), so I've uncommented since my domain is not resolving :

    include /usr/local/nginx/conf/cloudflare.conf;


    and I also restarted nginx...

    still does not resolve.. Should I open a new thread?

    I get a 522 error.. ( using the beta build)


    I'm typing nonsense.. going to bed..


    EDIT: damn ... was entering old ips into cf...need some sleep
     
    Last edited: Aug 12, 2016
    • Funny Funny x 1
  4. eva2000

    eva2000 Administrator Staff Member

    30,961
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    11:08 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah that would cause 522s
     
  5. inthecloudblog

    inthecloudblog Active Member

    194
    36
    28
    Jan 26, 2016
    Ratings:
    +82
    Local Time:
    10:08 AM
    1.4.6
    I've realized that :
    1- I posted in a non suitable section.
    2- Posted asking something when the setup was in progress.
    3- The original post was derrailed by me just because I was way too drowsy and used the old IP's from a droplet( DO) I had just destroyed and placing those in CF

    I really appreciate George's huge and steady dedication and effort for this project. Also please accept and apology for being so confusing
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    30,961
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    11:08 PM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1