Learn about Centmin Mod LEMP Stack today
Register Now

Beta Branch autoprotect.sh - apache .htaccess check & migration to nginx deny all

Discussion in 'Beta release code' started by eva2000, May 12, 2016.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:50 PM
    Nginx 1.13.x
    MariaDB 5.5
    Nginx web servers do not support Apache web server .htaccess files. So any web application which relies on .htaccess to protect directories will be fully open to the public on Nginx web servers unless your Nginx vhost config file has specific location directory matches to deny and/or password protect them.

    I'm developing an autoprotect.sh script (/usr/local/src/centminmod/tools/autoprotect.sh) which basically checks entire Centmin Mod LEMP stack server's Nginx vhost site's public web accessible directories for .htaccess files that exist. Then grabbing their directory path to .htaccess file and auto generating an Nginx equivalent location match and deny all setup when a web url curl check reveals that the HTTP status code of the directory path is not matching 403 or 404. Which means if any directories which already have equivalent Nginx deny all (403) or 404 HTTP status code based location matches will be skipped and no auto generated nginx location auto protect rules will be made. Update had to disable curl 403/404 prechecks though as running subsequent autoprotect.sh runs would empty out the domain's include auto protect file as 403 is detected from previous autoprotect.sh run. So instead all .htaccess 'deny from all' detected directories now get auto generated Nginx equivalent location match and deny all setups except if you want to manually bypass the directory from auto protection via a .autoprotect-bypass file - details below here.

    An example of Apache directory's .htaccess deny all file at /privatedirectory/.htaccess
    Code (Text):
    Order deny,allow
    Deny from all
    

    An example of Nginx directory at /privatedirectory having deny all set for 403 permission denied
    Code (Text):
    location ~* ^/privatedirectory { deny all; }
    

    Running autoprotect.sh will transverse through all Nginx vhost sites at /home/nginx/domains/ looking for any .htaccess files which do not already return a 403 or 404 HTTP status and generate the corresponding Nginx location match deny all config in an include file specific for each domain i.e. /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf where domain name = domain.com
    Code (Text):
    /usr/local/src/centminmod/tools/autoprotect.sh
    generated nginx include file: /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf
    
    autoprotect.sh run completed...
    

    contents of include file /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf where autoprotect.sh detected a matching .htaccess file at
    /home/nginx/domains/domain.com/public/privatedirectory/.htaccess which needs a nginx equivalent location match deny all rule generated
    Code (Text):
    # /home/nginx/domains/domain.com/public/privatedirectory
    location ~* ^/privatedirectory { deny all; }
    

    Then in your domain.com nginx vhost file you'd have the autoprotect include file placed above the root definition i.e.
    Code (Text):
      include /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf;
      root /home/nginx/domains/domain.com/public;
    

    setup a cronjob to run autoprotect.sh every 4 hours or whatever you want frequency wise. The autoprotect.sh script will also nginx service reload when triggered so any updated newly found .htaccess files will re-populate each individual site autoprotect include files i.e. /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf with updated rules.
    Code (Text):
    0 */4 * * * /usr/local/src/centminmod/tools/autoprotect.sh
    

    /usr/local/src/centminmod/tools/autoprotect.sh is not available yet, still doing internal testing for now.
     
    Last edited: Oct 21, 2016
    • Winner Winner x 8
  2. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:50 PM
    Nginx 1.13.x
    MariaDB 5.5
  3. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    7:50 PM
    Nginx 1.13.x
    MariaDB 5.5
    Update tools/autoprotect.sh to add .autoprotect-bypass file option

    Give end user the option to manually bypass autoprotect.sh script and NOT create a nginx deny all location match by manually creating a .autoprotect-bypass file within the directory you want to bypass and exclude from autoprotect.sh. You may want to do this if your nginx deny location match for a directory involves whitelisting ip addresses' access to the directory.

    The autoprotect.sh has no way of detecting if you already setup a nginx deny location match for a directory but also added whitelisted ips for the nginx location so will try to generate a nginx deny all rule if .htaccess file is in the directory with contents of 'deny from all' text. If you manually create a /privatedirectoryname/.autoprotect-bypass file, autoprotect.sh script skips generating the nginx deny all rule.

    Tip: if you need a complete list of directories which contain .htaccess deny from all files, you can just run tools/autoprotect.sh manually and then look in the contents of generated autoprotect include files i.e. for domain.com include file at
    /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf. Then see which directories you need to drop a .autoprotect-bypass file to exclude from autoprotecting and re-run tools/autoprotect.sh to exclude them.

    Can also grep for word location within /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf to see the directories too.
    Code (Text):
    grep location /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf
    


    Notes



    Using Nginx with any web app like wordpress requires understanding the web app's author intention to block or prevent sensitive files or directories with .htaccess and then understanding that Nginx doesn't support .htaccess so you need to write a custom location match/protection for each of those directories with .htaccess files provided by web app's author for security.

    The tools/autoprotect.sh Beta Branch - autoprotect.sh - apache .htaccess check & migration to nginx deny all aims to make it easier to identify and alert owners to which directories/files you need to more closely look at to manually write up a nginx custom rule to do the equivalent task as intended by .htaccess file. As per above outline, you can check with grep command which directories you need to pay closer attention to with command (and bypass if needed)

    where domain.com is yourdomain.com name
    Code (Text):
    grep location /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf
    

    Once, you have the nginx custom rule placed in nginx vhost or use tools/autoprotect.sh provided rule if it works, you can setup .autoprotect-bypass file in the relevant directory + re-run /usr/local/src/centminmod/tools/autoprotect.sh Beta Branch - autoprotect.sh - apache .htaccess check & migration to nginx deny all to exclude tools/autoprotect.sh from auto creating a rule as you manually added one in nginx vhost will do the equivalent task as intended by .htaccess file.
     
    Last edited: Aug 5, 2017
    • Like Like x 3
Thread Status:
Not open for further replies.