Want more timely Centmin Mod News Updates?
Become a Member

How to limit requests, and ban those hitting the limit

Discussion in 'Centmin Mod User Tutorials & Guides' started by Oxide, May 3, 2016.

  1. Oxide

    Oxide Active Member

    503
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    9:14 PM
    Requirements:
    - LUA RESTY.
    - Redis
    - Fail2Ban
    - ip_blacklist

    What we will do:
    - We will use ip_blacklist.lua to blacklist IP's that reaches the request limit.
    - We will limit requests to Documents to 20 requests a second.
    - When someone reaches that limit, they will be "banned" from accessing your server in 3600 seconds.
    - We will use redis to ban, redis stores the IP's in memory which is perfect & fast.

    0.) Make sure you have redis and fail2ban installed, you can run this command to install it:
    Code (Text):
    yum install redis fail2ban -y


    1.) Open up your nginx.conf, and make sure those are added within the http block.
    Code (Text):
    #Limit Requests PHP Documents
    limit_req_zone $binary_remote_addr zone=one:16m rate=20r/s;
    
    #Lua Packages etc
    lua_package_path "/usr/local/lib/lua/?.lua;;";
    lua_shared_dict ip_blacklist_cache 20m;


    2.) Make a new folder within: "/usr/local/nginx/conf" called "security_conf" - here we will place security optimizations. Place ip_blacklist.lua here: [Lua] -- a quick LUA access script for nginx to check IP addresses against an -- `ip_ - Pastebin.com (Remember to rename etc).

    3.) Make a new file inside security_conf called "global.conf". Put this content inside it:
    Code (Text):
    #DDoS Security Limit Requests
    access_by_lua_file /usr/local/nginx/conf/security_conf/ip_blacklist.lua;
    limit_req   zone=one  burst=20 nodelay;
    error_log /home/nginx/domains/nginx_error.log;


    We will make a global error.log, since I feel fail2ban works better that way instead of motioning multiply files.

    4.) Open up your domain.conf, then inside server block you add this:
    Code (Text):
    include /usr/local/nginx/conf/security_conf/global.conf;


    ++++++++++++++++++++++++++++++++++++++

    Now, all the nginx part is setup. You should be able to run "ngxreload" and restart without any errors.

    ++++++++++++++++++++++++++++++++++++++
    5.) We will now start setting up fail2ban, this is also quiet easy - copy & paste unless you want to modify something.

    Make a new file if it doesn't exist already called /etc/fail2ban/jail.local, inside there you add:
    Code (Text):
    [nginx-req-limit]
    
    enabled = true
    filter = nginx-req-limit
    action = ip_blacklist
    logpath = /home/nginx/domains/nginx_error.log
    findtime = 120
    bantime = 3600
    maxretry = 3


    6.) Inside /etc/fail2ban/action.d you make a new file called ip_blacklist.conf, you will put this content inside there:

    Code (Text):
    #
    # Author: Oxide @ Centminmod
    #
    # Blacklist option for LUA / REDIS
    #
    
    [Definition]
    
    # Option:  actionstart
    # Notes.:  command executed once at the start of Fail2Ban.
    # Values:  CMD
    #
    actionstart =
    
    # Option:  actionstop
    # Notes.:  command executed once at the end of Fail2Ban
    # Values:  CMD
    #
    actionstop =
    
    # Option:  actioncheck
    # Notes.:  command executed once before each actionban command
    # Values:  CMD
    #
    actioncheck =
    
    # Option:  actionban
    # Notes.:  command executed when banning an IP. Take care that the
    #          command is executed with Fail2Ban user rights.
    # Tags:    <ip>  IP address
    #          <failures>  number of failures
    #          <time>  unix timestamp of the ban time
    # Values:  CMD
    #
    actionban = redis-cli SADD ip_blacklist <ip>
    
    # Option:  actionunban
    # Notes.:  command executed when unbanning an IP. Take care that the
    #          command is executed with Fail2Ban user rights.
    # Tags:    <ip>  IP address
    #          <failures>  number of failures
    #          <time>  unix timestamp of the ban time
    # Values:  CMD
    #
    actionunban = redis-cli SREM ip_blacklist <ip>


    Now, run "service fail2ban restart" and you should be ready..

    Optional: You can add this to global.conf to block bad user-agents such as WordPress:
    Code (Text):
    #Block Bad User-Agents
    if ($http_user_agent ~* "PHP|curl|Wget|HTTrack|Verifying|PingBack|Joomla|Wordpress") { return 444; }
    if ($http_user_agent = "-") { return 444; }
    if ($http_user_agent = "") { return 444; }
    if ($http_user_agent = " ") { return 444; }


    This should be it, not too sure if i forget any steps. If you have any issues, feel free to ask away ! :D

    You can also move "limit_req zone=one burst=20 nodelay;" out of global.conf and place it inside the php block in php.conf, to limit only php files. I've had no issues limiting all files so far though.

    This will also work for CloudFlare.
     
    Last edited: May 3, 2016
    • Informative Informative x 5
  2. eva2000

    eva2000 Administrator Staff Member

    29,702
    6,707
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,009
    Local Time:
    9:14 PM
    Nginx 1.13.x
    MariaDB 5.5
    sweet @Oxide thanks for sharing seems it uses all my fav new tools nginx lua code, redis and nginx :)

    Just an added note for folks is that nginx lua module was disabled by default in 123.08stable and 123.09beta01 and can be re-enabled only in 123.09beta01 which supports nginx lua see Nginx 1.9.11 dynamic module compatibility

    persistent config setup at /etc/centminmod/custom_config.inc used
    Code (Text):
    ORESTY_LUANGINX='y'
    

    then run centmin.sh menu option 4 to recompile nginx 1.9.15 and higher for nginx lua support

    confirm via nginx -V command
     
    • Like Like x 4
  3. pamamolf

    pamamolf Well-Known Member

    2,659
    240
    63
    May 31, 2014
    Ratings:
    +422
    Local Time:
    2:14 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    • Informative Informative x 1
  4. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    1:14 PM
    1
    10
    Thanks @eva2000, step 5 causes fail2ban to fail to restart,

    Code:
    -- Unit fail2ban.service has begun starting up.
    Dec 28 10:52:41 ................ fail2ban-client[1262]: ERROR  Found no accessible config files for 'filter.d/ngi
    Dec 28 10:52:41 ................ fail2ban-client[1262]: ERROR  No section: 'Definition'
    Dec 28 10:52:41 ......... fail2ban-client[1262]: ERROR  No section: 'Definition'
    Dec 28 10:52:41 ....... fail2ban-client[1262]: ERROR  Unable to read the filter
    Dec 28 10:52:41 ......... fail2ban-client[1262]: ERROR  Errors in jail 'nginx-req-limit'. Skipping...
    Dec 28 10:52:41 .......... systemd[1]: fail2ban.service: control process exited, code=exited status=255
    Dec 28 10:52:41 ........ systemd[1]: Failed to start Fail2Ban Service.
    -- Subject: Unit fail2ban.service has failed
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,702
    6,707
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,009
    Local Time:
    9:14 PM
    Nginx 1.13.x
    MariaDB 5.5
    for post at https://community.centminmod.com/posts/42380/, I didn't mean follow the whole guide just part of setting up lua nginx in nginx.conf or nginx vhost
    Code (Text):
    #Lua Packages etc
    lua_package_path "/usr/local/lib/lua/?.lua;;";
    


    i haven't tried this whole guide but just this part is needed to setup lua packages provided by Centmin Mod when
    ORESTY_LUANGINX='y' set in persistent config /etc/centminmod/custom_config.inc and recompile nginx via centmin.sh menu option 4. Then follow the respective lua nginx modules documentation to setup your lua directives/options and code
     
  6. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    1:14 PM
    1
    10
    Crap, epic fail on my part, i'll do what you recommended.

    If i'm successful at rate limiting just one url ill post how i did it here so others can benefit.

    thanks
     
    • Like Like x 2
  7. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    1:14 PM
    1
    10
    got it working, thanks again @eva2000, gonna make a post soon with how i did it, if you dont mind ill include a part of the guide post u sent as i followed it before i made changes to my sites conf file, was actually easier than i though after.
     
    • Like Like x 1
  8. eva2000

    eva2000 Administrator Staff Member

    29,702
    6,707
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,009
    Local Time:
    9:14 PM
    Nginx 1.13.x
    MariaDB 5.5
    cheers, always happy to see end users post their own guides for stuff they do. Always interested to see how folks are using Centmin Mod :D
     
  9. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    1:14 PM
    1
    10
    Interesting i think i misspoke, it's not working because when i comment out the lines i added in my conf file i get the same results. Do you know if cloudflare ratelimits, i might have to switch the site on cloud flare to dns only for further testing.

    I can confirm some sort of rate limiting is going on, but not based on the criteria i set.
     
  10. eva2000

    eva2000 Administrator Staff Member

    29,702
    6,707
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,009
    Local Time:
    9:14 PM
    Nginx 1.13.x
    MariaDB 5.5
    cloudflare standard plans don't rate limit or have a high threshold limit unless you are participating in cloudflare traffic manager beta feature WebPerf - Cloudflare Traffic Control & Traffic Manager
     
  11. RoldanLT

    RoldanLT Well-Known Member

    3,868
    945
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,284
    Local Time:
    7:14 PM
    1.11
    10.2
    Can we replace Fail2ban with CSF on this tutorial?
     
  12. eva2000

    eva2000 Administrator Staff Member

    29,702
    6,707
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,009
    Local Time:
    9:14 PM
    Nginx 1.13.x
    MariaDB 5.5
    this tutorial doesn't touch CSF it adds ban ip to redis server database and lua nginx code reads it at nginx level AFAIK