Welcome to Centmin Mod Community
Become a Member

Security Sysadmin Block All Visitors using IPV6

Discussion in 'System Administration' started by ndha, Mar 27, 2018.

Tags:
  1. buik

    buik “It always seems impossible until it’s done.” Premium Member

    1,297
    350
    83
    Apr 29, 2016
    Ratings:
    +1,055
    Local Time:
    12:11 PM
    I have read this topic only briefly.
    @ndha Are you already using the CSF Blocklist?
    Enabling this + adding your own list based on scrapers you could find on the net.
    And you'll be fine.
     
  2. ndha

    ndha Member

    82
    10
    8
    Sep 28, 2014
    Ratings:
    +29
    Local Time:
    5:11 PM
    Latest
    10
    do you mean CSF Blocklist like csf -d xx.xx.x.xx ??
    Basically this scrappers doing like this..
    1. Their user login into APK
    2. Search something from that APK
    3. Then APK search into my site using Wordpress search function..
    4. It displays the search result from my site into APK..

    it cause very High CPU/LOAD for mysql and php.
     
  3. buik

    buik “It always seems impossible until it’s done.” Premium Member

    1,297
    350
    83
    Apr 29, 2016
    Ratings:
    +1,055
    Local Time:
    12:11 PM
    Nope thats the command line to block one single address.
    Check https://download.configserver.com/csf/readme.txt section
    IP Block Lists. To block whole lists of IP addresses at once.
     
  4. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Centmin Mod has csf-advancetweaks.sh for extending CSF Firewall blocklists too Centmin Mod 123.09beta01 extended CSF Firewall Tweaks and Beta Branch - revise inc/csftweaks.inc though you'd still need to do somework to block your specific scrappers if they aren't on one of the existing block lists. Though not much use if you have Cloudflare in front of your server as CSF Firewall won't be able to see the visitor's real IP as they will see Cloudflare's proxy IP.
     
  5. Tinkerd10

    Tinkerd10 New Member

    21
    3
    3
    Sep 28, 2016
    Ratings:
    +13
    Local Time:
    11:11 AM
    1.11
    5.5
    I have the same exact problem , i have being trying to find a solution for the past 4 months
    Manged to drop 3.5 billion request from 6.5b based on CF analytic using

    - Nignx Rate limit in general and for specific URL
    - Fail2ban with CF
    - Blocking range/ips and user-agnet with too many requests every 6 hours manually everyday
    - activating CF anti ddos for those specific URL
    - renaming search function and route url
    - Tracked down apps owners through github
    - cloudflare new worker feature
    - switching ips
    - removed all alternative domains and subdomain to narrow it down

    But at the end the amount of requests is too high, using rate limit and blocking using nginx it slow down the server even more

    Also looking through their code they use different user_agent with different Ips, and they even integrated the scrape / search function inside the app, so you will be blocking real users which is pointless
    Every day new Ip with 2k+ request in just few hours

    So now i closed the website only for members, if you want use the website you will have to access to it directly
    and monitor login/register url

    And that lead to all those request being sent to main route of the website, hoping they ll give up soon

    One solution left is to use google reCAPTCH atleast for search function
    instead of using cloudflare because there is a script in github to bypass Cloudflare anti-ddos and challenge
     
  6. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    nice tips there and thanks for sharing !
    ah smart renaming the search function routing url ! maybe something like Change the Search-URL of WordPress | WP Engineer

    (y) indeed alot of captcha plugins out there Search Results for “captcha” — WordPress Plugins but alot don't add to search but only to forms like contact.

    curious in what way are you using cloudflare workers ?
     
  7. Tinkerd10

    Tinkerd10 New Member

    21
    3
    3
    Sep 28, 2016
    Ratings:
    +13
    Local Time:
    11:11 AM
    1.11
    5.5
    I needed to take off some of the load off my server so i used non-origin script
    By blocking all requests for old route URL that has POST or GET
    Instead of using my own server

    It can be used also to rate limit and redirect
    But i can't go far with CF features, As the bill will be too high

    100+ tb Bandwidth, 4.5B req for a website that only server html css js images

    And that traffic can't be used for anything, it even drops ads revenue by dropping the ratio
     
  8. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    so your wrote up cloudflare workers to do that ?
     
  9. Tinkerd10

    Tinkerd10 New Member

    21
    3
    3
    Sep 28, 2016
    Ratings:
    +13
    Local Time:
    11:11 AM
    1.11
    5.5
    I used one of the pre-baked script from "Custom responses that don't hit origin servers"
    just changed in it what i needed Like

    Code:
    addEventListener('fetch', event => {
      event.respondWith(fetchAndApply(event.request))
    })
    
    async function fetchAndApply(request) { 
      if (request.method === 'POST' || request.method === 'GET') { //replaced PUT with GET
        return new Response('Sorry, this page is not available.', //add your own custom message : url, html
            { status: 403, statusText: 'Forbidden' }) // change status to 200 to mislead bots / Text: Success
      }
    
      return fetch(request)
    }
     
  10. ndha

    ndha Member

    82
    10
    8
    Sep 28, 2014
    Ratings:
    +29
    Local Time:
    5:11 PM
    Latest
    10
    @Tinkerd10 can you give tutorial about renaming search function and route url ??
    Also in which code you put this Custom responses that don't hit origin servers ??
    Thanks..
     
  11. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Thanks for the info. Cloudflare Workers seem like a very powerful thing to have offloading stuff to be processed on their edge servers :)
     
  12. Tinkerd10

    Tinkerd10 New Member

    21
    3
    3
    Sep 28, 2016
    Ratings:
    +13
    Local Time:
    11:11 AM
    1.11
    5.5
    @ndha For wordpress i'm using SearchWP
    But the website they were targeting is a custom made so i kept changing function name

    Despite everything was setup as it supposed to, The website load was still high
    So after playing around with some configuration i find out nginx cache wasn't working
    After checking tmpfs cache folder i found it empty even tho it has the correct permission and owner/group

    So i updated nginx cache config to this

    Code:
         fastcgi_cache  microcache;
         fastcgi_cache_key $scheme$host$request_uri$request_method;
         fastcgi_cache_valid 200 301 302 48h;
         fastcgi_cache_lock on;
         fastcgi_cache_use_stale updating error timeout invalid_header http_500;
    
         #fastcgi_hide_header Set-Cookie;
         #fastcgi_pass_header Set-Cookie;
         #fastcgi_pass_header Cookie;
         fastcgi_cache_methods GET HEAD;
    
         fastcgi_ignore_headers Cache-Control Expires;
    
         fastcgi_cache_bypass $skip_cache;
         fastcgi_no_cache $skip_cache;
         more_clear_headers Server; more_clear_headers "Pragma";
    
         add_header X-FastCGI-Cache $upstream_cache_status;
         add_header 'Access-Control-Allow-Origin' '*';
         add_header X-Frame-Options "SAMEORIGIN";
         add_header X-Content-Type-Options nosniff;

    Disabled csf -x (I had too for cache to works)
    I also store session and css/js files in tmpfs but for some reason it didn't work until i disable csf
    But i didn't investigate further yet

    Disabled drop.conf from all vhosts (Doing so i noticed the load dropped by 2 sec)
    Disabled access/error log on all vhosts expect nginx and main domain

    Enable CF anti ddos for everything else expect main domain
    "I believe CF started blockign VPN IP's also when you activate anti-ddos"


    Now the server load is under 0.7, website load without ads 0.553 ms (A new record for me )
    The result of 3 different domains "TLD" pointing to the same server

    [​IMG]

    [​IMG]
    [​IMG]
     
  13. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    strange as CSF firewall has nothing to do with nginx cache
     
  14. Tinkerd10

    Tinkerd10 New Member

    21
    3
    3
    Sep 28, 2016
    Ratings:
    +13
    Local Time:
    11:11 AM
    1.11
    5.5
    I didn't think so either, I honestly disable it to see if its affecting server load like drop.conf so i can optimise it
    and that's when i saw the cache started working, So i left it off
     
  15. eva2000

    eva2000 Administrator Staff Member

    44,750
    10,204
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,814
    Local Time:
    8:11 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    I'd get to bottom of issue as you'd want CSF Firewall enabled always !