Get the most out of your Centmin Mod LEMP stack
Become a Member

cfautouam - CloudFlare Under Attack Mode Automation

Discussion in 'Centmin Mod User Tutorials & Guides' started by Rake-GH, Apr 11, 2020.

Tags:
  1. Rake-GH

    Rake-GH Active Member

    179
    93
    28
    Jul 29, 2019
    USA
    Ratings:
    +144
    Local Time:
    6:38 PM
    default
    default
    I made this script as a proof of concept to learn and practice bash scripting, it came out pretty good and I'm using it on a couple sites including my Xenforo board. I hope someone finds this useful.

    The video shows you how it works and I show you how to code it.



    Get updated source code from the github repo:
    guided-hacking/cfautouam

    What does it do
    Enables Cloudflare's Under Attack Mode based on CPU load percentage using the Cloudflare API.

    Why
    Running your site on Under Attack Mode permanently is not great for visitors. This script will enable it under high CPU load which is indicative of a DDOS attack.

    Warning
    This is a beta script but it works fine in all my testing.

    How?
    It creates a service that runs on a timer, which executes our main shell script which gets the current CloudFlare Security Level and checks the CPU usage. If CPU usage is above our defined limit, it uses the CloudFlare API to set the Security Level to Under Attack Mode. If CPU usage normalizes and the time limit has passed, it will change the Security Level back to your defined "normal" Security Level.

    How to install
    Navigate to the parent path where you want to install. If you want to install to /home/cfautouam then navigate to /home

    wget https://raw.githubusercontent.com/guided-hacking/cfautouam/master/cfautouam.sh;
    Define the parent path where you want to install the script, your Cloudflare email, API key, Zone ID, regular_status and regular_status_s as it related to your normal security level


    Code:
    mkdir cfautouam;
    cp cfautouam.sh cfautouam/cfautouam.sh
    cd cfautouam;
    chmod +x cfautouam.sh;
    ./cfautouam.sh -install;
    
    It's now installed and running from the defined parent path, check the logs and confirm it's working. You can delete the original file.

    After confirming it works, set debug level to 0.

    Command Line Arguments
    -install : installs and enables service
    -uninstall : uninstalls and then deletes the sub folder
    -disable_script : temporarily disables the service from running
    -enable_script : re-enables the service
    -enable_uam : enables Under Attack Mode manually
    -disable_uam : disables Under Attack Mode manually

    Notes
    This script was designed to run out of it's own separate folder, if you change that you may have problems.

    source from video:
    Code:
    #!/bin/bash
    # Cloudflare Auto Under Attack Mode = CF Auto UAM
    # version 0.9beta
    
    # Security Level Enums
    SL_OFF=0
    SL_ESSENTIALLY_OFF=1
    SL_LOW=2
    SL_MEDIUM=3
    SL_HIGH=4
    SL_UNDER_ATTACK=5
    
    SL_OFF_S="off"
    SL_ESSENTIALLY_OFF_S="essentially_off"
    SL_LOW_S="low"
    SL_MEDIUM_S="medium"
    SL_HIGH_S="high"
    SL_UNDER_ATTACK_S="under_attack"
    
    #config
    debug_mode=1 # 1 = true, 0 = false
    install_parent_path="/home"
    cf_email=""
    cf_apikey=""
    cf_zoneid=""
    upper_cpu_limit=20 # 10 = 10% load, 20 = 20% load.  Total load, taking into account # of cores
    lower_cpu_limit=5
    regular_status=$SL_HIGH
    regular_status_s=$SL_HIGH_S
    time_limit_before_revert=$((60 * 10)) # 10 minutes by default
    #end config
    
    # Functions
    
    install() {
      mkdir $install_parent_path"/cfautouam"
    
      cat >$install_parent_path"/cfautouam/cfautouam.service" <<EOF
    [Unit]
    Description=Enable Cloudflare Under Attack Mode under high load
    [Service]
    ExecStart=$install_parent_path/cfautouam/cfautouam.sh
    EOF
    
      cat >$install_parent_path"/cfautouam/cfautouam.timer" <<EOF
    [Unit]
    Description=Enable Cloudflare Under Attack Mode under high load
    [Timer]
    OnBootSec=60
    OnUnitActiveSec=7
    AccuracySec=1
    [Install]
    WantedBy=timers.target
    EOF
    
      chmod +x $install_parent_path"/cfautouam/cfautouam.service"
      systemctl enable $install_parent_path"/cfautouam/cfautouam.timer"
      systemctl enable $install_parent_path"/cfautouam/cfautouam.service"
      systemctl start cfautouam.timer
      exit
    }
    
    uninstall() {
      systemctl stop cfautouam.timer
      systemctl stop cfautouam.service
      systemctl disable cfautouam.timer
      systemctl disable cfautouam.service
      rm -R $install_parent_path"/cfautouam"
      exit
    }
    
    disable_uam() {
      curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$cf_zoneid/settings/security_level" \
        -H "X-Auth-Email: $cf_email" \
        -H "X-Auth-Key: $cf_apikey" \
        -H "Content-Type: application/json" \
        --data "{\"value\":\"$regular_status_s\"}" &>/dev/null
    
      # log time
      date +%s >$install_parent_path"/cfautouam/uamdisabledtime"
    
      echo "$(date) - cfautouam - CPU Load: $curr_load - Disabled UAM" >>$install_parent_path"/cfautouam/cfautouam.log"
    }
    
    enable_uam() {
      curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$cf_zoneid/settings/security_level" \
        -H "X-Auth-Email: $cf_email" \
        -H "X-Auth-Key: $cf_apikey" \
        -H "Content-Type: application/json" \
        --data '{"value":"under_attack"}' &>/dev/null
    
      # log time
      date +%s >$install_parent_path"/cfautouam/uamenabledtime"
    
      echo "$(date) - cfautouam - CPU Load: $curr_load - Enabled UAM" >>$install_parent_path"/cfautouam/cfautouam.log"
    }
    
    get_current_load() {
      numcores=$(grep -c 'model name' /proc/cpuinfo)
      currload=$(uptime | awk -F'average:' '{ print $2 }' | awk '{print $1}' | sed 's/,/ /')
      currload=$(bc <<<"scale=2; $currload / $numcores * 100")
      currload=${currload%.*}
      return $currload
    }
    
    get_security_level() {
      curl -X GET "https://api.cloudflare.com/client/v4/zones/$cf_zoneid/settings/security_level" \
        -H "X-Auth-Email: $cf_email" \
        -H "X-Auth-Key: $cf_apikey" \
        -H "Content-Type: application/json" 2>/dev/null |
        awk -F":" '{ print $4 }' | awk -F',' '{ print $1 }' | tr -d '"' >$install_parent_path"/cfautouam/cfstatus"
    
      security_level=$(cat $install_parent_path"/cfautouam/cfstatus")
    
      case $security_level in
      "off")
        return $SL_OFF
        ;;
      "essentially_off")
        return $SL_ESSENTIALLY_OFF
        ;;
      "low")
        return $SL_LOW
        ;;
      "medium")
        return $SL_MEDIUM
        ;;
      "high")
        return $SL_HIGH
        ;;
      "under_attack")
        return $SL_UNDER_ATTACK
        ;;
      *)
        return 100 # error
        ;;
      esac
    }
    
    main() {
      # Get current protection level & load
      get_security_level
      curr_security_level=$?
      get_current_load
      curr_load=$?
    
      if [ $debug_mode == 1 ]; then
        #curr_load=5
        time_limit_before_revert=30
      fi
    
      # If UAM was recently enabled
    
      if [[ $curr_security_level == "$SL_UNDER_ATTACK" ]]; then
        uam_enabled_time=$(<uamenabledtime)
        currenttime=$(date +%s)
        timediff=$((currenttime - uam_enabled_time))
    
        # If time limit has passed
        if [[ $timediff -gt $time_limit_before_revert ]]; then
    
          # If time limit has passed & cpu limit has normalized
          if [[ $curr_load -lt $upper_cpu_limit ]]; then
            if [ $debug_mode == 1 ]; then
              echo "$(date) - cfautouam - CPU Load: $curr_load - CPU Below threshhold, time limit has passed" >>$install_parent_path"/cfautouam/cfautouam.log"
            fi
            disable_uam
          else
            if [ $debug_mode == 1 ]; then
              echo "$(date) - cfautouam - CPU Load: $curr_load - CPU Above threshhold, time limit has passed - do nothing" >>$install_parent_path"/cfautouam/cfautouam.log"
            fi
          fi
    
        else
          if [ $debug_mode == 1 ]; then
            echo "$(date) - cfautouam - CPU Load: $curr_load - UAM already set, waiting out time limit" >>$install_parent_path"/cfautouam/cfautouam.log"
          fi
        fi
        exit
      fi
    
      # Enable and Disable UAM based on load
    
      #if load is higher than limit
      if [[ $curr_load -gt $upper_cpu_limit && $curr_security_level == "$regular_status" ]]; then
        enable_uam
      #else if load is lower than limit
      elif [[ $curr_load -lt $lower_cpu_limit && $curr_security_level == "$SL_UNDER_ATTACK" ]]; then
        disable_uam
      else
        if [ $debug_mode == 1 ]; then
          echo "$(date) - cfautouam - CPU Load: $curr_load - no change necessary" >>$install_parent_path"/cfautouam/cfautouam.log"
        fi
      fi
    }
    
    # End Functions
    
    # Main
    
    if [ "$1" = '-install' ]; then
      install
      exit
    elif [ "$1" = '-uninstall' ]; then
      uninstall
      exit
    elif [ "$1" = '-disable_script' ]; then
      systemctl disable cfautouam.timer
      systemctl disable cfautouam.service
      echo "$(date) - cfautouam - Script Manually Disabled" >>$install_parent_path"/cfautouam/cfautouam.log"
      disable_uam
      rm  $install_parent_path"/cfautouam/uamdisabledtime"
      rm  $install_parent_path"/cfautouam/uamenabledtime"
      exit
    elif [ "$1" = '-enable_script' ]; then
      systemctl enable $install_parent_path"/cfautouam/cfautouam.timer"
      systemctl enable $install_parent_path"/cfautouam/cfautouam.service"
      systemctl start cfautouam.timer
      echo "$(date) - cfautouam - Script Manually Enabled" >>$install_parent_path"/cfautouam/cfautouam.log"
      exit
    elif [ "$1" = '-enable_uam' ]; then
      echo "$(date) - cfautouam - UAM Manually Enabled" >>$install_parent_path"/cfautouam/cfautouam.log"
      enable_uam
      exit
    elif [ "$1" = '-disable_uam' ]; then
      echo "$(date) - cfautouam - UAM Manually Disabled" >>$install_parent_path"/cfautouam/cfautouam.log"
      disable_uam
    exit
    elif [ -z "$1" ]; then
      main
      exit
    else
      echo "cfautouam - Invalid argument"
      exit
    fi
    
     
    Last edited: Apr 11, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Thanks for sharing :)

    Just be aware of hosting files/scripts at /home, which can probably be read by local linux users if non-root permissions and they have sufficient permissions so they could read API keys etc. If only you use server that is fine but if hosting other users/sharing then /home might not be best choice. Reason why I place most of my scripting tools at /root/tools :)

    Also change it to support Cloudflare API Tokens Managing API Tokens and Keys instead of Cloudflare Global API key so you can restrict permissions for Cloudflare API Token to your specific Zone/domain and features
     
  3. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Just had a look at your Github code and use absolute cpu load as a gauge of load for DDOS attacks might not be wise as cpu load can occur for non-DDOS activities which are legit on the server i.e. backing up data or legit mysql related cpu load. You might want to just calculate the load of your web server's cpu usage and use that as indicator of DDOS attack layer 7 attacks against your web server.

    Example Nginx via pidstat which is provided by systat package
    Code (Text):
    pidstat -ulh -C nginx
    Linux 3.10.0-1062.18.1.el7.x86_64 (domain.com)         04/13/2020      _x86_64_        (8 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  Command
     1586763417  1000     16369    0.00    0.00    0.00    0.00     0  nginx: worker process                                 
     1586763417  1000     16370    0.00    0.00    0.00    0.00     1  nginx: worker process                                 
     1586763417  1000     16371    0.00    0.00    0.00    0.00     2  nginx: worker process                                 
     1586763417  1000     16372    0.00    0.00    0.00    0.00     3  nginx: worker process                                 
     1586763417  1000     16373    0.00    0.00    0.00    0.00     4  nginx: worker process                                 
     1586763417  1000     16374    0.00    0.00    0.00    0.00     5  nginx: worker process                                 
     1586763417  1000     16375    0.00    0.00    0.00    0.00     6  nginx: worker process                                 
     1586763417  1000     16376    0.00    0.00    0.00    0.00     7  nginx: worker process                                 
     1586763417     0     59743    0.00    0.00    0.00    0.00     1  nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    

    example stress test of Nginx web server running pidstat to query cummulative cpu load of 7th column of pidstat output for just nginx processes
    Code (Text):
    cpu_pc=$(($(nproc)*100))
    pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc 'NR>3 {cpu += $7} END {print cpu"%",cpu/cpumax}'
    446.1% 0.557625
    

    this reports nginx worker/master processes had total aggregrate %CPU of 446.1% which on a 8 cpu thread system would have cpu_pc value of 8x100 = 800 so 446.1/800 = 0.557625 cpu load where optimal is no more than 1.0

    another example
    Code (Text):
    cpu_pc=$(($(nproc)*100))
    pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {print cpu"%",cpu/cpumax}'
    510.92% 0.63865
    

    cpu load average normalised is 0.63865 stressing nginx - out of 1.0 optimal

    whole integer calculation for comparison to optimal 1.0x100 = 100
    Code (Text):
    cpu_pc=$(($(nproc)*100))
    pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {printf "%.0f\n", (cpu/cpumax)*100}'
    64
    

    so cpu load would be high if >100%. So 64% is less than 100% so normal and shouldn't trigger Cloudflare Under Attack Mode

    You can also modify pidstats to record over 5 second interval so Nginx calculated cpu load is the load at end of 5 second interval
    Code (Text):
    cpu_pc=$(($(nproc)*100))
    nginx_workers=$(ps -afC nginx | grep -v grep | grep -c 'nginx: worker')
    
    pidstat -ulh -C nginx 1 5 | grep -A${nginx_workers} 'UID' | tail -${nginx_workers} | egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {printf "%.0f\n", (cpu/cpumax)*100}'
    63
    
     
  4. Rake-GH

    Rake-GH Active Member

    179
    93
    28
    Jul 29, 2019
    USA
    Ratings:
    +144
    Local Time:
    6:38 PM
    default
    default
    Thanks for the advice, I will try to work your tips into the next update
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    For Nginx and PHP-FPM stacks though using a more detailed cpu load detection for both Nginx and PHP-FPM processes might be needed as DDOS attacks at layer 7 application level will drive up both Nginx and PHP-FPM related cpu usage.

    For PHP-FPM under load
    Code (Text):
    cpu_pc=$(($(nproc)*100))
    pidstat -ulh -C php-fpm 1 1
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  Command
     1586832886  1000     30831    0.00    1.00    0.00    1.00     0  php-fpm: pool www
     1586832886  1000     30832    1.00    0.00    0.00    1.00     3  php-fpm: pool www
     1586832886  1000     30833    0.00    1.00    0.00    1.00     1  php-fpm: pool www
     1586832886  1000     30834    0.00    1.00    0.00    1.00     2  php-fpm: pool www
     1586832886  1000     30835    1.00    1.00    0.00    2.00     0  php-fpm: pool www
     1586832886  1000     30836    1.00    1.00    0.00    2.00     2  php-fpm: pool www
     1586832886  1000     30838    1.00    1.00    0.00    2.00     2  php-fpm: pool www
     1586832886  1000     30841    1.00    0.00    0.00    1.00     1  php-fpm: pool www
     1586832886  1000     30842    1.00    0.00    0.00    1.00     2  php-fpm: pool www
     1586832886  1000     30846    1.00    1.00    0.00    2.00     0  php-fpm: pool www
     1586832886  1000     30847    1.00    0.00    0.00    1.00     1  php-fpm: pool www
     1586832886  1000     30848    1.00    0.00    0.00    1.00     3  php-fpm: pool www
     1586832886  1000     30849    1.00    0.00    0.00    1.00     1  php-fpm: pool www
     1586832886  1000     30850    1.00    0.00    0.00    1.00     2  php-fpm: pool www

    Code (Text):
    cpu_pc=$(($(nproc)*100))
    pidstat -ulh -C php-fpm 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {print cpu"%",cpu/cpumax}'
    32.98% 0.041225
    
    pidstat -ulh -C php-fpm 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {printf "%.0f\n", (cpu/cpumax)*100}'
    5
     
  6. STN

    STN CentMaxMod

    9
    4
    3
    Jun 9, 2020
    Ratings:
    +7
    Local Time:
    4:38 AM
    1.19.0
    Mariadb 10.3
    You can use page rules to always have cloudflare under attack mode on certain "troublesome" areas of site. I have cf under attack on login, register and search pages which are what majority of bots target and my server can breathe again.

    It's a stock cpanel install without any optimisation because that was the limit of my knowledge at the time :LOL: and it still feels fast. Without those rules, there's a notable difference in cpu/mysql use.