Nginx Throttle Offline

Jimmy · Aug 2, 2020

Sometimes my site gets attacked and to take some of the pressure off I just do a "sitestatus offline" and it serves the html page which is an easy handle for the server.

Would it be possible or make sense to have the "sitestatus offline" trigger at a certain CPU threshold set in the conf file?

1. Set CPU TOP threshold in config file. CPU percent when script is triggered.
2. Site / server gets attacked or just flooded with requests it can't handle.
3. CPU hits a CPU TOP threshold from the config.
4. "sitestatus offline" is activated.
5. Set CPU LOWER threshold (in #1 but here for the list) in the config.
6. CPU hits the LOWER threshold.
7. "sitestatus online" is activated.

The CPU TOP is where the script is triggerd. CPU LOWER allows the owner to decide at which point the sitestatus online is triggered.

The script could keep going off and on for as long as needed. Would be great for smaller sites which can't afford paid plans. A down and dirty simple technique. Doesn't solve the overall problem but could save someone like me from getting throttled by SSDnodes. I'm sure other people are having the same thing happen when the CPU pegs.

Plus when a background task runs for backup it pegs the processor on my throttled SSDNodes machine. During those times, putting the server in offline makes the process go faster. Meaning, the server sites are offline for a little bit vs. slow for a long time.

Just an idea.

eva2000 · Aug 2, 2020

Jimmy said: ↑

Sometimes my site gets attacked and to take some of the pressure off I just do a "sitestatus offline" and it serves the html page which is an easy handle for the server.

Would it be possible or make sense to have the "sitestatus offline" trigger at a certain CPU threshold set in the conf file?
Click to expand...

Interesting, never really intended for Centmin Mod 123.09beta01 or higher's sitestatus maintenance mode (Beta Branch - sitestatus maintenance mode) to be used that way. But guess it can work nicely in such cases

@Rake-GH wrote a Cloudflare I am underattack script which is based on cpu load but I posted some comments as to why using system cpu load might not be accurate at cfautouam - CloudFlare Under Attack Mode Automation as cpu load system wide can be triggered by non-http requests i.e. backup scripts etc. So I commented that you can base cpu load only on nginx and/or php-fpm based cpu load via pidstat command
eva2000 said: ↑
Just had a look at your Github code and use absolute cpu load as a gauge of load for DDOS attacks might not be wise as cpu load can occur for non-DDOS activities which are legit on the server i.e. backing up data or legit mysql related cpu load. You might want to just calculate the load of your web server's cpu usage and use that as indicator of DDOS attack layer 7 attacks against your web server.

Example Nginx via pidstat which is provided by systat package
Code (Text):
pidstat -ulh -C nginx
Linux 3.10.0-1062.18.1.el7.x86_64 (domain.com)         04/13/2020      _x86_64_        (8 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  Command
 1586763417  1000     16369    0.00    0.00    0.00    0.00     0  nginx: worker process                              
 1586763417  1000     16370    0.00    0.00    0.00    0.00     1  nginx: worker process                              
 1586763417  1000     16371    0.00    0.00    0.00    0.00     2  nginx: worker process                              
 1586763417  1000     16372    0.00    0.00    0.00    0.00     3  nginx: worker process                              
 1586763417  1000     16373    0.00    0.00    0.00    0.00     4  nginx: worker process                              
 1586763417  1000     16374    0.00    0.00    0.00    0.00     5  nginx: worker process                              
 1586763417  1000     16375    0.00    0.00    0.00    0.00     6  nginx: worker process                              
 1586763417  1000     16376    0.00    0.00    0.00    0.00     7  nginx: worker process                              
 1586763417     0     59743    0.00    0.00    0.00    0.00     1  nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
example stress test of Nginx web server running pidstat to query cummulative cpu load of 7th column of pidstat output for just nginx processes
Code (Text):
cpu_pc=$(($(nproc)*100))
pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc 'NR>3 {cpu += $7} END {print cpu"%",cpu/cpumax}'
446.1% 0.557625
this reports nginx worker/master processes had total aggregrate %CPU of 446.1% which on a 8 cpu thread system would have cpu_pc value of 8x100 = 800 so 446.1/800 = 0.557625 cpu load where optimal is no more than 1.0

another example
Code (Text):
cpu_pc=$(($(nproc)*100))
pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {print cpu"%",cpu/cpumax}'
510.92% 0.63865
cpu load average normalised is 0.63865 stressing nginx - out of 1.0 optimal

whole integer calculation for comparison to optimal 1.0x100 = 100
Code (Text):
cpu_pc=$(($(nproc)*100))
pidstat -ulh -C nginx 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {printf "%.0f\n", (cpu/cpumax)*100}'
64
so cpu load would be high if >100%. So 64% is less than 100% so normal and shouldn't trigger Cloudflare Under Attack Mode

You can also modify pidstats to record over 5 second interval so Nginx calculated cpu load is the load at end of 5 second interval
Code (Text):
cpu_pc=$(($(nproc)*100))
nginx_workers=$(ps -afC nginx | grep -v grep | grep -c 'nginx: worker')

pidstat -ulh -C nginx 1 5 | grep -A${nginx_workers} 'UID' | tail -${nginx_workers} | egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {printf "%.0f\n", (cpu/cpumax)*100}'
63
Click to expand...

eva2000 · Aug 2, 2020

Jimmy said: ↑

Plus when a background task runs for backup it pegs the processor on my throttled SSDNodes machine. During those times, putting the server in offline makes the process go faster. Meaning, the server sites are offline for a little bit vs. slow for a long time.
Click to expand...

Guess that would be one case for system cpu load heh but then you can do the same for pidstat on mysql load like I outlined for nginx and php-fpm pidstat derived cpu loads.

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Nginx Throttle Offline

Jimmy Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Nginx Throttle Offline

Jimmy Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.