Get the most out of your Centmin Mod LEMP stack
Become a Member

Cloudflare Firewall > Rate Limiting: Optimal settings?

Discussion in 'Domains, DNS, Email & SSL Certificates' started by rdan, May 11, 2018.

  1. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    Considering a website gets average 500 real time visitor.
    Most of the users are on Shared/Dynamic IP address.
     
  2. eva2000

    eva2000 Administrator Staff Member

    34,632
    7,655
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,773
    Local Time:
    2:34 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    It would vary depending on how your site is laid out too i.e. number of images, css, js being called per page too.

    You'd have to inspect site's access.log in /home/nginx/domains/yourdomain.com/log/access.log and maybe rotated gzip compressed logs too as outlined in Config page Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS and maybe use zcat instead of cat or Centmin Mod 123.09beta01 also has pzcat if you have more than 2 cpu cores which uses multi-threaded pigz instead of single threaded gzip (zcat) for inspecting compressed access logs Sysadmin - zcat compressed access log processing benchmarks

    To use pzcat on both uncompressed and compressed logs, need to use -f force flag with pzcat i.e.
    Code (Text):
    cd /home/nginx/domains/yourdomain.com/log/
    pzcat -f access.log access.log-20180428.gz
    

    which will output contents of both uncompressed access.log and compressed log and then pipe them through and use awk,grep,fgrep and other shell commands to manipulate the date i.e. filter IP addresses and date to calculate the rate of requests/s.

    That's what I do to calculate Nginx's request rate for a point in time at Forum DDOS Attacked - Linode null routed

    For example to inspect main hostname's nginx log which is at separate location at /var/log/nginx there are both uncompressed current localhost.access.log and compressed logs with suffix -201805xx.gz date timestamps
    Code (Text):
    ls -lah /var/log/nginx
    total 728K
    drwxr-xr-x 2 root  root 4.0K May 11 03:10 .
    drwxr-xr-x 8 root  root 4.0K Apr 11 04:59 ..
    -rw-rw---- 1 nginx root  22K May 11 12:16 localhost.access.log
    -rw-rw---- 1 root  root  20K May  4 03:06 localhost.access.log-20180504.gz
    -rw-rw---- 1 nginx root  15K May  5 03:08 localhost.access.log-20180505.gz
    -rw-rw---- 1 nginx root  18K May  6 03:27 localhost.access.log-20180506.gz
    -rw-rw---- 1 nginx root  40K May  7 03:28 localhost.access.log-20180507.gz
    -rw-rw---- 1 nginx root  37K May  8 03:49 localhost.access.log-20180508.gz
    -rw-rw---- 1 nginx root  23K May  9 03:35 localhost.access.log-20180509.gz
    -rw-rw---- 1 nginx root  15K May 10 03:29 localhost.access.log-20180510.gz
    -rw-rw---- 1 nginx root  88K May 11 03:10 localhost.access.log-20180511
    -rw-rw---- 1 nginx root  45K May 11 12:15 localhost.error.log
    -rw-rw---- 1 root  root  25K May  4 03:05 localhost.error.log-20180504.gz
    -rw-rw---- 1 nginx root  19K May  5 03:07 localhost.error.log-20180505.gz
    -rw-rw---- 1 nginx root  22K May  6 03:26 localhost.error.log-20180506.gz
    -rw-rw---- 1 nginx root  50K May  7 03:27 localhost.error.log-20180507.gz
    -rw-rw---- 1 nginx root  46K May  8 03:49 localhost.error.log-20180508.gz
    -rw-rw---- 1 nginx root  28K May  9 03:34 localhost.error.log-20180509.gz
    -rw-rw---- 1 nginx root  17K May 10 03:28 localhost.error.log-20180510.gz
    -rw-rw---- 1 nginx root 141K May 11 03:07 localhost.error.log-20180511
    

    so to inspect it for a particular IP address i.e. 174.129.1.66

    just current non-rotated access log reload nginx to flush access log buffer to disk and cd into log directory and use cat command piped to a grep filter for IP entry
    Code (Text):
    ngxreload
    cd /var/log/nginx
    cat localhost.access.log | grep '174.129.1.66'
    

    example output has 3 entries, you may have 1000s
    Code (Text):
    ngxreload
    cd /var/log/nginx
    cat localhost.access.log | grep '174.129.1.66'
    174.129.1.66 - - [11/May/2018:03:20:44 +0000] "HEAD /haircut-places-in-san-antonio HTTP/1.1" 404 0 "-" "MBCrawler/1.0 (https://monitorbacklinks.com)"
    174.129.1.66 - - [11/May/2018:03:20:44 +0000] "GET /haircut-places-in-san-antonio HTTP/1.1" 404 162 "-" "MBCrawler/1.0 (https://monitorbacklinks.com)"
    174.129.1.66 - - [11/May/2018:11:52:21 +0000] "HEAD /haircut-places-in-san-antonio HTTP/1.1" 404 0 "-" "MBCrawler/1.0 (https://monitorbacklinks.com)"
    174.129.1.66 - - [11/May/2018:11:52:21 +0000] "GET /haircut-places-in-san-antonio HTTP/1.1" 404 162 "-" "MBCrawler/1.0 (https://monitorbacklinks.com)"
    

    to count number in just current log pipe previous command through wc -l to count number of entries matching that grep filtered IP address
    Code (Text):
    ngxreload
    cd /var/log/nginx
    cat localhost.access.log | grep '174.129.1.66' | wc -l
    4
    

    Now if you need to look through all rotated gzip compressed logs and current uncompressed logs use pzcat if you have 2 or more cpu cores or use zcat if you have 1 cpu core
    Code (Text):
    ngxreload
    cd /var/log/nginx
    pzcat -f localhost.access.log localhost.access.log-* | grep '174.129.1.66' | wc -l
    40
    

    which shows there's 40 matching entries from all rotated gzip compressed logs and current uncompressed logs dating back to May 4th log rotation time for localhost.access.log-20180504.gz which has a start date of May 3 at 8:31 AM
    Code (Text):
    ngxreload
    cd /var/log/nginx
    pzcat localhost.access.log-20180504.gz | head -n1
    93.158.161.176 - - [03/May/2018:08:31:23 +0000] "GET /should-i-grow-my-hair-out-men/phenomenalhaircare-sisterlocks-showing-length-in-bundles-and-rods HTTP/1.1" 404 162 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
    

    Convert that May 3 time into linux epoch time change forward slashs / to dashes - and putting space between year and time
    Code (Text):
    date -d "03-May-2018 08:31:23" +%s
    1525336283
    

    current date time in epoch time
    Code (Text):
    date +%s
    1526043594
    

    Time period is over 1526043594 - 1525336283 = 707,311 seconds
    Code (Text):
    echo $((1526043594 - 1525336283))
    707311
    

    Or calculate number of 404 entries requested for url /haircut-places-in-san-antonio via grep filter - shows 84 entries
    Code (Text):
    ngxreload
    cd /var/log/nginx
    pzcat -f localhost.access.log localhost.access.log-* | grep '/haircut-places-in-san-antonio' | wc -l      
    84
    

    Then you can go further to filter using date command and some bash calculation/maths for just a point in time period and work out number of hits for that filtered IP over that time interval to work out the request rate specifically.

    Probably other tools you can use too. Google is your friend :)
     
    • Informative Informative x 1
  3. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    That seems to much work :D.
    I just wait for 24 hours stats for "Requests Through Cloudflare".

    [​IMG]

    Then divide it to have an average.
     
    • Like Like x 1
  4. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    So from the stats last 6 hours.
    6 hours = 360 minutes.
    2,277,000 / 360 = 6325 request per minute.
     
  5. eva2000

    eva2000 Administrator Staff Member

    34,632
    7,655
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,773
    Local Time:
    2:34 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    haha power of GUI and cloudflare :D
     
  6. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    So with my 1 month stats:
    upload_2018-6-11_19-54-45.png

    I would be paying $260 USD if I have it enabled the entire month :eek:.
     
  7. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    52,139,310 - 10,000 (free) = 52,129,310
    52,129,310 / 10,000 = 5212.931
    5212.931 x 0.05 = $ 260.64655
     
  8. eva2000

    eva2000 Administrator Staff Member

    34,632
    7,655
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,773
    Local Time:
    2:34 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Wrong calculation rate limiting is per request processed through rate limiting rules iirc Billing for Cloudflare Rate Limiting

     
  9. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
    Yes, and my Rate Limit rule cover my entire site previously (www.example.com/*).
    Rate Limiting also doesn't count request that is HIT/Cached on Edge (only count Uncached Request).
     
  10. eva2000

    eva2000 Administrator Staff Member

    34,632
    7,655
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,773
    Local Time:
    2:34 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    interesting no where in their doc/pricing info do they mention that part. Cloudflare support confirmed this ?
     
  11. rdan

    rdan Premium Member Premium Member

    4,197
    1,015
    113
    May 25, 2014
    Ratings:
    +1,439
    Local Time:
    12:34 AM
    Mainline
    10.2
  12. eva2000

    eva2000 Administrator Staff Member

    34,632
    7,655
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,773
    Local Time:
    2:34 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah liked it until saw the pricing :D
     
..