Learn about Centmin Mod LEMP Stack today
Become a Member

Security to access this site

Discussion in 'Forum Feedback & Suggestions' started by Itworx4me, Aug 15, 2020.

  1. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
    Lately I have been getting hit with security questions, delays to this site from cloudflare. Today I got this:
    centmin.PNG

    Is this how its going to be when visiting this site?

    Thanks,
    Itworx4me
     
  2. Jay Chen

    Jay Chen Active Member

    134
    37
    28
    Sep 10, 2017
    Ratings:
    +72
    Local Time:
    8:24 AM
    Didn't happen to me, not even once.
     
  3. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    From your current IP address, I only see 2 Cloudflare Firewall events for Cloudflare Bot Management firewall rules I setup related to forum searches.
    1. Did this occur on a forum search or new post search/click ?
    2. Did you hover over the forum's new post/recent post links without clicking them ?
    3. Did you experience the challenge as a logged in forum member or logged out guest ?
    4. Does your web browser have anything enabled for javascript or cookie blocking used ? Or using a VPN, HTTP Proxy or TOR client ?
    Cloudflare Bot Management firewall rule is seeing you as a bot and not human for some reason.
     
    Last edited: Aug 15, 2020
  4. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
    I will have to pay more attention next time but I believe it was when I clicked on new post search.
     
  5. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Ok. I've also logged a ticket with Cloudflare to see if Bot Management Cloudflare Bot Management: machine learning and more is incorrectly classifying you as a bot and not human :)
     
  6. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
  7. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yeah it's the same Cloudflare Bot Management firewall rule being triggered, waiting on CF Bot Management team to get back to me as to why it's seeing you as a bot and not a human :)
     
  8. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    I wrote a script to parse my Cloudflare logpush logs for CF edge server processed requests and filtered on domain, @Itworx4me's IP address xxx.xxx.xxx.xxx, for Bot Management bot score <=10 (closer to 100 = human and closer 0 = bot) and filtered for CF firewall challenge requests only and for /find-new/posts urls only. You can see all Cloudflare's logpush log fields explained at https://developers.cloudflare.com/logs/log-fields

    Results show 5 firewall challenge events with bot score <=10
    Code (Text):
    ./cflog-parser.sh parse community.centminmod.com/find-new/posts xxx.xxx.xxx.xxx 20200815 10 challenge
    h=community.centminmod.com
    ip=xxx.xxx.xxx.xxx
    datedir=20200815
    botscore=10
    firewall=challenge
    path=/find-new/posts
    /usr/bin/pzcat /home/cfcmm-logs/20200815/*.log.gz | jq -r --arg i $ip --arg host $h --arg bs $botscore --arg c $fwmatch --arg reqpath $path 'select(.BotScore <=($bs | tonumber) and .ClientRequestHost == $host and .ClientIP == $i and .FirewallMatchesActions[] == $c and .ClientRequestPath == $reqpath) | "\(.EdgeStartTimestamp) \(.ClientIP) \(.RayID) \(.ParentRayID) \(.ClientRequestURI) \(.ClientRequestMethod) \(.ClientRequestReferer) \(.EdgeResponseStatus) \(.OriginResponseStatus) \(.EdgeRequestHost) \(.EdgeColoCode) \(.ClientCountry) \(.ClientIPClass) [\(.WorkerStatus)-\(.WorkerSubrequest)-\(.WorkerSubrequestCount)] \(.EdgePathingOp)-\(.EdgePathingSrc)-\(.EdgePathingStatus)-\(.EdgeRateLimitAction) \(.FirewallMatchesActions):\(.FirewallMatchesRuleIDs):\(.FirewallMatchesSources) \(.WAFAction)-\(.WAFRuleID) \(.BotScore) x \(.BotScoreSrc) \(.ClientRequestUserAgent)"' | egrep -i -v 'cdn-cgi|index.rss|xidel|UptimeRobot|HetrixTools|CloudFlare-Prefetch'
    
    2020-08-15T19:46:47Z xxx.xxx.xxx.xxx 5c356c768a39f5a5 00 /find-new/posts GET https://community.centminmod.com/ 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 4 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-15T19:46:35Z xxx.xxx.xxx.xxx 5c356c2a6d87f5a5 00 /find-new/posts GET https://community.centminmod.com/ 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 4 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-15T21:11:42Z xxx.xxx.xxx.xxx 5c35e8dd69801476 00 /find-new/posts GET https://community.centminmod.com/threads/security-to-access-this-site.20197/ 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 4 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-15T21:12:00Z xxx.xxx.xxx.xxx 5c35e94aeeee1476 00 /find-new/posts GET https://community.centminmod.com/threads/security-to-access-this-site.20197/ 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 2 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-15T23:55:03Z xxx.xxx.xxx.xxx 5c36d8243bc5e38a 00 /find-new/posts GET https://community.centminmod.com/ 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 4 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    

    Reported bot scores are either 2 or 4 derived from Cloudflare's machine learning source
    Code (Text):
    4 x Machine Learning
    2 x Machine Learning
    

    Still waiting on Cloudflare Bot Management team to investigate why it's Bot Management machine learning is picking up a legit user as a bot with such a low bot score.
     
  9. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
  10. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Cheers. You're hitting 2 Cloudflare Bot Management Firewall rules I have for search and find-new paths setup unfortunately right now. These rules have been in place for months now, so whatever is happening right now is different from usual.

    Still waiting on Bot Management team to investigate as to why your IP visits to these Firewall rules are detected with such a low bot score <10 but other requests to other paths are show much higher 'human' bot scores

    You triggered a Cloudflare Bot Management Firewall rule for search with your detected bot score of 8 which is higher than your previous 2-4 so you get met with a js challenge instead of a captcha challenge this time.
    You triggered a rule for find-new with triggered a captcha challenge.

    My custom Cloudflare Edge server log parser shows for your IP, the Bot Management bot score for all visits to just this forum range from a low of 8 (non-human) to average 58.71 with max considered to be human of 97 and 99% percentile of 96.
    Code (Text):
    ./cflog-parser.sh botrange community.centminmod.com xxx.xxx.xxx.xxx 20200817 100 none
    min: 8.00 avg: 47.63 max: 96.00 95%: 96.00 99%: 96.00
    

    Then to filter for actual Cloudflare Edge server log entries for your IP for August 17th where bot score is <=10
    Code (Text):
    ./cflog-parser.sh parse community.centminmod.com xxx.xxx.xxx.xxx 20200817 10 all
    h=community.centminmod.com
    ip=xxx.xxx.xxx.xxx
    datedir=20200817
    botscore=10
    firewall=all
    path=
    ua=
    /usr/bin/pzcat /home/cfcmm-logs/20200817/*.log.gz | jq -r --arg i $ip --arg host $h --arg bs $botscore 'select(.BotScore <=($bs | tonumber) and .ClientRequestHost == $host and .ClientIP == $i and .FirewallMatchesActions[] != null) | "\(.EdgeStartTimestamp) \(.ClientIP) \(.RayID) \(.ParentRayID) \(.ClientRequestURI) \(.ClientRequestMethod) \(.ClientRequestReferer) \(.EdgeResponseStatus) \(.OriginResponseStatus) \(.EdgeRequestHost) \(.EdgeColoCode) \(.ClientCountry) \(.ClientIPClass) [\(.WorkerStatus)-\(.WorkerSubrequest)-\(.WorkerSubrequestCount)] \(.EdgePathingOp)-\(.EdgePathingSrc)-\(.EdgePathingStatus)-\(.EdgeRateLimitAction) \(.FirewallMatchesActions):\(.FirewallMatchesRuleIDs):\(.FirewallMatchesSources) \(.WAFAction)-\(.WAFRuleID) \(.BotScore) x \(.BotScoreSrc) \(.ClientRequestUserAgent)"' | egrep -i -v 'cdn-cgi|index.rss|xidel|UptimeRobot|HetrixTools|CloudFlare-Prefetch'
    
    2020-08-17T23:11:00Z xxx.xxx.xxx.xxx 5c471259389c0921 00 /find-new/posts GET https://community.centminmod.com/ 503 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-jschlNew- ["jschallenge"]:["RULEID"]:["firewallRules"] unknown- 8 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-17T23:12:42Z xxx.xxx.xxx.xxx 5c4714da8bc60921 00 /search/member?user_id=1247&content=post GET https://community.centminmod.com/find-new/11152900/posts 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 8 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    2020-08-17T23:12:57Z xxx.xxx.xxx.xxx 5c471538bd7c0921 00 /search/member?user_id=1247&content=post GET https://community.centminmod.com/find-new/11152904/posts 403 0  SEA us noRecord [unknown-false-0] chl-filterBasedFirewall-captchaNew- ["challenge"]:["RULEID"]:["firewallRules"] unknown- 9 x Machine Learning Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    

    You can see the js challenge rule detected you with bot score of 8
    Code (Text):
    8 x Machine Learning

    While captcha challenge, 1st time was bot score of 8 and 2nd time was 9
    Code (Text):
    8 x Machine Learning
    9 x Machine Learning
    

    All derived from Machine Learning
     
  11. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
    Its very weird that this all started this past week. Never had an issue until now. :(
     
  12. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yeah just got reply from CF Bot Management team
    fingers crossed :)
     
  13. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    @Itworx4me Cloudflare has rolled out the new Bot Management machine learning model to 50% of my traffic as a test and I made sure that includes your IP/region. So check if you still get hit my challenge page requests now
     
  14. Itworx4me

    Itworx4me Member

    229
    22
    18
    Mar 14, 2017
    Ratings:
    +39
    Local Time:
    5:24 AM
    Nginx 1.17.X
    MariaDB 10.3.X
    Seems to be back to normal. Thanks George for all your help.

    Itworx4me
     
  15. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:24 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Great to hear :)