Join the community today
Become a Member

Memcached Memcrashed - Major amplification attacks from UDP port 11211

Discussion in 'Other Centmin Mod Installed software' started by pamamolf, Feb 28, 2018.

  1. pamamolf

    pamamolf Well-Known Member

    3,112
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    6:27 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

    In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

    The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

    Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

    Memcrashed
    Obscure amplification attacks happen all the time. We often see "chargen" or "call of duty" packets hitting our servers.

    A discovery of a new amplification vector though, allowing very great amplification, happens rarely. This new memcached UDP DDoS is definitely in this category.

    The DDosMon from Qihoo 360 monitors amplification attack vectors and this chart shows recent memcached/11211 attacks:



    More info here:

    Memcrashed - Major amplification attacks from UDP port 11211
     
    • Like Like x 1
  2. pamamolf

    pamamolf Well-Known Member

    3,112
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    6:27 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    New version is out that fixes that issue:

    Memcached 1.5.6 Release Notes:

    Overview
    This is a bugfix release, but it primarily disables the UDP protocol by default.

    In the last few days reports of UDP amplification attacks utilizing inesure memcached instances have surfaced. Attackers are able to set large values into memcached, then send requests via spoofed UDP packets. Memcached will then send a very large number of very large UDP packets back in response.

    12 years ago, the UDP version of the protocol had more widespread use: TCP overhead could be very high. In the last few years, I've not heard of anyone using UDP anymore. Proxies and special clients allow connection reuse, which lowers the overhead. Also, RAM values are so large that TCP buffers just don't add up as much as they used to.

    That said, I don't have any way of knowing how many UDP installations there are. Everyone who uses UDP and upgrades past this version, will find the UDP protocol disabled unless they explicitly enable it via -U 11211. Hopefully this one-time pain is acceptable.

    Thanks for everyone who reached out in the last couple days to help understand the problem and coordinate patches sent to linux and BSD distro's.

    Fixes:
    • disable UDP port by default
    • systemd instancing support & rpm build improvements
    • fix gcc warnings in beta GCC
    • fix build with clang
    • fix for dtrace compilation on freebsd
     
  3. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Thanks for heads up. FYI, by default Centmin Mod 123.09beta01's CSF Firewall blocks port 11211 out of the box unless you whitelist it yourself. Also Memcached is set to listent to localhost/127.0.0.1 so won't response to public requests anyway.

    Centmin Mod users can update Memcached as per instructions at Memcached - Memcached Server 1.5.6 Update Available
     
    Last edited: Feb 28, 2018
    • Like Like x 1
  4. bassie

    bassie Active Member

    897
    213
    43
    Apr 29, 2016
    Ratings:
    +633
    Local Time:
    5:27 PM
    Wonder if Red hat is gonna patch Memcached EL6 and 7
    upstream as both are vulnerable if you call it that way as the UDP protocol is enabled by default.
     
  5. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah you'd hope so. I haven't used memcached provided by redhat/centos yum repos in years. I also build or compile my own memcached binaries :)
     
  6. Revenge

    Revenge Active Member

    404
    84
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +300
    Local Time:
    4:27 PM
    1.9.x
    10.1.x
    Issue in Vpsdime Dallas location because of this:

     
    • Informative Informative x 1
  7. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Whoops.. this is going to bite alot of folks who don't actually use a firewall on their server ! Luckily, Centmin Mod out of box uses CSF Firewall :)
     
    • Like Like x 2
    • Winner Winner x 1
  8. Revenge

    Revenge Active Member

    404
    84
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +300
    Local Time:
    4:27 PM
    1.9.x
    10.1.x
    Yes, a firewall or a proper iptables configuration.

    Meanwhile, Vpsdime updated their report and closed 11211 port for outbound connections.

     
    Last edited: Mar 1, 2018
    • Informative Informative x 1
  9. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    I'm working on a server now, and has Memcached that was listening on 127.0.0.1 TCP and UDP, and when I nmap the server, it's showing OPEN/FILTERED, even though port 11211 is blocked by CSF.

    Code:
    nmap TARGET -p 11211 -sU -sS --script memcached-info
    
    Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-01 19:24 UTC
    Nmap scan report for TARGET
    Host is up (0.12s latency).
    PORT      STATE         SERVICE
    11211/tcp filtered      unknown
    11211/udp open|filtered unknown
    
    Nmap done: 1 IP address (1 host up) scanned in 2.74 seconds
    
    When I try my own:
    Code:
    nmap TARGET -p 11211 -sU -sS --script memcached-info 
    
    Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-01 19:17 UTC
    Nmap scan report for TARGET
    Host is up (0.018s latency).
    PORT      STATE  SERVICE
    11211/tcp closed unknown
    11211/udp closed unknown
    
    Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds
    
    Exactly same Memcached version and config. Only difference is mine are CentOS7, and the one showing open is CentOS6.9

    Tried another CentOS6.9 server, and it's reporting OPEN/FILTERED again, even though it's blocked by CSF and only listening on 127.0.0.1
     
  10. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    Just seen the updated to 1.5.6 as well (which these servers are all running)

    # rpm -qa | grep memcached
    memcached-1.5.6-1.el6.remi.x86_64
     
  11. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah apparently Github was hit with this February 28th DDoS Incident Report 1.35Tbps attack ! :eek:
     
    • Optimistic Optimistic x 1
  12. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    that's strange these are all fresh Centmin Mod installs ? you double checked /etc/csf/csf.conf doesn't open port 11211 ?
     
  13. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    No, these aren't centmin. However, csf only has 3 udp ports open, and even with memcached updated to 1.5.6 and not even posting a listen down to udp 11211 they still report the same. Always CentOS6
     
  14. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    so to clarify nmap reports 11211 open even with memcached 1.5.6 updated and ports blocked from public on both centmin mod installs and non-centmin mod installs that use centos 6.x ? or just non-centmin mod installs with centos 6.x ?
     
  15. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    I'm just updating a CentOS6.9 Centminmod install now with 1.5.6 and will report back shortly.
     
  16. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    1.5.6 Remi install on 6.9
    Code:
    # netstat -nlp | grep 11211             
    tcp        0      0 127.0.0.1:11211             0.0.0.0:*                   LISTEN      2531/memcached     
    
    Code:
    PORT      STATE         SERVICE
    11211/tcp filtered      unknown
    11211/udp open|filtered unknown
    
    1.5.6 on Centminmod 6.9
    Code:
    # netstat -nlp | grep 11211   
    tcp        0      0 127.0.0.1:11211             0.0.0.0:*                   LISTEN      31241/memcached 
    
    Code:
    Host is up (0.088s latency).
    PORT      STATE    SERVICE
    11211/tcp closed   unknown
    11211/udp filtered unknown
    
    1.5.3 remi on CentOS7
    Code:
    # netstat -nlp | grep 11211 
    tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      1102/memcached     
    udp        0      0 127.0.0.1:11211         0.0.0.0:*                           1102/memcached     
    
    Code:
    Host is up (0.018s latency).
    PORT      STATE  SERVICE
    11211/tcp closed unknown
    11211/udp closed unknown
    

    CSF on ALL the servers
    Code:
    # Allow incoming UDP ports
    UDP_IN = "20,21,53"
    
     
    • Informative Informative x 1
  17. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    I checked on my Centmin Mod CentOS 6.9 installs and even Memcache 1.5.2-1.5.4 are reported closed as open|filtered in nmap due to CSF Firewall.

    edit: wait might have read that incorrectly - state = open|filtered versus closed

    Port Scanning Basics | Nmap Network Scanning

     
  18. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    I've ran a TCPDUMP on 11211 UDP when doing an nmap, and it's not getting to the packet capture, so it's being dropped / not responded to (as expected).

    Code:
    open|filtered
    
    Nmap places ports in this state when it is unable to determine whether 
    a port is open or filtered. This occurs for scan types in which open
    ports give no response. The lack of response could also mean that a
    packet filter dropped the probe or any response it elicited. So Nmap
    does not know for sure whether the port is open or being filtered. The
    UDP, IP protocol, FIN, NULL, and Xmas scans classify ports this way.
    
     
    • Like Like x 1
    • Agree Agree x 1
  19. eva2000

    eva2000 Administrator Staff Member

    35,559
    7,841
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,085
    Local Time:
    1:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    came to same conclusion open|filtered for our purposes just means = filtered or closed = ok :)
     
    • Like Like x 2
  20. Matt

    Matt Moderator Staff Member

    760
    342
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +508
    Local Time:
    4:27 PM
    1.7.1
    MariaDB 10
    Had 2 customers with HiVelocity receive the same mail from them today

    Both of these servers have CSF installed and active with only the 3 default ports allowed for UDP.
     
    • Informative Informative x 2
..