/ Register

Want more timely Centmin Mod News Updates?
Become a Member

Memcached Memcrashed - Major amplification attacks from UDP port 11211

Discussion in 'Other Centmin Mod Installed software' started by pamamolf, Feb 28, 2018.

Previous Thread Next Thread
Loading...
Page 1 of 2
  1. Feb 28, 2018 #1
    pamamolf

    pamamolf Premium Member Premium Member

    4,087
    428
    83
    May 31, 2014
    Ratings:
    +834
    Local Time:
    5:14 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

    In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

    The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

    Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

    Memcrashed
    Obscure amplification attacks happen all the time. We often see "chargen" or "call of duty" packets hitting our servers.

    A discovery of a new amplification vector though, allowing very great amplification, happens rarely. This new memcached UDP DDoS is definitely in this category.

    The DDosMon from Qihoo 360 monitors amplification attack vectors and this chart shows recent memcached/11211 attacks:



    More info here:

    Memcrashed - Major amplification attacks from UDP port 11211
     
  2. Feb 28, 2018 #2
    pamamolf

    pamamolf Premium Member Premium Member

    4,087
    428
    83
    May 31, 2014
    Ratings:
    +834
    Local Time:
    5:14 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    New version is out that fixes that issue:

    Memcached 1.5.6 Release Notes:

    Overview
    This is a bugfix release, but it primarily disables the UDP protocol by default.

    In the last few days reports of UDP amplification attacks utilizing inesure memcached instances have surfaced. Attackers are able to set large values into memcached, then send requests via spoofed UDP packets. Memcached will then send a very large number of very large UDP packets back in response.

    12 years ago, the UDP version of the protocol had more widespread use: TCP overhead could be very high. In the last few years, I've not heard of anyone using UDP anymore. Proxies and special clients allow connection reuse, which lowers the overhead. Also, RAM values are so large that TCP buffers just don't add up as much as they used to.

    That said, I don't have any way of knowing how many UDP installations there are. Everyone who uses UDP and upgrades past this version, will find the UDP protocol disabled unless they explicitly enable it via -U 11211. Hopefully this one-time pain is acceptable.

    Thanks for everyone who reached out in the last couple days to help understand the problem and coordinate patches sent to linux and BSD distro's.

    Fixes:
    • disable UDP port by default
    • systemd instancing support & rpm build improvements
    • fix gcc warnings in beta GCC
    • fix build with clang
    • fix for dtrace compilation on freebsd
     
  3. Feb 28, 2018 #3
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Thanks for heads up. FYI, by default Centmin Mod 123.09beta01's CSF Firewall blocks port 11211 out of the box unless you whitelist it yourself. Also Memcached is set to listent to localhost/127.0.0.1 so won't response to public requests anyway.

    Centmin Mod users can update Memcached as per instructions at Memcached - Memcached Server 1.5.6 Update Available
     
    Last edited: Feb 28, 2018
  4. Feb 28, 2018 #4
    buik

    buik “The best traveler is one without a camera.”

    2,027
    524
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,675
    Local Time:
    4:14 PM
    Wonder if Red hat is gonna patch Memcached EL6 and 7
    upstream as both are vulnerable if you call it that way as the UDP protocol is enabled by default.
     
  5. Feb 28, 2018 #5
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah you'd hope so. I haven't used memcached provided by redhat/centos yum repos in years. I also build or compile my own memcached binaries :)
     
  6. Mar 1, 2018 #6
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    3:14 PM
    1.9.x
    10.1.x
    Issue in Vpsdime Dallas location because of this:

     
  7. Mar 1, 2018 #7
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Whoops.. this is going to bite alot of folks who don't actually use a firewall on their server ! Luckily, Centmin Mod out of box uses CSF Firewall :)
     
  8. Mar 1, 2018 #8
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    3:14 PM
    1.9.x
    10.1.x
    Yes, a firewall or a proper iptables configuration.

    Meanwhile, Vpsdime updated their report and closed 11211 port for outbound connections.

     
    Last edited: Mar 1, 2018
  9. Mar 2, 2018 #9
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    I'm working on a server now, and has Memcached that was listening on 127.0.0.1 TCP and UDP, and when I nmap the server, it's showing OPEN/FILTERED, even though port 11211 is blocked by CSF.

    Code:
    nmap TARGET -p 11211 -sU -sS --script memcached-info

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-01 19:24 UTC
Nmap scan report for TARGET
Host is up (0.12s latency).
PORT      STATE         SERVICE
11211/tcp filtered      unknown
11211/udp open|filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.74 seconds
    When I try my own:
    Code:
    nmap TARGET -p 11211 -sU -sS --script memcached-info 

Starting Nmap 6.40 ( http://nmap.org ) at 2018-03-01 19:17 UTC
Nmap scan report for TARGET
Host is up (0.018s latency).
PORT      STATE  SERVICE
11211/tcp closed unknown
11211/udp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds
    Exactly same Memcached version and config. Only difference is mine are CentOS7, and the one showing open is CentOS6.9

    Tried another CentOS6.9 server, and it's reporting OPEN/FILTERED again, even though it's blocked by CSF and only listening on 127.0.0.1
     
  10. Mar 2, 2018 #10
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    Just seen the updated to 1.5.6 as well (which these servers are all running)

    # rpm -qa | grep memcached
    memcached-1.5.6-1.el6.remi.x86_64
     
  11. Mar 2, 2018 #11
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah apparently Github was hit with this February 28th DDoS Incident Report 1.35Tbps attack ! :eek:
     
  12. Mar 2, 2018 #12
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    that's strange these are all fresh Centmin Mod installs ? you double checked /etc/csf/csf.conf doesn't open port 11211 ?
     
  13. Mar 2, 2018 #13
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    No, these aren't centmin. However, csf only has 3 udp ports open, and even with memcached updated to 1.5.6 and not even posting a listen down to udp 11211 they still report the same. Always CentOS6
     
  14. Mar 2, 2018 #14
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    so to clarify nmap reports 11211 open even with memcached 1.5.6 updated and ports blocked from public on both centmin mod installs and non-centmin mod installs that use centos 6.x ? or just non-centmin mod installs with centos 6.x ?
     
  15. Mar 2, 2018 #15
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    I'm just updating a CentOS6.9 Centminmod install now with 1.5.6 and will report back shortly.
     
  16. Mar 2, 2018 #16
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    1.5.6 Remi install on 6.9
    Code:
    # netstat -nlp | grep 11211             
tcp        0      0 127.0.0.1:11211             0.0.0.0:*                   LISTEN      2531/memcached
    Code:
    PORT      STATE         SERVICE
11211/tcp filtered      unknown
11211/udp open|filtered unknown
    1.5.6 on Centminmod 6.9
    Code:
    # netstat -nlp | grep 11211   
tcp        0      0 127.0.0.1:11211             0.0.0.0:*                   LISTEN      31241/memcached
    Code:
    Host is up (0.088s latency).
PORT      STATE    SERVICE
11211/tcp closed   unknown
11211/udp filtered unknown
    1.5.3 remi on CentOS7
    Code:
    # netstat -nlp | grep 11211 
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      1102/memcached     
udp        0      0 127.0.0.1:11211         0.0.0.0:*                           1102/memcached
    Code:
    Host is up (0.018s latency).
PORT      STATE  SERVICE
11211/tcp closed unknown
11211/udp closed unknown

    CSF on ALL the servers
    Code:
    # Allow incoming UDP ports
UDP_IN = "20,21,53"
     
  17. Mar 2, 2018 #17
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    I checked on my Centmin Mod CentOS 6.9 installs and even Memcache 1.5.2-1.5.4 are reported closed as open|filtered in nmap due to CSF Firewall.

    edit: wait might have read that incorrectly - state = open|filtered versus closed

    Port Scanning Basics | Nmap Network Scanning

     
  18. Mar 2, 2018 #18
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    I've ran a TCPDUMP on 11211 UDP when doing an nmap, and it's not getting to the packet capture, so it's being dropped / not responded to (as expected).

    Code:
    open|filtered

Nmap places ports in this state when it is unable to determine whether 
a port is open or filtered. This occurs for scan types in which open
ports give no response. The lack of response could also mean that a
packet filter dropped the probe or any response it elicited. So Nmap
does not know for sure whether the port is open or being filtered. The
UDP, IP protocol, FIN, NULL, and Xmas scans classify ports this way.
     
  19. Mar 2, 2018 #19
    eva2000

    eva2000 Administrator Staff Member

    54,901
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    1:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    came to same conclusion open|filtered for our purposes just means = filtered or closed = ok :)
     
  20. Mar 2, 2018 #20
    Matt

    Matt Well-Known Member

    932
    415
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +671
    Local Time:
    3:14 PM
    1.5.15
    MariaDB 10.2
    Had 2 customers with HiVelocity receive the same mail from them today

    Both of these servers have CSF installed and active with only the 3 default ports allowed for UDP.
     
Previous Thread Next Thread
Loading...
Page 1 of 2

.