/ Register

Want to subscribe to topics you're interested in?
Become a Member

Nginx Nginx down when DDoS

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Oxide, Apr 18, 2016.

Tags:
Previous Thread Next Thread
Loading...
Page 5 of 8
  1. May 2, 2016 #81
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    not directly but if you use fail2ban then it's reading ip from nginx logs which when real ip is configured properly in nginx as outlined in links above, then fail2ban passes nginx access log real ip to iptables

    that's also how i do it with custom script i wrote too, reading real ip from nginx access log which filters on 444 status errors to pass to CSF Firewall to block
     
  2. May 2, 2016 #82
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    Obviously none of you two understand how it works.

    Fail2ban will grab the IP from log generated by Nginx. That's completely fine, the IP will be blocked by IP Tables or IPSET. Doesn't matter.

    However, iptables blocks does not take effect if you use cloudflare.. Why? Because IPTables does not communicate with nginx, in order to get the real ip.. So when a user visits, iptables just see cloudflare ip and will allow him to access the site.. Since CLOUDFLARE IS A PROXY..

    The only service on your server, that can see the real ip is nginx.. Means iptables will block it but only if they access your site directly and not via a proxy..

    @eva2000 @Revenge
     
  3. May 2, 2016 #83
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    yes that is true if you have cloudflare in front.. both me and @Revenge referring to non-cloudflare setups

    for cloudflare what you do is script it to pass the nginx access log derived real ip to cloudflare's blacklist via cloudflare's own API :)
     
  4. May 2, 2016 #84
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    Correct.. But their API also has a limit. Either way, stop going offtopic - i just pointed out a thing to warn other users it won't work properly with cloudflare..

    not too sure how you can 'referrer' to non cloudflare setups when i stated a few times "if you use cloudflare" :p
     
  5. May 2, 2016 #85
    Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    9:22 AM
    1.9.x
    10.1.x
    @Oxide i though you were talking fail2ban were not able to get the real IP.
    You are correct, iptables will see the cloudflare IP, so even thought the IP is banned, iptables will not see it.
    Fortunately, fail2ban is able to communicate with Cloudflare and tell them to ban the real ip through their api.
     
  6. May 2, 2016 #86
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    checked /var/log/messages

    Code:
    May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:46 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: net_ratelimit: 2976 callbacks suppressed
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:51 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: net_ratelimit: 2863 callbacks suppressed
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:22:56 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: net_ratelimit: 2454 callbacks suppressed
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:01 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: net_ratelimit: 2158 callbacks suppressed
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:06 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:36 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: net_ratelimit: 1066 callbacks suppressed
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:41 centos-7-x64 kernel: nf_conntrack: table full, dropping packet
May  1 15:23:52 centos-7-x64 kernel: net_ratelimit: 184 callbacks suppressed
May  1 15:23:52 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:55 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:55 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:57 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:57 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:57 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:58 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:58 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:58 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:23:59 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: net_ratelimit: 5 callbacks suppressed
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:00 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:01 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:02 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:05 centos-7-x64 kernel: net_ratelimit: 12 callbacks suppressed
May  1 15:24:05 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:05 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:05 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:05 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:07 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:07 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:07 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:07 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:08 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:09 centos-7-x64 kernel: TCP: too many orphaned sockets
May  1 15:24:10 centos-7-x64 kernel: net_ratelimit: 3 callbacks suppressed
May  1 15:24:10 centos-7-x64 kernel: TCP: too many orphaned sockets
    This error only occured on centmin servers.
     
  7. May 2, 2016 #87
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    Issue solved:

    Code:
    echo 32768 > /sys/module/nf_conntrack/parameters/hashsize
echo 'net.netfilter.nf_conntrack_count = 131072' >> /etc/sysctl.conf
sysctl -p
    Tho, this didnt work so i had to run:

    sysctl -w net.ipv4.netfilter.ip_conntrack_max=500000
     
    Last edited: May 2, 2016
  8. May 2, 2016 #88
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Yeah that would need tweaking beyond the CentOS defaults and Centmin Mod install for 123.09beta01 at least raises those limits in latest updated 123.09beta01 centminmod/tcp.inc at 123.09beta01 · centminmod/centminmod · GitHub For centos 7 would be lines 46 to 100. But that doesn't explain non-centmin mod CentOS out of box defaults would be the same.

    123.09beta01 fresh install (not upgraded but fresh install) has
    Code (Text):
    net.netfilter.nf_conntrack_max = 524288

    I had an outstanding commit for inc/tcp.inc that I just pushed to 123.09beta01 too
     
  9. May 2, 2016 #89
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    perhaps because csf generates this? no idea..
     
  10. May 2, 2016 #90
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    woudn't be related to CSF Firewall

    for CentOS 7, centmin mod 123.09beta01 latest updates create tcp settings properly at /etc/sysctl.d/101-sysctl.conf as per routine at centminmod/tcp.inc at 123.09beta01 · centminmod/centminmod · GitHub

    for CentOS 6, centmin mod 123.09beta01 latest updates create tcp settings at /etc/sysctl.conf as per routine at centminmod/tcp.inc at 123.09beta01 · centminmod/centminmod · GitHub

    for Centos 7, you can confirm it works by removing your added entry in /etc/sysctl.conf and sysctl -p command to register the change, then run command
    Code (Text):
    sysctl -a | grep conntrack_max

    edit: i see the diff now net.netfilter.nf_conntrack_max vs net.ipv4.netfilter.ip_conntrack_max so looks like i'd need to update it for 123.09beta01 though it does register the same value either syntax for me
     
    Last edited: May 2, 2016
  11. May 2, 2016 #91
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    the server was on beta01 and it didnt have that.. guess you just added it? :)
     
  12. May 2, 2016 #92
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
  13. May 2, 2016 #93
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    yeah, thats the one i installed that you told me to - it didnt have it when i checked the conf on the server

    soon as i increased it a little it worked fine

    it was 65536
     
  14. May 2, 2016 #94
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    Last edited: May 2, 2016
  15. May 2, 2016 #95
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    could be related to CSF Firewall for connection tracking but it's disabled by default in Centmin Mod installs in /etc/csf/csf.conf
    Code (Text):
    ###############################################################################
# SECTION:Connection Tracking
###############################################################################
# Connection Tracking. This option enables tracking of all connections from IP
# addresses to the server. If the total number of connections is greater than
# this value then the offending IP address is blocked. This can be used to help
# prevent some types of DOS attack.
#
# Care should be taken with this option. It's entirely possible that you will
# see false-positives. Some protocols can be connection hungry, e.g. FTP, IMAPD
# and HTTP so it could be quite easy to trigger, especially with a lot of
# closed connections in TIME_WAIT. However, for a server that is prone to DOS
# attacks this may be very useful. A reasonable setting for this option might
# be around 300.
#
# To disable this feature, set this to 0
CT_LIMIT = "0"

# Connection Tracking interval. Set this to the the number of seconds between
# connection tracking scans
CT_INTERVAL = "30"

# Send an email alert if an IP address is blocked due to connection tracking
CT_EMAIL_ALERT = "1"

# If you want to make IP blocks permanent then set this to 1, otherwise blocks
# will be temporary and will be cleared after CT_BLOCK_TIME seconds
CT_PERMANENT = "0"

# If you opt for temporary IP blocks for CT, then the following is the interval
# in seconds that the IP will remained blocked for (e.g. 1800 = 30 mins)
CT_BLOCK_TIME = "1800"

# If you don't want to count the TIME_WAIT state against the connection count
# then set the following to "1"
CT_SKIP_TIME_WAIT = "0"

# If you only want to count specific states (e.g. SYN_RECV) then add the states
# to the following as a comma separated list. E.g. "SYN_RECV,TIME_WAIT"
#
# Leave this option empty to count all states against CT_LIMIT
CT_STATES = ""

# If you only want to count specific ports (e.g. 80,443) then add the ports
# to the following as a comma separated list. E.g. "80,443"
#
# Leave this option empty to count all ports against CT_LIMIT
CT_PORTS = ""


    EDIT: that's connection per ip tracking whoops. Looks like CSF Firewall does make use of connection tracking then.
     
    Last edited: May 2, 2016
  16. May 2, 2016 #96
    Oxide

    Oxide Active Member

    534
    29
    28
    Mar 19, 2015
    Ratings:
    +59
    Local Time:
    6:22 PM
    Hey, at least we managed to solve it. It was annoying me, a lot :) I'm happy now ^_^

    Thanks to everyone, and especially @Ahmad habibi for telling me to check /var/log/messages in real time during the attack.
     
  17. May 2, 2016 #97
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    centmin mod 123.09beta01 just got that tiny step better because of it - pays to troubleshoot yourself when you can too :)

    majority of may 2 commits to 123.09beta01 are to improve the tcp tweaks out of box see Commits · centminmod/centminmod · GitHub
     
  18. May 3, 2016 #98
    pamamolf

    pamamolf Well-Known Member

    4,125
    429
    83
    May 31, 2014
    Ratings:
    +841
    Local Time:
    11:22 AM
    Nginx-1.29.x
    MariaDB 10.6.x
    How can we add now the new changes related to this issue without reinstalling?

    Also i am looking for a config file not related only to this Wordpress attack but an one in general that will block users with many and fast connections to it without anyway to block a user agent.....
     
  19. May 3, 2016 #99
    eva2000

    eva2000 Administrator Staff Member

    58,894
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:22 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    just update to latest 123.09beta01 code via centmin.sh menu option 23 and re-run centmin.sh once - should auto append/adjust your system tcp settings with the new changes

    that's just normal nginx request and connection rate limiting in play then
     
  20. May 3, 2016 #100
    pamamolf

    pamamolf Well-Known Member

    4,125
    429
    83
    May 31, 2014
    Ratings:
    +841
    Local Time:
    11:22 AM
    Nginx-1.29.x
    MariaDB 10.6.x
    Do i have to restart any services for the changes to be active?

    Yes but the default nginx limiting does not ban users ip's....
     
Previous Thread Next Thread
Loading...
Page 5 of 8

.