Want to subscribe to topics you're interested in?
Become a Member

Nginx Nginx down when DDoS

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Oxide, Apr 18, 2016.

  1. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    @eva2000 i am using OVH Kernel. Could this be why?

    Would OVH Kernel has some limits, that stops nginx from recieveing a certain amount of requests??

    A friend of mine runs Debian, nginx - it doesn't even lag. Not sure if it's centmin installation, or OVH Kernel.

    The file they are attacking, is a html file - so it should use zero amount of resources. I also response with 444 (null byte). The CPU, is literally using nothing and the web server is down. Logic?
     
  2. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    hard to say but i doubt the OVH kernel would have such a limitation but can't be sure as i only use centos distro kernels
     
  3. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    I've sent them a ticket now. A friend of mine uses Debian, with Nginx. Using Proxmox, and he has no issues with receiving large amount of requests.

    Do you have any test sites for nginx? I can ask them to launch the request attack on yours, then you'd be able to see too. Happens on all my centminmod servers..
     
  4. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    no test sites nor have time.. dealing with my own Forum DDOS Attacked - Linode null routed | Centmin Mod Community but no time due to my mother's heart surgery etc
     
  5. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    Rented a digitalocean smallest VPS for $5.00 USD

    IT was not able to take it down, all i did was yum install nginx.. no custom config

    [​IMG]

    its either centmin or ovh kernel..
     
  6. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    one thing i can think of is you're attacking yourself and CSF firewall has additional connection tracking enabled, CSF could be temp blocking your ip - which is what you'd want to happen for the attacker's ip address to be blocked !
    yes that specific error is a timeout with php not nginx
    yeah you'd need to troubleshoot yourself.. try centmin mod with temp disable of csf firewall
     
    Last edited: Apr 24, 2016
  7. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    i do not have csf firewall enabled, also the attack is a actual xml-rpc attack (pingback)
     
  8. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    @eva2000 how would you even change kernel? there is no option when reinstalling
     
  9. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    there should be an option never used OVH only SoYouStart though so might be in different place so might need to ask OVH support or get their documentation
     
  10. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    There are none guide lines according to the support. By that, i assume Centmin is not comptiable with OVH? o_O
     
  11. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  12. pamamolf

    pamamolf Premium Member Premium Member

    3,474
    334
    83
    May 31, 2014
    Ratings:
    +641
    Local Time:
    3:12 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    You have an option when you installing your OS to select default kernel or Ovh kernel and there is only one exception for latest huge servers that need to load Ovh custom kernel for some needed drivers but Centminmod works great on them :)
     
    • Informative Informative x 1
  13. Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:12 PM
    1.9.x
    10.1.x
    Exactly, i always use the standard kernel instead of the OVH kernel. We can change that when installing the OS, although is also possible to change it later.
     
    • Informative Informative x 1
  14. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM

    Please don't confuse "OVH.COM" with "SOYOUSTART" or "KIMSUFI". They have all different panels and options, downsides and upsides.

    [​IMG]

    [​IMG]
     
  15. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    OVH Said this now in regards of the attack.

    "After verification, we believe that the webserver crashes before the attack
    has time to impact the server in itself
    . In any ways, we have placed your IP
    under a profile designed to protect against those kind of attacks. If you are
    still experiencing issues, please provide us with details on those new
    attacks."

    By that I assume misconfiguration, but I am using the default centminmod install so not sure what could go wrong? Unless there is something that really fucks up when using OVH Kernel w/ centmin. Since services like FTP also stop responding.

    no idea, i really need help with this - able to pay :)

    it just makes no sense how nginx is crashing if what OVH is saying is true. It uses 0% CPU, so it crashes - before impact? WHAT?

    nginx debug didnt show anything either... that i saw.
     
    Last edited: Apr 25, 2016
  16. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    I disabled all kind of logging.
    I block invalid user-agents and return 444 (Null bytes)

    Server seems to crash, web server. It uses 0% CPU, which makes.. No sense. Unless I have fucked up somewhere in my nginx config.

    Oh, and the server we're testing is a big dedi (SP-64) - Permanent Anti-DDoS PRO.

    Yes, we are also attacking a index.html (No PHP, nothing).

    Code:
    server {
    #         listen   80;
    listen   80 default_server backlog=2048 reuseport;
    server_name main-ovh.octolus.net;
    root   html;
    
    
    location /nginx_status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    #allow youripaddress;
    deny all;
    }
    
    if ($http_user_agent ~* "PHP|curl|Wget|HTTrack|Nmap|Verifying|PingBack|Pingdom|Joomla|Wordpress") { return 444; }
    if ($http_user_agent = "-") { return 444; }
    if ($http_user_agent = "") { return 444; }
    if ($http_user_agent = " ") { return 444; }
    
    
    
    
    include /usr/local/nginx/conf/staticfiles.conf;
    include /usr/local/nginx/conf/include_opcache.conf;
    include /usr/local/nginx/conf/php.conf;
    #include /usr/local/nginx/conf/phpstatus.conf;
    include /usr/local/nginx/conf/drop.conf;
    #include /usr/local/nginx/conf/errorpage.conf;
    include /usr/local/nginx/conf/vts_mainserver.conf;
    
    }

    Code:
    user              nginx nginx;
    worker_processes auto;
    worker_priority -10;
    
    worker_rlimit_nofile 260000;
    timer_resolution 100ms;
    
    pcre_jit on;
    
    pid         logs/nginx.pid;
    
    events {
    worker_connections  65536;
    accept_mutex on;
    accept_mutex_delay 200ms;
    use epoll;
    multi_accept on;
    }
    
    http {
    
    more_set_headers "Server: nginx octolus";
    
    set_real_ip_from 199.27.128.0/21;
    set_real_ip_from 173.245.48.0/20;
    set_real_ip_from 103.21.244.0/22;
    set_real_ip_from 103.22.200.0/22;
    set_real_ip_from 103.31.4.0/22;
    set_real_ip_from 141.101.64.0/18;
    set_real_ip_from 108.162.192.0/18;
    set_real_ip_from 190.93.240.0/20;
    set_real_ip_from 188.114.96.0/20;
    set_real_ip_from 197.234.240.0/22;
    set_real_ip_from 198.41.128.0/17;
    set_real_ip_from 162.158.0.0/15;
    set_real_ip_from 104.16.0.0/12;
    real_ip_header CF-Connecting-IP;
    
    
    include /usr/local/nginx/conf/vts_http.conf;
    include /usr/local/nginx/conf/geoip.conf;
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
    include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
    
    log_format      main    '$remote_addr - $remote_user [$time_local] $request '
    '"$status" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
    ' "$connection" "$connection_requests" "$request_time"';
    
    access_log  logs/access.log combined buffer=128k flush=5m;
    error_log   logs/error.log warn;
    
    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;
    
    sendfile on;
    sendfile_max_chunk 512k;
    tcp_nopush  on;
    tcp_nodelay on;
    server_tokens off;
    server_name_in_redirect off;
    
    keepalive_timeout  8;
    keepalive_requests 1000;
    lingering_time 20s;
    lingering_timeout 5s;
    keepalive_disable msie6;
    
    gzip on;
    gzip_vary   on;
    gzip_disable "MSIE [1-6]\.";
    gzip_static on;
    gzip_min_length   1400;
    gzip_buffers      32 8k;
    gzip_http_version 1.0;
    gzip_comp_level 5;
    gzip_proxied    any;
    gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    
    client_body_buffer_size 256k;
    client_body_in_file_only off;
    client_body_timeout 10s;
    client_header_buffer_size 64k;
    ## how long a connection has to complete sending
    ## it's headers for request to be processed
    client_header_timeout  8s;
    client_max_body_size 50m;
    connection_pool_size  512;
    directio  4m;
    ignore_invalid_headers on;     
    large_client_header_buffers 8 64k;
    output_buffers   8 256k;
    postpone_output  1460;
    proxy_temp_path  /tmp/nginx_proxy/;
    request_pool_size  32k;
    reset_timedout_connection on;
    send_timeout     15s;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 64;
    
    # for nginx proxy backends to prevent redirects to backend port
    # port_in_redirect off;
    
    open_file_cache max=50000 inactive=60s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;
    open_log_file_cache max=10000 inactive=30s min_uses=2;
    
    ## limit number of concurrency connections per ip to 16
    ## add to your server {} section the next line
    ## limit_conn limit_per_ip 16;
    ## uncomment below line allows 500K sessions
    # limit_conn_log_level error;
    #######################################
    # use limit_zone for Nginx <v1.1.7 and lower
    # limit_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    # use limit_conn_zone for Nginx >v1.1.8 and higher
    # limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    
    include /usr/local/nginx/conf/conf.d/*.conf;
    }
    
     
  17. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    Tested with Siege.

    I can take over 49,000 requests a second, so Nginx is working fine. Only left thing I can think it can be, would be CSF Firewall.
     
  18. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    [​IMG]

    [​IMG]


    See how CPU goes from 97 to 100%? It idles at 100% soon as it goes under attack.. Logic?
     
  19. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    10:12 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    suggests it's either at server CSF firewall level or at OVH level ddos migitation that it's occuring
     
  20. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:12 PM
    indeed, i disabled csf and it still happens.. HAS to be ovh lol