Welcome to Centmin Mod Community
Register Now

Nginx Identifying bottleneck when load testing on nginx?

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Peace, Feb 20, 2016.

Tags:
  1. Peace

    Peace New Member

    4
    2
    3
    Feb 20, 2016
    Ratings:
    +2
    Local Time:
    4:45 PM
    I'm trying to optimize a Digital Ocean droplet (512mb), testing using loader.io

    I'm testing on my homepage, which is xenForo (over HTTPS). With the help of everyone at the xenForo community, I also setup FastCGI page cache for guest users, which got me from 100 req/sec to 2,000 req/sec.

    But anything beyond 2,000 req/sec is resulting in a lot of timeouts and slow responses (goes from 20ms average to 1500ms average). I'm trying to identify the bottleneck. It's not CPU/memory yet, because load barely reaches 0.30 and memory usage is about half. I tried resizing to a much bigger droplet, and the timeouts still happen.

    It's not FastCGI because the load testing performance is nearly identical on a basic .html file.

    During the timeouts, error.log is empty. Nothing seems to be throwing errors (that I can find). Kern.log has this log:

    Code:
    TCP: Possible SYN flooding on port 80. Sending cookies.  Check SNMP counters
    TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.
    
    I tried disabling syncookies, which stopped those errors, but the timeouts still persisted.

    During the timeouts, I start seeing a buildup of TIME_WAIT:

    Code:
    netstat -ntla | awk '{print $6}' | sort | uniq -c | sort -rn
       6268 ESTABLISHED
        831 TIME_WAIT
          6 LISTEN
          2 FIN_WAIT1
          1 Foreign
          1 established)
    My question is, where else can I look to determine the bottleneck here? Are there other error logs or commands I can use to monitor?

    Here is my nginx.conf (FastCGI and regular browser cache are in my default file). I've tried multi_accept, which seems to worsen the timeouts. I know worker_connections is ridiculous, but it doesn't seem to matter how much I raise or lower it:

    Code:
    user www-data;
    worker_processes auto;
    worker_rlimit_nofile 200000;
    pid /run/nginx.pid;
    
    events {
        worker_connections 200000;
        # multi_accept on;
        use epoll;
    
    }
    
    http {
    
        ##
        # Basic Settings
        ##
    
        open_file_cache max=200000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;
        server_tokens off;
    
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 30;
        types_hash_max_size 2048;
    
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
    
        ##
        # Logging Settings
        ##
        access_log off;
        # access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
    
        ##
        # Gzip Settings
        ##
    
        gzip on;
        gzip_disable "msie6";
    
        gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
    }
    
    Here is my sysctl.conf

    Code:
    vm.swappiness = 10
    vm.dirty_ratio = 60
    vm.dirty_background_ratio = 2
    
    ### GENERAL NETWORK SECURITY OPTIONS ###
    
    # Number of times SYNACKs for passive TCP connection.
    net.ipv4.tcp_synack_retries = 2
    
    # Allowed local port range
    net.ipv4.ip_local_port_range = 2000 65535
    
    # Protect Against TCP Time-Wait
    net.ipv4.tcp_rfc1337 = 1
    
    # Decrease the time default value for tcp_fin_timeout connection
    net.ipv4.tcp_fin_timeout = 15
    
    # Decrease the time default value for connections to keep alive
    net.ipv4.tcp_keepalive_time = 300
    net.ipv4.tcp_keepalive_probes = 5
    net.ipv4.tcp_keepalive_intvl = 15
    net.ipv4.tcp_syncookies = 1
    
    ### TUNING NETWORK PERFORMANCE ###
    
    # Default Socket Receive Buffer
    net.core.rmem_default = 31457280
    
    # Maximum Socket Receive Buffer
    net.core.rmem_max = 12582912
    
    # Default Socket Send Buffer
    net.core.wmem_default = 31457280
    
    # Maximum Socket Send Buffer
    net.core.wmem_max = 12582912
    
    # Increase number of incoming connections
    net.core.somaxconn = 4096
    
    And I put these in limits.conf:

    Code:
    *         hard    nofile      500000
    *         soft    nofile      500000
    root      hard    nofile      500000
    root      soft    nofile      500000
    Thanks!
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    7:45 AM
    Nginx 1.13.x
    MariaDB 5.5
    Are you using Centmin Mod LEMP stack ? those settings and configuration format don't look anything like Centmin Mod's out of box format ? This community is for discussing Centmin Mod LEMP stack's Nginx and PHP-FPM configuration ;)

    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)

    What you described is mainly higher level kernel and TCP tuning and as such you're on your for that as to advice from me. This level of optimisation I usually reserve for regular paid optimisation clients I deal with :) But other members are free to provide suggestions etc.

    Just one thing to remember you could be also running up against the limited size of your network ethernet configuration of your server.
     
    • Like Like x 1
  3. Peace

    Peace New Member

    4
    2
    3
    Feb 20, 2016
    Ratings:
    +2
    Local Time:
    4:45 PM
    Hey @eva2000 thanks for the response! Those are definitely not out-of-box configurations, all custom from tweaking - do you have a website for your paid services? We would absolutely be interested in that!
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    7:45 AM
    Nginx 1.13.x
    MariaDB 5.5
    might want to private message me though my rates start in the low 4 figures $$$$ so for most folks not their cup of tea :)
     
    • Like Like x 1
    • Funny Funny x 1
  5. Peace

    Peace New Member

    4
    2
    3
    Feb 20, 2016
    Ratings:
    +2
    Local Time:
    4:45 PM
    Awesome, thank you! I will send a PM :)