Join the community today
Register Now

Nginx Download speed limit

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by linuxtux, Oct 8, 2017.

  1. linuxtux

    linuxtux New Member

    5
    1
    3
    Oct 8, 2017
    Ratings:
    +1
    Local Time:
    10:10 PM
    Hello,
    Since a few months ago we have see a strange 5.0MB/s limit when download files from my servers with centminmod. First i thought that the problem was my provider but we have installed new centminmod servers in other providers and have the same 5.0MB/s limit.
    The limit is not fixed but many many times "lock" at this speed even with local gigabit network.
    From same servers downloads from ssh even using scp are fine and we have install apache to try in same servers (stopped nginx) and apache dont have any issues. We have problems with centos 6 and 7 and with stable and beta versions.

    Do you know what is the problem? We have a few old installations (+2 years old) and dont have issues in this installs. Try with same nginx version on new servers but still have problem.

    Versions from server that DONT have that issue:
    [root@server /]# nginx -v
    nginx version: nginx/1.9.13
    [root@server /]# php -v
    PHP 5.4.45 (cli) (built: Apr 2 2016 11:04:51)

    Any new centminmod install have the issue.

    Any idea?

    Thanks,
    [​IMG]
    [​IMG]
     
    Last edited: Oct 8, 2017
  2. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    You'll need to post on the forums with the following info
    • Server or VPS details ? XEN, KVM, OpenVZ, VMWare or dedicated server ? OS ? CentOS 6.7 or 7.2 ? 32bit or 64bit ?
    • ram, disk , cpus ?
    • Was it fresh install or upgrade ?
    • Method of install ? Via centmin.sh menu option 1, Git install or curl one liner install as outlined at centminmod.com/download.html ?
    • How long ago did you install Centmin Mod ?
    • Example wget commands tested ?
    • Geographic local of centmin mod servers and download sources ?
    • Size of download files ?
    • Have you tested disk i/o speed on old and new servers ?
    i.e. wget command with verbose output from 123.09beta01 washington dc vps
    Code (Text):
    wget -v -O /dev/null http://speedtest.newark.linode.com/100MB-newark.bin
    --2017-10-08 03:01:44--  http://speedtest.newark.linode.com/100MB-newark.bin
    Resolving speedtest.newark.linode.com... 2600:3c03::4b, 50.116.57.237
    Connecting to speedtest.newark.linode.com|2600:3c03::4b|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 104857600 (100M) [application/octet-stream]
    Saving to: '/dev/null'
    
    /dev/null                                                      100%[==================================================================================================================================================>] 100.00M   101MB/s    in 1.0s 
    
    2017-10-08 03:01:45 (101 MB/s) - '/dev/null' saved [104857600/104857600]
    

    for posting code, wget commands and output you might want to use CODE tags for code How to use forum BBCODE code tags :)

    When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)
    • Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf
    • Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf
    • Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com
    • Vhost public web root will be at /home/nginx/domains/newdomain.com/public
    • Vhost log directory will be at /home/nginx/domains/newdomain.com/log
    Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)

    Also please post the contents of /usr/local/nginx/conf/nginx.conf for one working old install and one new install with issues.

    Output for or one working old install and one new install with issues
    Code (Text):
    df -hT
     
    Last edited: Oct 8, 2017
  3. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    still need above info, but made an update to 123.09beta01 nginx.conf for more optimised performance for large file serving from Centmin Mod Nginx Beta Branch - fine tuning nginx.conf for large file downloads in 123.09beta01

    if you update code via centmin.sh menu option 23 submenu option 2 and then re-run centmin.sh, it should auto apply the new settings. If not just manually make the changes in nginx.conf
     
  4. linuxtux

    linuxtux New Member

    5
    1
    3
    Oct 8, 2017
    Ratings:
    +1
    Local Time:
    10:10 PM
    Hello,
    The problem is download a file served by a centminmod nginx server not donwload externaly.

    We use a lot of plataforms (openvz, kvm and dedicated server) and have the problem in all. To test this we have setup a new vmware vm on a cloud provider (outside our dc and hardware) with centos 7.2 64 bits with 4vcpu, 2GB ram and 20GBs hdd.
    Its a fresh installation by running

    Code (Text):
    yum -y update; curl -O https://centminmod.com/installer.sh && chmod 0700 installer.sh && bash installer.sh


    We have install centminmod create a new vhosts by option 2 with IP of server and then put a 800MB file on it. Then try download that file in another server running:

    Code (Text):
    wget -v -O /dev/null IP/p.mkv


    And speed dis bellow 5MBs. If i install a apache server without any modifications (yum install apache, stop nginx and put same file) we have much better speeds (20MBs) so the problem is not IOs, network or server itself but nginx.


    Code (Text):
    2017-10-09 01:23:25 (4.53 MB/s) - “/dev/null” saved [882787329/882787329]


    Even in localhost with 1Gbps network have that issue.

    vhost:

    Code (Text):
    # Centmin Mod Getting Started Guide
    # must read http://centminmod.com/getstarted.html
    
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    # if unsure use return 302 before using return 301
    #server {
    #            listen   80;
    #            server_name 176.xxx.xxx.xxx;
    #            return 301 $scheme://www.176.xxx.xxx.xxx$request_uri;
    #       }
    
    server {
      server_name 176.xxx.xxx.xxx www.176.xxx.xxx.xxx;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/176.xxx.xxx.xxx/log/access.log combined buffer=256k flush=60m;
      error_log /home/nginx/domains/176.xxx.xxx.xxx/log/error.log;
    
      root /home/nginx/domains/176.xxx.xxx.xxx/public;
    
      # prevent access to ./directories and files
      location ~ (?:^|/)\. {
       deny all;
      }
    
      location / {
    
    # block common exploits, sql injections etc
    #include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Enable for vBulletin usage WITHOUT vbSEO installed
      # More example Nginx vhost configurations at
      # http://centminmod.com/nginx_configure.html
      #try_files            $uri $uri/ /index.php;
    
      }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    


    nginx.conf of this brand new install:
    Code (Text):
    user              nginx nginx;
    worker_processes 2;
    worker_priority -10;
    
    worker_rlimit_nofile 260000;
    timer_resolution 100ms;
    
    pcre_jit on;
    
    pid         logs/nginx.pid;
    
    events {
        worker_connections  15000;
        accept_mutex on;
        accept_mutex_delay 200ms;
        use epoll;
        #multi_accept on;
    }
    
    http {
    limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
    #limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;
    
    more_set_headers "Server: nginx centminmod";
    
    include /usr/local/nginx/conf/vts_http.conf;
    include /usr/local/nginx/conf/geoip.conf;
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
    include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
    
    log_format      main    '$remote_addr - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    access_log  logs/access.log combined buffer=128k flush=5m;
    error_log   logs/error.log warn;
    
        index  index.php index.html index.htm;
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
    
            sendfile on;
            sendfile_max_chunk 512k;
            tcp_nopush  on;
            tcp_nodelay on;
            server_tokens off;
            server_name_in_redirect off;
    
            keepalive_timeout  5;
            keepalive_requests 1000;
            lingering_time 20s;
            lingering_timeout 5s;
            keepalive_disable msie6;
    
            gzip on;
            gzip_vary   on;
            gzip_disable "MSIE [1-6]\.";
            gzip_static on;
            gzip_min_length   1400;
            gzip_buffers      32 8k;
            gzip_http_version 1.0;
            gzip_comp_level 5;
            gzip_proxied    any;
            gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    
     client_body_buffer_size 256k;
     client_body_in_file_only off;
     client_body_timeout 10s;
     client_header_buffer_size 64k;
    ## how long a connection has to complete sending
    ## it's headers for request to be processed
     client_header_timeout  8s;
     client_max_body_size 50m;
     connection_pool_size  512;
     directio  4m;
     ignore_invalid_headers on;
     large_client_header_buffers 8 64k;
     output_buffers   8 256k;
     postpone_output  1460;
     proxy_temp_path  /tmp/nginx_proxy/;
     request_pool_size  32k;
     reset_timedout_connection on;
     send_timeout     15s;
     types_hash_max_size 2048;
     server_names_hash_bucket_size 64;
    
    # for nginx proxy backends to prevent redirects to backend port
    # port_in_redirect off;
    
    open_file_cache max=50000 inactive=60s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;
    open_log_file_cache max=10000 inactive=30s min_uses=2;
    
    ## limit number of concurrency connections per ip to 16
    ## add to your server {} section the next line
    ## limit_conn limit_per_ip 16;
    ## uncomment below line allows 500K sessions
    # limit_conn_log_level error;
    #######################################
    # use limit_zone for Nginx <v1.1.7 and lower
    # limit_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    # use limit_conn_zone for Nginx >v1.1.8 and higher
    # limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    
     include /usr/local/nginx/conf/conf.d/*.conf;
    }
    


    Code (Text):
    [root@localhost conf.d]# df -hT
    Filesystem              Type      Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root xfs        18G  5.3G   13G  30% /
    devtmpfs                devtmpfs  984M     0  984M   0% /dev
    tmpfs                   tmpfs     993M     0  993M   0% /dev/shm
    tmpfs                   tmpfs     993M  8.5M  985M   1% /run
    tmpfs                   tmpfs     993M     0  993M   0% /sys/fs/cgroup
    /dev/sda2               xfs       497M  163M  335M  33% /boot
    tmpfs                   tmpfs     199M     0  199M   0% /run/user/0
    


    Have made the changes in nginx
    Code (Text):
    client_max_body_size 200m;
    output_buffers   1 512k;
    send_timeout     30s;


    And have worst speeds (same client and server):
    Code (Text):
    Length: 882787329 (842M) [application/octet-stream]
    Saving to: “/dev/null”
    
    100%[===================================================================================================================================================================================================>] 882,787,329 5.06M/s   in 10m 2s
    
    2017-10-09 01:38:51 (1.40 MB/s) - “/dev/null” saved [882787329/882787329]
    


    Changes reverted and back to <5MBs

    About servers dont have problem, for example its a old centos 6.9 on a openvz node. In that node (SAME node) have new installs with problems.

    Code (Text):
    [root@server /]# df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    /dev/simfs     simfs      31G   11G   15G  41% /
    none           devtmpfs   12G  4.0K   12G   1% /dev
    none           tmpfs      12G     0   12G   0% /dev/shm
    tmpfs          tmpfs      12G  228K   12G   1% /tmp
    
    

    Code (Text):
    server {
      server_name old.xxxxx.com www.old.xxxxx.com;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      #access_log /home/nginx/domains/old.xxxx.com/log/access.log;
      error_log /home/nginx/domains/old.xxxx.com/log/error.log;
    
      root /home/nginx/domains/old.xxxx.com/public;
    
      location / {
    
       index index.php index.html index.htm;
       try_files $uri $uri/ @kohana; #index.php$uri?$args;
      }
    location @kohana {
                    rewrite ^/(.+)$ /index.php$request_uri?$args last;
      }
    location ~* \.php.*$ {
                    fastcgi_pass 127.0.0.1:9000;
                    include fastcgi_params;
                    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                    fastcgi_param KOHANA_ENV production;
                    fastcgi_cache off;
                    fastcgi_index index.php;
      }
    
      #include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
    


    Code (Text):
    user              nginx nginx;
    worker_processes 4;
    worker_priority -10;
    
    worker_rlimit_nofile 51200;
    timer_resolution 100ms;
    
    #error_log         logs/error.log;
    
    pid               logs/nginx.pid;
    
    events {
        worker_connections  2048;
        use epoll;
        #multi_accept on;
    }
    
    http {
    include /usr/local/nginx/conf/geoip.conf;
    include /usr/local/nginx/conf/maintenance.conf;
    
    more_set_headers "X-Powered-By: centminmod";
    
    include /usr/local/nginx/conf/vts_http.conf;
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
    
    include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
    
    log_format      main    '$remote_addr - $remote_user [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                    ' "$connection" "$connection_requests" "$request_time"';
    
    #access_log logs/access.log combined buffer=32k;
    
        index  index.php index.html index.htm;
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
    
            sendfile on;
            #sendfile_max_chunk 1m;
            tcp_nopush  on;
            tcp_nodelay off;
            server_tokens off;
            server_name_in_redirect off;
    
            keepalive_timeout  10;
            keepalive_disable msie6;
    
            gzip on;
            gzip_vary   on;
            gzip_disable "MSIE [1-6]\.";
            gzip_static on;
            gzip_min_length   1400;
            gzip_buffers      32 8k;
            gzip_http_version 1.0;
            gzip_comp_level 5;
            gzip_proxied    any;
            gzip_types text/plain text/css text/xml application/javascript  application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;
    
     client_body_buffer_size 256k;
     client_body_in_file_only off;
     client_body_timeout 60s;
     client_header_buffer_size 64k;
    ## how long a connection has to complete sending
    ## it's headers for request to be processed
     client_header_timeout  20s;
     client_max_body_size 10024k;
     connection_pool_size  512;
     directio  4m;
     ignore_invalid_headers on;
     large_client_header_buffers 8 64k;
     output_buffers   8 256k;
     postpone_output  1460;
     proxy_temp_path  /tmp/nginx_proxy/;
     request_pool_size  32k;
     reset_timedout_connection on;
     send_timeout     60s;
     types_hash_max_size 2048;
     server_names_hash_bucket_size 64;
    
    # for nginx proxy backends to prevent redirects to backend port
    # port_in_redirect off;
    
    open_file_cache max=5000 inactive=30s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors off;
    open_log_file_cache max=1024 inactive=30s min_uses=2;
    
    ## limit number of concurrency connections per ip to 16
    ## add to your server {} section the next line
    ## limit_conn limit_per_ip 16;
    ## uncomment below line allows 500K sessions
    # limit_conn_log_level error;
    #######################################
    # use limit_zone for Nginx <v1.1.7 and lower
    # limit_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
    # use limit_conn_zone for Nginx >v1.1.8 and higher
    # limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
    #######################################
     include /usr/local/nginx/conf/conf.d/*.conf;
    }
    


    This server (from same client) and in same network of the others have a much better speeds (servers are in diffrent countrys but its much better that 5MBs.)
    Code (Text):
    root@xxxx [~]# wget -v -O /dev/null old.xxxx.com/xxx.xxx
    --2017-10-09 01:35:24--  http://old.xxxxx.com/x.xxx
    Resolving old.xxxx.com... xxx.xxx.xxx.xxx
    Connecting to old.stock-off.com|xxx.xxx.xxx.xxx|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 882787329 (842M) [application/octet-stream]
    Saving to: “/dev/null”
    
    100%[===========================================================================================================================================>] 882,787,329 15.2M/s   in 59s
    
    2017-10-09 01:36:25 (14.2 MB/s) - “/dev/null” saved [882787329/882787329]
    


    Now download from same good server from another in same datacenter:

    Code (Text):
    HTTP request sent, awaiting response... 200 OK
    Length: 882787329 (842M) [application/octet-stream]
    Saving to: “/dev/null”
    
    100%[=================================================================================================================>] 882,787,329  104M/s   in 8.2s
    
    2017-10-09 01:39:47 (103 MB/s) - “/dev/null” saved [882787329/882787329]
    

    Thanks in advance.
     
    Last edited: Oct 9, 2017
  5. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    FYI if you test with 800MB file, then nginx setting isn't enough as that limits to 200M, so need to raise value from 50m or 200m to 1024m
    Code (Text):
    client_max_body_size 200m;
    
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    on working server i see slightly different settings
    Code (Text):
     client_body_timeout 60s;
     client_header_timeout  20s;
     client_max_body_size 10024k;
     send_timeout     60s;
    

    try on problematic server
    Code (Text):
     client_body_timeout 60s;
     client_header_timeout  20s;
     client_max_body_size 1024m;
     send_timeout     60s;
    
     
  7. linuxtux

    linuxtux New Member

    5
    1
    3
    Oct 8, 2017
    Ratings:
    +1
    Local Time:
    10:10 PM
    Hello,
    We have changed client_max_body_size to 1024m and download speed improved a little bit (from 5MBs to 7MBs) but still have more better speeds from old server and old server have a even less limit.

    Code (Text):
     client_max_body_size 10024k;


    We will install another brand new VM with beta version and back with results soon.
     
  8. linuxtux

    linuxtux New Member

    5
    1
    3
    Oct 8, 2017
    Ratings:
    +1
    Local Time:
    10:10 PM
    Same result with that settings :/
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:10 AM
    Nginx 1.13.x
    MariaDB 5.5
    Also try adding to your specific nginx vhost domain.com.conf file instead just after # ssi on; line the following. For value directio_alignment set to 4096 if your /home partition is using xfs filesystem or to 512 if using ext4
    Code (Text):
    # ssi on;
    
    client_max_body_size 1024m;
    directio  4m;
    directio_alignment 512;
    reset_timedout_connection off;
    send_timeout 120s;
    keepalive_timeout  60;
    keepalive_requests 100;
    sendfile on;
    sendfile_max_chunk 1m;
    output_buffers 1 512k;
    

    for large files might need much more tuning post-install than nginx defaults as the values might not be good for non-large file performance but good for large files. Also Centmin Mod Nginx supports Nginx thread pools so you can configure those yourself too Boosting NGINX Performance 9x with Thread Pools

    Also noticed on problematic server you have accept_mutex on but 123.09beta01 default nginx.conf has accept_mutex off
     
    Last edited: Oct 9, 2017