Nginx Download speed limit

Hello,
Since a few months ago we have see a strange 5.0MB/s limit when download files from my servers with centminmod. First i thought that the problem was my provider but we have installed new centminmod servers in other providers and have the same 5.0MB/s limit.
The limit is not fixed but many many times "lock" at this speed even with local gigabit network.
From same servers downloads from ssh even using scp are fine and we have install apache to try in same servers (stopped nginx) and apache dont have any issues. We have problems with centos 6 and 7 and with stable and beta versions.

Do you know what is the problem? We have a few old installations (+2 years old) and dont have issues in this installs. Try with same nginx version on new servers but still have problem.

Versions from server that DONT have that issue:
[root@server /]# nginx -v
nginx version: nginx/1.9.13
[root@server /]# php -v
PHP 5.4.45 (cli) (built: Apr 2 2016 11:04:51)

Any new centminmod install have the issue.

Any idea?

---

Thanks,

 

Hello,
The problem is download a file served by a centminmod nginx server not donwload externaly.

We use a lot of plataforms (openvz, kvm and dedicated server) and have the problem in all. To test this we have setup a new vmware vm on a cloud provider (outside our dc and hardware) with centos 7.2 64 bits with 4vcpu, 2GB ram and 20GBs hdd.
Its a fresh installation by running

Code (Text):

yum -y update; curl -O https://centminmod.com/installer.sh && chmod 0700 installer.sh && bash installer.sh

We have install centminmod create a new vhosts by option 2 with IP of server and then put a 800MB file on it. Then try download that file in another server running:

Code (Text):

wget -v -O /dev/null IP/p.mkv

And speed dis bellow 5MBs. If i install a apache server without any modifications (yum install apache, stop nginx and put same file) we have much better speeds (20MBs) so the problem is not IOs, network or server itself but nginx.

Code (Text):

2017-10-09 01:23:25 (4.53 MB/s) - “/dev/null” saved [882787329/882787329]

Even in localhost with 1Gbps network have that issue.

vhost:

Code (Text):

# Centmin Mod Getting Started Guide
# must read http://centminmod.com/getstarted.html

# redirect from non-www to www
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
#server {
#            listen   80;
#            server_name 176.xxx.xxx.xxx;
#            return 301 $scheme://www.176.xxx.xxx.xxx$request_uri;
#       }

server {
  server_name 176.xxx.xxx.xxx www.176.xxx.xxx.xxx;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/176.xxx.xxx.xxx/log/access.log combined buffer=256k flush=60m;
  error_log /home/nginx/domains/176.xxx.xxx.xxx/log/error.log;

  root /home/nginx/domains/176.xxx.xxx.xxx/public;

  # prevent access to ./directories and files
  location ~ (?:^|/)\. {
   deny all;
  }

  location / {

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Enable for vBulletin usage WITHOUT vbSEO installed
  # More example Nginx vhost configurations at
  # http://centminmod.com/nginx_configure.html
  #try_files            $uri $uri/ /index.php;

  }

  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

nginx.conf of this brand new install:

Code (Text):

user              nginx nginx;
worker_processes 2;
worker_priority -10;

worker_rlimit_nofile 260000;
timer_resolution 100ms;

pcre_jit on;

pid         logs/nginx.pid;

events {
    worker_connections  15000;
    accept_mutex on;
    accept_mutex_delay 200ms;
    use epoll;
    #multi_accept on;
}

http {
limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;

more_set_headers "Server: nginx centminmod";

include /usr/local/nginx/conf/vts_http.conf;
include /usr/local/nginx/conf/geoip.conf;
#include /usr/local/nginx/conf/pagespeedadmin.conf;
include /usr/local/nginx/conf/fastcgi_param_https_map.conf;

log_format      main    '$remote_addr - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

access_log  logs/access.log combined buffer=128k flush=5m;
error_log   logs/error.log warn;

    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

        sendfile on;
        sendfile_max_chunk 512k;
        tcp_nopush  on;
        tcp_nodelay on;
        server_tokens off;
        server_name_in_redirect off;

        keepalive_timeout  5;
        keepalive_requests 1000;
        lingering_time 20s;
        lingering_timeout 5s;
        keepalive_disable msie6;

        gzip on;
        gzip_vary   on;
        gzip_disable "MSIE [1-6]\.";
        gzip_static on;
        gzip_min_length   1400;
        gzip_buffers      32 8k;
        gzip_http_version 1.0;
        gzip_comp_level 5;
        gzip_proxied    any;
        gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

 client_body_buffer_size 256k;
 client_body_in_file_only off;
 client_body_timeout 10s;
 client_header_buffer_size 64k;
## how long a connection has to complete sending
## it's headers for request to be processed
 client_header_timeout  8s;
 client_max_body_size 50m;
 connection_pool_size  512;
 directio  4m;
 ignore_invalid_headers on;
 large_client_header_buffers 8 64k;
 output_buffers   8 256k;
 postpone_output  1460;
 proxy_temp_path  /tmp/nginx_proxy/;
 request_pool_size  32k;
 reset_timedout_connection on;
 send_timeout     15s;
 types_hash_max_size 2048;
 server_names_hash_bucket_size 64;

# for nginx proxy backends to prevent redirects to backend port
# port_in_redirect off;

open_file_cache max=50000 inactive=60s;
open_file_cache_valid 120s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
open_log_file_cache max=10000 inactive=30s min_uses=2;

## limit number of concurrency connections per ip to 16
## add to your server {} section the next line
## limit_conn limit_per_ip 16;
## uncomment below line allows 500K sessions
# limit_conn_log_level error;
#######################################
# use limit_zone for Nginx <v1.1.7 and lower
# limit_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################
# use limit_conn_zone for Nginx >v1.1.8 and higher
# limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################

 include /usr/local/nginx/conf/conf.d/*.conf;
}

Code (Text):

[root@localhost conf.d]# df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root xfs        18G  5.3G   13G  30% /
devtmpfs                devtmpfs  984M     0  984M   0% /dev
tmpfs                   tmpfs     993M     0  993M   0% /dev/shm
tmpfs                   tmpfs     993M  8.5M  985M   1% /run
tmpfs                   tmpfs     993M     0  993M   0% /sys/fs/cgroup
/dev/sda2               xfs       497M  163M  335M  33% /boot
tmpfs                   tmpfs     199M     0  199M   0% /run/user/0

Have made the changes in nginx

Code (Text):

client_max_body_size 200m;
output_buffers   1 512k;
send_timeout     30s;

And have worst speeds (same client and server):

Code (Text):

Length: 882787329 (842M) [application/octet-stream]
Saving to: “/dev/null”

100%[===================================================================================================================================================================================================>] 882,787,329 5.06M/s   in 10m 2s

2017-10-09 01:38:51 (1.40 MB/s) - “/dev/null” saved [882787329/882787329]

Changes reverted and back to <5MBs

About servers dont have problem, for example its a old centos 6.9 on a openvz node. In that node (SAME node) have new installs with problems.

Code (Text):

[root@server /]# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/simfs     simfs      31G   11G   15G  41% /
none           devtmpfs   12G  4.0K   12G   1% /dev
none           tmpfs      12G     0   12G   0% /dev/shm
tmpfs          tmpfs      12G  228K   12G   1% /tmp

Code (Text):

server {
  server_name old.xxxxx.com www.old.xxxxx.com;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  #access_log /home/nginx/domains/old.xxxx.com/log/access.log;
  error_log /home/nginx/domains/old.xxxx.com/log/error.log;

  root /home/nginx/domains/old.xxxx.com/public;

  location / {

   index index.php index.html index.htm;
   try_files $uri $uri/ @kohana; #index.php$uri?$args;
  }
location @kohana {
                rewrite ^/(.+)$ /index.php$request_uri?$args last;
  }
location ~* \.php.*$ {
                fastcgi_pass 127.0.0.1:9000;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param KOHANA_ENV production;
                fastcgi_cache off;
                fastcgi_index index.php;
  }

  #include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;

Code (Text):

user              nginx nginx;
worker_processes 4;
worker_priority -10;

worker_rlimit_nofile 51200;
timer_resolution 100ms;

#error_log         logs/error.log;

pid               logs/nginx.pid;

events {
    worker_connections  2048;
    use epoll;
    #multi_accept on;
}

http {
include /usr/local/nginx/conf/geoip.conf;
include /usr/local/nginx/conf/maintenance.conf;

more_set_headers "X-Powered-By: centminmod";

include /usr/local/nginx/conf/vts_http.conf;
#include /usr/local/nginx/conf/pagespeedadmin.conf;

include /usr/local/nginx/conf/fastcgi_param_https_map.conf;

log_format      main    '$remote_addr - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

#access_log logs/access.log combined buffer=32k;

    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

        sendfile on;
        #sendfile_max_chunk 1m;
        tcp_nopush  on;
        tcp_nodelay off;
        server_tokens off;
        server_name_in_redirect off;

        keepalive_timeout  10;
        keepalive_disable msie6;

        gzip on;
        gzip_vary   on;
        gzip_disable "MSIE [1-6]\.";
        gzip_static on;
        gzip_min_length   1400;
        gzip_buffers      32 8k;
        gzip_http_version 1.0;
        gzip_comp_level 5;
        gzip_proxied    any;
        gzip_types text/plain text/css text/xml application/javascript  application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

 client_body_buffer_size 256k;
 client_body_in_file_only off;
 client_body_timeout 60s;
 client_header_buffer_size 64k;
## how long a connection has to complete sending
## it's headers for request to be processed
 client_header_timeout  20s;
 client_max_body_size 10024k;
 connection_pool_size  512;
 directio  4m;
 ignore_invalid_headers on;
 large_client_header_buffers 8 64k;
 output_buffers   8 256k;
 postpone_output  1460;
 proxy_temp_path  /tmp/nginx_proxy/;
 request_pool_size  32k;
 reset_timedout_connection on;
 send_timeout     60s;
 types_hash_max_size 2048;
 server_names_hash_bucket_size 64;

# for nginx proxy backends to prevent redirects to backend port
# port_in_redirect off;

open_file_cache max=5000 inactive=30s;
open_file_cache_valid 120s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
open_log_file_cache max=1024 inactive=30s min_uses=2;

## limit number of concurrency connections per ip to 16
## add to your server {} section the next line
## limit_conn limit_per_ip 16;
## uncomment below line allows 500K sessions
# limit_conn_log_level error;
#######################################
# use limit_zone for Nginx <v1.1.7 and lower
# limit_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################
# use limit_conn_zone for Nginx >v1.1.8 and higher
# limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################
 include /usr/local/nginx/conf/conf.d/*.conf;
}

This server (from same client) and in same network of the others have a much better speeds (servers are in diffrent countrys but its much better that 5MBs.)

Code (Text):

root@xxxx [~]# wget -v -O /dev/null old.xxxx.com/xxx.xxx
--2017-10-09 01:35:24--  http://old.xxxxx.com/x.xxx
Resolving old.xxxx.com... xxx.xxx.xxx.xxx
Connecting to old.stock-off.com|xxx.xxx.xxx.xxx|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 882787329 (842M) [application/octet-stream]
Saving to: “/dev/null”

100%[===========================================================================================================================================>] 882,787,329 15.2M/s   in 59s

2017-10-09 01:36:25 (14.2 MB/s) - “/dev/null” saved [882787329/882787329]

Now download from same good server from another in same datacenter:

Code (Text):

HTTP request sent, awaiting response... 200 OK
Length: 882787329 (842M) [application/octet-stream]
Saving to: “/dev/null”

100%[=================================================================================================================>] 882,787,329  104M/s   in 8.2s

2017-10-09 01:39:47 (103 MB/s) - “/dev/null” saved [882787329/882787329]

Thanks in advance.

 

Hello,
We have changed client_max_body_size to 1024m and download speed improved a little bit (from 5MBs to 7MBs) but still have more better speeds from old server and old server have a even less limit.

Code (Text):

 client_max_body_size 10024k;

We will install another brand new VM with beta version and back with results soon.

 

Same result with that settings :/