Nginx [nginx-announce] nginx-1.9.1

eva2000 · May 27, 2015

Changes with nginx 1.9.1 26 May 2015

*) Change: now SSLv3 protocol is disabled by default.

*) Change: some long deprecated directives are not supported anymore.

*) Feature: the "reuseport" parameter of the "listen" directive.
Thanks to Sepherosa Ziehau and Yingqi Lu.

*) Feature: the $upstream_connect_time variable.

*) Bugfix: in the "hash" directive on big-endian platforms.

*) Bugfix: nginx might fail to start on some old Linux variants; the bug
had appeared in 1.7.11.

*) Bugfix: in IP address parsing.
Thanks to Sergey Polovko.

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx-announce mailing list
nginx-announce@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-announce

Continue reading...

eva2000 · May 27, 2015

sweet been waiting for listen reuseport option for a while now nginx: 4f6efabcb09b

Module ngx_http_core_module

reuseport
this parameter (1.9.1) instructs to create an individual listening socket for each worker process (using theSO_REUSEPORT socket option), allowing a kernel to distribute incoming connections between worker processes more effectively. This currently works only on Linux and DragonFly BSD. Inappropriate use of this option may have its security implications.

SO_REUSEPORT (since Linux 3.9)
Permits multiple AF_INET or AF_INET6 sockets to be bound to an
identical socket address. This option must be set on each
socket (including the first socket) prior to calling bind(2)
on the socket. To prevent port hijacking, all of the
processes binding to the same address must have the same
effective UID. This option can be employed with both TCP and
UDP sockets.

For TCP sockets, this option allows accept(2) load
distribution in a multi-threaded server to be improved by
using a distinct listener socket for each thread. This
provides improved load distribution as compared to traditional
techniques such using a single accept(2)ing thread that
distributes connections, or having multiple threads that
compete to accept(2) from the same socket.

For UDP sockets, the use of this option can provide better
distribution of incoming datagrams to multiple processes (or
threads) as compared to the traditional technique of having
multiple processes compete to receive datagrams on the same
socket.
Click to expand...

Click to expand...

The "reuseport" option of the "listen" directive. When configured, an individual listen socket on a given address is created for each worker process. This allows to reduce in-kernel lock contention on configurations with high accept rates, resulting in better performance. As of now it works on Linux and DragonFly BSD. Note that on Linux incoming connection requests are currently tied up to a specific listen socket, and if some sockets are closed, connection requests will be reset, see The SO_REUSEPORT socket option [LWN.net] With nginx, this may happen if the number of worker processes is reduced. There is no such problem on DragonFly BSD. Based on previous work by Sepherosa Ziehau and Yingqi Lu.
Click to expand...

benchmarks from Tengine The Tengine Web Server

Background
We've implemented the support of SO_REUSEPORT [1] in Tengine. To see the performance improvement, we did a very simple benchmark with four Linux boxes. Three boxes were employed as clients, and the other one as a web server with Tengine listening on port 81 and Nginx listening on port 82. All the hardware specifications of the boxes were the same.

We ran three test cases with concurrency from 100 to 1000. The test cases were:

Tengine with SO_REUSEPORT enabled (reuse_port on).

Nginx with accept lock (accept_mutex on).

Nginx without accept lock (accept_mutex off).

The benchmark software we used was ApacheBench. Here's a command line example:

ab -r -n 10000000 -c 100 http://ip:81/empty.gif
Click to expand...

eva2000 · May 27, 2015

One very long and detailed explanation that covers SO_REUSEPORT at linux - Socket options SO_REUSEADDR and SO_REUSEPORT, how do they differ? Do they mean the same across all major operating systems? - Stack Overflow !

There are a couple of basics you should know before we look at these two options. A TCP/UDP connection is identified by a tuple of five values:

{<protocol>, <src addr>, <src port>, <dest addr>, <dest port>}

Any unique combination of these values identifies a connection. As a result, no two connections can have the same five values, otherwise the system would not be able to distinguish these connections any longer.

The protocol of a socket is set when a socket is created with the socket() function. The source address and port are set with the bind() function. The destination address and port are set with the connect() function. Since UDP is a connectionless protocol, UDP sockets can be used without connecting them. Yet it is allowed to connect them and in some cases very advantageous for your code and general application design. In connectionless mode, UDP sockets that were not explicitly bound when data is sent over them for the first time are usually automatically bound by the system, as an unbound UDP socket cannot receive any (reply) data. Same is true for an unbound TCP socket, it is automatically bound before it will be connected.
Click to expand...

By default, no two sockets can be bound to the same combination of source address and source port. As long as the source port is different, the source address is actually irrelevant. Binding socketA to A:X and socketB to B:Y, where A and B are addresses and X and Y are ports, is always possible as long as X != Y holds true. However, even if X == Y, the binding is still possible as long as A != B holds true. E.g. socketA belongs to a FTP server program and is bound to 192.168.0.1:21 and socketB belongs to another FTP server program and is bound to 10.0.0.1:21, both bindings will succeed. Keep in mind, though, that a socket may be locally bound to "any address". If a socket is bound to 0.0.0.0:21, it is bound to all existing local addresses at the same time and in that case no other socket can be bound to port 21, regardless which specific IP address it tries to bind to, as 0.0.0.0 conflicts with all existing local IP addresses.

Anything said so far is pretty much equal for all major operating system. Things start to get OS specific when address reuse comes into play. We start with BSD, since as I said above, it is the mother of all socket implementations.
Click to expand...

Another article on SO_REUSEPORT Free Programmer's Blog | Linux 3.9 introduced new way of writing socket servers

eva2000 · May 27, 2015

works fine

nginx -V
nginx version: nginx/1.9.1
built by gcc 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)
built with OpenSSL 1.0.2a 19 Mar 2015
TLS SNI support enabled
configure arguments: --with-ld-opt=-Wl,-rpath,/usr/local/lib --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module --with-http_secure_link_module --with-http_flv_module --with-http_realip_module --with-http_geoip_module --with-openssl-opt=enable-tlsext --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.3 --add-module=../headers-more-nginx-module-0.25 --add-module=../nginx-accesskey-2.0.3 --add-module=../nginx-http-concat-master --with-http_dav_module --add-module=../nginx-dav-ext-module-0.0.3 --add-module=../openresty-memc-nginx-module-1518da4 --add-module=../openresty-srcache-nginx-module-ffa9ab7 --add-module=../ngx_devel_kit-0.2.19 --add-module=../set-misc-nginx-module-0.28 --add-module=../echo-nginx-module-0.57 --add-module=../lua-nginx-module-0.9.16rc1 --add-module=../lua-upstream-nginx-module-0.02 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --with-openssl=../openssl-1.0.2a --with-libatomic --with-threads --with-stream --with-stream_ssl_module --with-pcre=../pcre-8.37 --with-pcre-jit --with-http_spdy_module --add-module=../ngx_pagespeed-release-1.9.32.3-beta
Click to expand...

rdan · May 27, 2015

eva2000 said: ↑

reuseport
Click to expand...

Is not possibe to use on Kernel 2.6.32-504.16.2.el6.x86_64 ?

eva2000 · May 27, 2015

RoldanLT said: ↑

Is not possibe to use on Kernel 2.6.32-504.16.2.el6.x86_64 ?
Click to expand...

Redhat/CentOS backported SO_REUSEPORT to 2.6.32

rdan · May 27, 2015

To use?
listen 80 reuseport;

That's it?

eva2000 · May 27, 2015

RoldanLT said: ↑

To use?
listen 80 reuseport;

That's it?
Click to expand...

AFAIK yup

rdan · May 27, 2015

Applying it to only 2 of my domain, I got this error:
Code:
nginx: [emerg] duplicate listen options for 0.0.0.0:80 in /usr/local/nginx/conf/conf.d/mydomain.conf:8
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
My listed line:

listen 80 reuseport;
Click to expand...

Should I use?

listen MY-IP:80 reuseport;
Click to expand...

On every listed directive?

eva2000 · May 27, 2015

reuseport can only use one instance per ip address, listen 80 will default to main ip

so only list reuseport once for entire server if you only have 1 ip address i.e. put it in virtual.conf server default context

eva2000 · May 27, 2015

Just updated a few servers to Nginx 1.9.1 with lua and reuseport enabled

i.e.

H2O HTTP/2 & Nginx SPDY/3.1 SSL World Country Flags Demo

wordpress7.centminmod.com - Centmin Mod LEMP stack Wordpress Auto Installer on CentOS 7.0

rdan · May 27, 2015

eva2000 said: ↑

if you only have 1 ip address i.e. put it in virtual.conf server default context
Click to expand...

And it will automatically apply in all domain hosted?

eva2000 · May 27, 2015

RoldanLT said: ↑

And it will automatically apply in all domain hosted?
Click to expand...

from manual Module ngx_http_core_module

Sets the address and port for IP, or the path for a UNIX-domain socket on which the server will accept requests. Both address and port, or only address or only port can be specified.

The listen directive can have several additional parameters specific to socket-related system calls. These parameters can be specified in any listen directive, but only once for a given address:port pair.
Click to expand...

so it applies per ipaddress:port pair

eva2000 · May 27, 2015

FYI just updated forums to Nginx 1.9.1 with lua support as well as reuseport option enabled

eva2000 · May 28, 2015

Nginx blog also covers SO_REUSEPORT at Socket Sharding in NGINX Release 1.9.1 - NGINX

NGINX release 1.9.1 introduces a new feature that enables use of the SO_REUSEPORT socket option, which is available in newer versions of many operating systems, including DragonFly BSD and Linux (kernel version 3.9 and later). This option allows multiple sockets to listen on the same IP address and port combination. The kernel then load balances incoming connections across the sockets, effectively sharding the socket.

(For NGINX Plus customers, this feature will be available in Release 7, which is scheduled for later this year.)

The SO_REUSEPORT socket option has many real-world applications, such as the potential for easy rolling upgrades of services. For NGINX, it improves performance by more evenly distributing connections across workers.
Click to expand...

beautiful chart for SO_REUSEPORT benchmarks

rdan · May 28, 2015

eva2000 said: ↑

reuseport
Click to expand...

Since enabling this, I experience an increase in CPU usage, from 1-2 % usage now 4% average.

eva2000 · May 28, 2015

RoldanLT said: ↑

Since enabling this, I experience an increase in CPU usage, from 1-2 % usage now 4% average.
Click to expand...

too minor a change i think

rdan · May 28, 2015

eva2000 said: ↑

too minor a change i think
Click to expand...

Yeah, if it helps to speed the the site then why not ? same as ngx_pagespeed which consumes more cpu .

eva2000 · May 28, 2015

i switched to 1.9.1 reuseport about 22 1/2hrs ago and newrelic shows my cpu load average.

far left spike is the nginx recompile

rdan · May 28, 2015

To verify if I enable it correctly.
I have 8 domains on the server with each vhost file and server name.
Just adding the reuseport on single instance of listen 80; directive will make it enable on all domain right?

Not required on the hostname right?
And I also changed my default_server as it defaults into the hostname after install.

Log in / Register

Forums

Join the community today

Nginx [nginx-announce] nginx-1.9.1

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

.

Log in / Register

Useful Searches

Join the community today

Nginx [nginx-announce] nginx-1.9.1

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

.