Welcome to Centmin Mod Community
Register Now

Nginx segfaults

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by ehsan, Sep 14, 2015.

  1. ehsan

    ehsan New Member

    9
    2
    3
    Sep 22, 2014
    Ratings:
    +2
    Local Time:
    5:32 PM
    1.9.4
    10.0.21-MariaDB
    Hello
    I know this is not related to centmin and centmin installer script but just hope somebody may have same issue.
    I'm getting segfaults on nginx :
    Code:
    kernel: nginx[2007]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2012]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2002]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2025]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2001]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[1998]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2022]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c190 error 4 in nginx[400000+c19000]
    kernel: nginx[2016]: segfault at 1b8 ip 0000000000527c62 sp 00007fff3187c1f0 error 4 in nginx[400000+c19000]
    I'm on 123.08stable and nginx 1.9.4 , already tried both openssl and libressl , different nginx versions like 1.8.0 or 1.7.7 and no luck, still getting those errors from kernel.
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    12:02 AM
    Nginx 1.13.x
    MariaDB 5.5
    CentOS version ? 32bit or 64bit ? web host ?

    openvz ? kvm ? xen ?

    output from
    Code:
    nginx -V
    wrap output in QUOTE tags

    and
    Code:
    uname -a
    wrap output in CODE tags How to use forum BBCODE code tags | Centmin Mod Community

    also any web sites on server using SSL https ? if so post their SSL nginx vhost domain.com.conf config file contents in CODE tags
     
  3. ehsan

    ehsan New Member

    9
    2
    3
    Sep 22, 2014
    Ratings:
    +2
    Local Time:
    5:32 PM
    1.9.4
    10.0.21-MariaDB
    Sorry for not posting details.
    CentOS release 6.7 which is a vmware esxi 6 vm
    uname output:
    Code:
    Linux xxx.xxx.xxx 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    nginx details:
    Code:
    nginx version: nginx/1.9.4
    built by clang 3.4.2 (tags/RELEASE_34/dot2-final)
    built with OpenSSL 1.0.2d 9 Jul 2015
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -mtune=native -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-const-variable -Wno-conditional-uninitialized -Wno-mismatched-tags -Wno-c++11-extensions -Wno-sometimes-uninitialized -Wno-parentheses-equality -Wno-tautological-compare -Wno-self-assign -Wno-deprecated-register -Wno-deprecated -Wno-invalid-source-encoding -Wno-pointer-sign -Wno-parentheses -Wno-enum-conversion' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module --with-http_secure_link_module --with-http_realip_module --with-http_geoip_module --with-openssl-opt=enable-tlsext --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.3 --add-module=../nginx-accesskey-2.0.3 --add-module=../nginx-http-concat-master --add-module=../openresty-memc-nginx-module-4f6f78f --add-module=../openresty-srcache-nginx-module-ffa9ab7 --add-module=../ngx_devel_kit-0.2.19 --add-module=../set-misc-nginx-module-0.29 --add-module=../echo-nginx-module-0.58 --add-module=../redis2-nginx-module-0.12 --add-module=../ngx_http_redis-0.3.7 --add-module=../lua-nginx-module-0.9.16 --add-module=../lua-upstream-nginx-module-0.03 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --add-module=../nginx-module-vts --add-module=../headers-more-nginx-module-0.261 --with-openssl=../openssl-1.0.2d --with-libatomic --with-threads --with-stream --with-stream_ssl_module --with-pcre=../pcre-8.37 --with-pcre-jit --with-http_spdy_module --add-module=../ngx_pagespeed-release-1.9.32.6-beta
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    12:02 AM
    Nginx 1.13.x
    MariaDB 5.5
    Latest update on September 14, 2015 (just now) to Centmin Mod 1.2.3-eva2000.08 stable (123.08stable) has added Nginx debug mode support. Nginx debug mode is disabled by default.

    To enable, edit centmin.sh and change variable NGINX_DEBUG=n default to NGINX_DEBUG=y and recompile Nginx via centmin.sh menu option 4 and then setting error_log in nginx vhosts to debug as outlined at nginx.org/en/docs/debugging_log.html & wiki.nginx.org/Debugging.

    You do not want to leave Nginx debug mode running forever, so after debugging, set NGINX_DEBUG=n in centmin.sh and recompile Nginx again via centmin.sh menu option 4 and remove error_log debugging mode to disable Nginx debug mode again.

    To update your Centmin Mod builds code for Nginx debug mode support if you do not have a NGINX_DEBUG variable in centmin.sh, follow instructions at centminmod.com/upgrade.html and respective version threads below:

    Centmin Mod is provided as is, but you can try debugging mode for Nginx for further troubleshooting if you have problems with Nginx (i.e. segfaults / signal 11 issues) as outlined at nginx.org/en/docs/debugging_log.html & wiki.nginx.org/Debugging.
     
    • Like Like x 1
  5. ehsan

    ehsan New Member

    9
    2
    3
    Sep 22, 2014
    Ratings:
    +2
    Local Time:
    5:32 PM
    1.9.4
    10.0.21-MariaDB
    Thanks for the debug tip. I will give it a try.
    I'm not using SSL on this server. I searched through some old bugs in nginx website and there were some issues with ssl options. Do you recommend recompiling nginx without ssl ? (and how?)
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    12:02 AM
    Nginx 1.13.x
    MariaDB 5.5
    no no if not using SSL leave that as is
     
  7. ehsan

    ehsan New Member

    9
    2
    3
    Sep 22, 2014
    Ratings:
    +2
    Local Time:
    5:32 PM
    1.9.4
    10.0.21-MariaDB
    Maybe I'm lucky but I updated the centminmod source regarding the debugmod switch and recompiled the nginx, so far so good! I will post if error occurs again.
    Thanks for support anyway.
     
  8. ehsan

    ehsan New Member

    9
    2
    3
    Sep 22, 2014
    Ratings:
    +2
    Local Time:
    5:32 PM
    1.9.4
    10.0.21-MariaDB
    This debug-mode generate huge amount of data for a very busy server with a complicated nginx config. +300mb in 5 mins.
    So could not extract useful data when error happens!
    maybe this:
    Code:
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 close cached open file: /home/proxy_cache/b/0f/4ad84e6cf941ea873aef5c0288ef60fb, fd:210, c:0, u:1, 0
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 run cleanup: 00007FC3CE630228
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 http file cache cleanup
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 http file cache free, fd: 210
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 free: 00007FC3CE62E000, unused: 21328
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 free: 00007FC3CE2ED000
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 hc free: 0000000000000000 0
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 hc busy: 0000000000000000 0
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 reusable connection: 1
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 event timer add: 177: 10000:1442234595317
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 post event 00007FC4D31C4180
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 delete posted event 00007FC4D31C4180
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 http keepalive handler
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 malloc: 00007FC3CE2ED000:65536
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 recv: fd:177 -1 of 65536
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 recv() not ready (11: Resource temporarily unavailable)
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 free: 00007FC3CE2ED000
    2015/09/14 17:13:05 [debug] 27365#27365: *24691 http empty handler
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 event timer del: 177: 1442234595317
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 http keepalive handler
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 close http connection: 177
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 reusable connection: 0
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 free: 0000000000000000
    2015/09/14 17:13:15 [debug] 27365#27365: *24691 free: 00007FC3CE866200, unused: 128
    I am trying to simplify my conf and see what happens.
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    12:02 AM
    Nginx 1.13.x
    MariaDB 5.5