Discover Centmin Mod today
Register Now

Nginx Lot of 502 errors after upgrading to PHP 5.6.4

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Andy, Dec 20, 2014.

  1. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    After upgrading to latest PHP today, my forum got lot of 502 errors
    Code:
    tail -n 100  /var/log/php-fpm/www-error.log
    [20-Dec-2014 03:00:16] WARNING: [pool www] child 26361 exited on signal 6 (SIGABRT) after 0.464584 seconds from start
    [20-Dec-2014 03:00:20] WARNING: [pool www] child 26540 exited on signal 6 (SIGABRT) after 3.603412 seconds from start
    [20-Dec-2014 03:00:23] WARNING: [pool www] child 27937 exited on signal 6 (SIGABRT) after 3.662455 seconds from start
    [20-Dec-2014 03:00:28] WARNING: [pool www] child 29486 exited on signal 6 (SIGABRT) after 4.958942 seconds from start
    [20-Dec-2014 03:00:29] WARNING: [pool www] child 31451 exited on signal 6 (SIGABRT) after 0.386156 seconds from start
    [20-Dec-2014 03:00:40] WARNING: [pool www] child 31604 exited on signal 6 (SIGABRT) after 11.043355 seconds from start
    [20-Dec-2014 03:00:50] WARNING: [pool www] child 3452 exited on signal 6 (SIGABRT) after 10.443477 seconds from start
    [20-Dec-2014 03:00:56] WARNING: [pool www] child 7480 exited on signal 6 (SIGABRT) after 5.702593 seconds from start
    [20-Dec-2014 03:00:57] WARNING: [pool www] child 9665 exited on signal 6 (SIGABRT) after 1.345332 seconds from start
    [20-Dec-2014 03:00:59] WARNING: [pool www] child 10180 exited on signal 6 (SIGABRT) after 1.466874 seconds from start
    [20-Dec-2014 03:01:04] WARNING: [pool www] child 10742 exited on signal 6 (SIGABRT) after 5.184686 seconds from start
    [20-Dec-2014 03:01:06] WARNING: [pool www] child 12759 exited on signal 6 (SIGABRT) after 2.074248 seconds from start
    [20-Dec-2014 03:01:07] WARNING: [pool www] child 13569 exited on signal 6 (SIGABRT) after 1.348228 seconds from start
    [20-Dec-2014 03:01:19] WARNING: [pool www] child 14092 exited on signal 6 (SIGABRT) after 11.464885 seconds from start
    [20-Dec-2014 03:01:27] WARNING: [pool www] child 18537 exited on signal 6 (SIGABRT) after 8.428756 seconds from start
    [20-Dec-2014 03:01:30] WARNING: [pool www] child 21829 exited on signal 6 (SIGABRT) after 2.837490 seconds from start
    [20-Dec-2014 03:01:31] WARNING: [pool www] child 22945 exited on signal 6 (SIGABRT) after 0.451239 seconds from start
    [20-Dec-2014 03:01:32] WARNING: [pool www] child 23121 exited on signal 6 (SIGABRT) after 1.592240 seconds from start
    [20-Dec-2014 03:01:33] WARNING: [pool www] child 23737 exited on signal 6 (SIGABRT) after 0.767898 seconds from start
    [20-Dec-2014 03:01:34] WARNING: [pool www] child 24035 exited on signal 6 (SIGABRT) after 0.918446 seconds from start
    [20-Dec-2014 03:01:34] WARNING: [pool www] child 24390 exited on signal 6 (SIGABRT) after 0.051168 seconds from start
    [20-Dec-2014 03:01:37] WARNING: [pool www] child 24411 exited on signal 6 (SIGABRT) after 2.931439 seconds from start
    [20-Dec-2014 03:01:40] WARNING: [pool www] child 25547 exited on signal 6 (SIGABRT) after 2.834405 seconds from start
    [20-Dec-2014 03:01:40] WARNING: [pool www] child 26644 exited on signal 6 (SIGABRT) after 0.523551 seconds from start
    [20-Dec-2014 03:01:41] WARNING: [pool www] child 26847 exited on signal 6 (SIGABRT) after 0.820849 seconds from start
    [20-Dec-2014 03:01:41] WARNING: [pool www] child 27167 exited on signal 6 (SIGABRT) after 0.108075 seconds from start
    [20-Dec-2014 03:01:44] WARNING: [pool www] child 27209 exited on signal 6 (SIGABRT) after 2.612356 seconds from start
    Got lot of errors in the domain error log
    Code:
    2014/12/20 03:01:44 [error] 19062#0: *362516 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 37.150.45.19, server: www.quantnet.com, request: "GET /forum/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000",

     
  2. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    try downgrading to PHP 5.6.3 and see

    btw, are you using default php ondemand ? or static or dynamic pm in php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf)

    Code:
    pm = ondemand
     
    Last edited: Dec 20, 2014
  3. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    Actually I don't have this kind of error.
    And I really felt PHP 5.6.4 is much faster.
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    well the first set of warnings NOT errors is normal it's just saying php process exited after hitting pm.max_requests which is by default set to 500 to keep php memory management in check. It ain't an error really.

    yeah PHP 5.6.4 seems faster - this forum is running PHP-FPM 5.6.4 right now too
     
  5. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    Yes, downgrading to 5.6.3 and I no longer have those warning/errors.
    Yes, I have pm=ondemand
     
  6. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    Upgraded to 5.6.4 again and see the same errors again
    Code:
    2014/12/20 16:43:41 [error] 32684#0: *48998 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 135.23.111.105, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/mfe-programs-rankings/"
    2014/12/20 16:43:44 [error] 32683#0: *49011 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 103.252.200.65, server: www.quantnet.com, request: "GET /threads/on-line-courses-on-financial-engineering.592/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.google.com.sg/"
    2014/12/20 16:43:47 [error] 32684#0: *48950 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 67.85.109.197, server: www.quantnet.com, request: "POST /deferred.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/level-9.73/?prefix_id=47"
    2014/12/20 16:43:56 [error] 32684#0: *49023 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.145.216.8, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/threads/cqf-entrance-math-test.8955/"
    2014/12/20 16:44:04 [error] 32684#0: *35808 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 58.182.92.47, server: www.quantnet.com, request: "POST /search/latest-posts.json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/"
    2014/12/20 16:44:10 [error] 32683#0: *49034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.236.34.104, server: www.quantnet.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com"
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    is it any better if you change php-fpm.conf from ondemand to static ?
    Code:
    pm = static
     
  8. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    did it and do a nprestart. I will see if it helps.
    Thanks
     
  9. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    May I know your CPU core and threads?
     
  10. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 8
    On-line CPU(s) list: 0-7
    Thread(s) per core: 2
    Core(s) per socket: 4
    Socket(s): 1
    NUMA node(s): 1
    Vendor ID: GenuineIntel
    CPU family: 6
    Model: 60
    Model name: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
    Stepping: 3
    CPU MHz: 799.992
    BogoMIPS: 6584.58
    Virtualization: VT-x
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 256K
    L3 cache: 8192K
    NUMA node0 CPU(s): 0-7
     
  11. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    Also notice that the server load is higher, around 1.10 now. Before, it is never more than 0.50
     
  12. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    Yeah I have this warnings also Hmmm .
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    and downgrading to PHP 5.6.3 doesn't have those errors ?
     
  14. Andy

    Andy Active Member

    312
    45
    28
    Aug 6, 2014
    Ratings:
    +54
    Local Time:
    1:30 AM
    change to pm=dynamic seems to help. I'm still on 5.6.4 and don't see those errors/warning anymore
     
  15. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    I don't want to downgrade :)
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    so could be related to ondemand as PHP-FPM process would exit after completion which could cause 502 as Nginx can no longer communicate with PHP but it could be a bug in PHP 5.6.4 and ondemand as there is no problem with that in prior versions.

    can you guys post your contents of your existing php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf) wrapped in CODE tags or post on gist.github.com etc
     
  17. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    Here's mine:
    Code:
    ; Log level
    ; Possible Values: alert, error, warning, notice, debug
    ; Default Value: notice
    log_level = warning
    pid = /var/run/php-fpm/php-fpm.pid
    error_log = /var/log/php-fpm/www-error.log
    emergency_restart_threshold = 10
    emergency_restart_interval = 1m
    process_control_timeout = 10s
    ;include=/usr/local/nginx/conf/phpfpmd/*.conf
    
    [www]
    user = nginx
    group = nginx
    
    listen = 127.0.0.1:9000
    listen.allowed_clients = 127.0.0.1
    ;listen.backlog = -1
    
    ;listen = /tmp/php5-fpm.sock
    ;listen.owner = nginx
    ;listen.group = nginx
    ;listen.mode = 0666
    
    
    pm = dynamic
    pm.max_children = 16
    pm.start_servers = 6
    pm.min_spare_servers = 2
    pm.max_spare_servers = 10
    pm.max_requests = 200
    
    
    ; PHP 5.3.9 setting
    ; The number of seconds after which an idle process will be killed.
    ; Note: Used only when pm is set to 'ondemand'
    ; Default Value: 10s
    pm.process_idle_timeout = 10s;
    
    rlimit_files = 65536
    rlimit_core = 0
    
    ; The timeout for serving a single request after which the worker process will
    ; be killed. This option should be used when the 'max_execution_time' ini option
    ; does not stop script execution for some reason. A value of '0' means 'off'.
    ; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
    ; Default Value: 0
    ;request_terminate_timeout = 0
    ;request_slowlog_timeout = 0
    slowlog = /var/log/php-fpm/www-slow.log
    
    pm.status_path = /phpstatus
    ping.path = /phpping
    ping.response = pong
    
    ; Limits the extensions of the main script FPM will allow to parse. This can
    ; prevent configuration mistakes on the web server side. You should only limit
    ; FPM to .php extensions to prevent malicious users to use other extensions to
    ; exectute php code.
    ; Note: set an empty value to allow all extensions.
    ; Default Value: .php
    security.limit_extensions = .php .php3 .php4 .php5
    
    ; catch_workers_output = yes
    php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,152
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    3:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    and your error log entries ? a small sample :)
     
  19. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    Opz I already deleted the content.
    Same as andy on 1st post.
     
  20. RoldanLT

    RoldanLT Well-Known Member

    3,899
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    1:30 PM
    1.11
    10.2
    I just change my pm.max_requests to 500 now.