Discover Centmin Mod today
Register Now

Nginx Lot of 502 errors after upgrading to PHP 5.6.4

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Andy, Dec 20, 2014.

  1. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    After upgrading to latest PHP today, my forum got lot of 502 errors
    Code:
    tail -n 100  /var/log/php-fpm/www-error.log
    [20-Dec-2014 03:00:16] WARNING: [pool www] child 26361 exited on signal 6 (SIGABRT) after 0.464584 seconds from start
    [20-Dec-2014 03:00:20] WARNING: [pool www] child 26540 exited on signal 6 (SIGABRT) after 3.603412 seconds from start
    [20-Dec-2014 03:00:23] WARNING: [pool www] child 27937 exited on signal 6 (SIGABRT) after 3.662455 seconds from start
    [20-Dec-2014 03:00:28] WARNING: [pool www] child 29486 exited on signal 6 (SIGABRT) after 4.958942 seconds from start
    [20-Dec-2014 03:00:29] WARNING: [pool www] child 31451 exited on signal 6 (SIGABRT) after 0.386156 seconds from start
    [20-Dec-2014 03:00:40] WARNING: [pool www] child 31604 exited on signal 6 (SIGABRT) after 11.043355 seconds from start
    [20-Dec-2014 03:00:50] WARNING: [pool www] child 3452 exited on signal 6 (SIGABRT) after 10.443477 seconds from start
    [20-Dec-2014 03:00:56] WARNING: [pool www] child 7480 exited on signal 6 (SIGABRT) after 5.702593 seconds from start
    [20-Dec-2014 03:00:57] WARNING: [pool www] child 9665 exited on signal 6 (SIGABRT) after 1.345332 seconds from start
    [20-Dec-2014 03:00:59] WARNING: [pool www] child 10180 exited on signal 6 (SIGABRT) after 1.466874 seconds from start
    [20-Dec-2014 03:01:04] WARNING: [pool www] child 10742 exited on signal 6 (SIGABRT) after 5.184686 seconds from start
    [20-Dec-2014 03:01:06] WARNING: [pool www] child 12759 exited on signal 6 (SIGABRT) after 2.074248 seconds from start
    [20-Dec-2014 03:01:07] WARNING: [pool www] child 13569 exited on signal 6 (SIGABRT) after 1.348228 seconds from start
    [20-Dec-2014 03:01:19] WARNING: [pool www] child 14092 exited on signal 6 (SIGABRT) after 11.464885 seconds from start
    [20-Dec-2014 03:01:27] WARNING: [pool www] child 18537 exited on signal 6 (SIGABRT) after 8.428756 seconds from start
    [20-Dec-2014 03:01:30] WARNING: [pool www] child 21829 exited on signal 6 (SIGABRT) after 2.837490 seconds from start
    [20-Dec-2014 03:01:31] WARNING: [pool www] child 22945 exited on signal 6 (SIGABRT) after 0.451239 seconds from start
    [20-Dec-2014 03:01:32] WARNING: [pool www] child 23121 exited on signal 6 (SIGABRT) after 1.592240 seconds from start
    [20-Dec-2014 03:01:33] WARNING: [pool www] child 23737 exited on signal 6 (SIGABRT) after 0.767898 seconds from start
    [20-Dec-2014 03:01:34] WARNING: [pool www] child 24035 exited on signal 6 (SIGABRT) after 0.918446 seconds from start
    [20-Dec-2014 03:01:34] WARNING: [pool www] child 24390 exited on signal 6 (SIGABRT) after 0.051168 seconds from start
    [20-Dec-2014 03:01:37] WARNING: [pool www] child 24411 exited on signal 6 (SIGABRT) after 2.931439 seconds from start
    [20-Dec-2014 03:01:40] WARNING: [pool www] child 25547 exited on signal 6 (SIGABRT) after 2.834405 seconds from start
    [20-Dec-2014 03:01:40] WARNING: [pool www] child 26644 exited on signal 6 (SIGABRT) after 0.523551 seconds from start
    [20-Dec-2014 03:01:41] WARNING: [pool www] child 26847 exited on signal 6 (SIGABRT) after 0.820849 seconds from start
    [20-Dec-2014 03:01:41] WARNING: [pool www] child 27167 exited on signal 6 (SIGABRT) after 0.108075 seconds from start
    [20-Dec-2014 03:01:44] WARNING: [pool www] child 27209 exited on signal 6 (SIGABRT) after 2.612356 seconds from start
    Got lot of errors in the domain error log
    Code:
    2014/12/20 03:01:44 [error] 19062#0: *362516 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 37.150.45.19, server: www.quantnet.com, request: "GET /forum/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000",

     
  2. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    try downgrading to PHP 5.6.3 and see

    btw, are you using default php ondemand ? or static or dynamic pm in php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf)

    Code:
    pm = ondemand
     
    Last edited: Dec 20, 2014
  3. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    Actually I don't have this kind of error.
    And I really felt PHP 5.6.4 is much faster.
     
  4. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    well the first set of warnings NOT errors is normal it's just saying php process exited after hitting pm.max_requests which is by default set to 500 to keep php memory management in check. It ain't an error really.

    yeah PHP 5.6.4 seems faster - this forum is running PHP-FPM 5.6.4 right now too
     
  5. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    Yes, downgrading to 5.6.3 and I no longer have those warning/errors.
    Yes, I have pm=ondemand
     
  6. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    Upgraded to 5.6.4 again and see the same errors again
    Code:
    2014/12/20 16:43:41 [error] 32684#0: *48998 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 135.23.111.105, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/mfe-programs-rankings/"
    2014/12/20 16:43:44 [error] 32683#0: *49011 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 103.252.200.65, server: www.quantnet.com, request: "GET /threads/on-line-courses-on-financial-engineering.592/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.google.com.sg/"
    2014/12/20 16:43:47 [error] 32684#0: *48950 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 67.85.109.197, server: www.quantnet.com, request: "POST /deferred.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/level-9.73/?prefix_id=47"
    2014/12/20 16:43:56 [error] 32684#0: *49023 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.145.216.8, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/threads/cqf-entrance-math-test.8955/"
    2014/12/20 16:44:04 [error] 32684#0: *35808 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 58.182.92.47, server: www.quantnet.com, request: "POST /search/latest-posts.json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/"
    2014/12/20 16:44:10 [error] 32683#0: *49034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.236.34.104, server: www.quantnet.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com"
     
  7. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    is it any better if you change php-fpm.conf from ondemand to static ?
    Code:
    pm = static
     
  8. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    did it and do a nprestart. I will see if it helps.
    Thanks
     
  9. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    May I know your CPU core and threads?
     
  10. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 8
    On-line CPU(s) list: 0-7
    Thread(s) per core: 2
    Core(s) per socket: 4
    Socket(s): 1
    NUMA node(s): 1
    Vendor ID: GenuineIntel
    CPU family: 6
    Model: 60
    Model name: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
    Stepping: 3
    CPU MHz: 799.992
    BogoMIPS: 6584.58
    Virtualization: VT-x
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 256K
    L3 cache: 8192K
    NUMA node0 CPU(s): 0-7
     
  11. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    Also notice that the server load is higher, around 1.10 now. Before, it is never more than 0.50
     
  12. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    Yeah I have this warnings also Hmmm .
     
  13. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    and downgrading to PHP 5.6.3 doesn't have those errors ?
     
  14. Andy

    Andy Premium Member Premium Member

    420
    60
    28
    Aug 6, 2014
    Ratings:
    +84
    Local Time:
    8:17 AM
    change to pm=dynamic seems to help. I'm still on 5.6.4 and don't see those errors/warning anymore
     
  15. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    I don't want to downgrade :)
     
  16. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    so could be related to ondemand as PHP-FPM process would exit after completion which could cause 502 as Nginx can no longer communicate with PHP but it could be a bug in PHP 5.6.4 and ondemand as there is no problem with that in prior versions.

    can you guys post your contents of your existing php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf) wrapped in CODE tags or post on gist.github.com etc
     
  17. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    Here's mine:
    Code:
    ; Log level
    ; Possible Values: alert, error, warning, notice, debug
    ; Default Value: notice
    log_level = warning
    pid = /var/run/php-fpm/php-fpm.pid
    error_log = /var/log/php-fpm/www-error.log
    emergency_restart_threshold = 10
    emergency_restart_interval = 1m
    process_control_timeout = 10s
    ;include=/usr/local/nginx/conf/phpfpmd/*.conf
    
    [www]
    user = nginx
    group = nginx
    
    listen = 127.0.0.1:9000
    listen.allowed_clients = 127.0.0.1
    ;listen.backlog = -1
    
    ;listen = /tmp/php5-fpm.sock
    ;listen.owner = nginx
    ;listen.group = nginx
    ;listen.mode = 0666
    
    
    pm = dynamic
    pm.max_children = 16
    pm.start_servers = 6
    pm.min_spare_servers = 2
    pm.max_spare_servers = 10
    pm.max_requests = 200
    
    
    ; PHP 5.3.9 setting
    ; The number of seconds after which an idle process will be killed.
    ; Note: Used only when pm is set to 'ondemand'
    ; Default Value: 10s
    pm.process_idle_timeout = 10s;
    
    rlimit_files = 65536
    rlimit_core = 0
    
    ; The timeout for serving a single request after which the worker process will
    ; be killed. This option should be used when the 'max_execution_time' ini option
    ; does not stop script execution for some reason. A value of '0' means 'off'.
    ; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
    ; Default Value: 0
    ;request_terminate_timeout = 0
    ;request_slowlog_timeout = 0
    slowlog = /var/log/php-fpm/www-slow.log
    
    pm.status_path = /phpstatus
    ping.path = /phpping
    ping.response = pong
    
    ; Limits the extensions of the main script FPM will allow to parse. This can
    ; prevent configuration mistakes on the web server side. You should only limit
    ; FPM to .php extensions to prevent malicious users to use other extensions to
    ; exectute php code.
    ; Note: set an empty value to allow all extensions.
    ; Default Value: .php
    security.limit_extensions = .php .php3 .php4 .php5
    
    ; catch_workers_output = yes
    php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
     
  18. eva2000

    eva2000 Administrator Staff Member

    36,456
    7,998
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,323
    Local Time:
    10:17 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    and your error log entries ? a small sample :)
     
  19. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    Opz I already deleted the content.
    Same as andy on 1st post.
     
  20. rdan

    rdan Premium Member Premium Member

    4,312
    1,046
    113
    May 25, 2014
    Ratings:
    +1,507
    Local Time:
    8:17 PM
    Mainline
    10.2
    I just change my pm.max_requests to 500 now.
     
..