/ Register

Welcome to Centmin Mod Community
Register Now

Nginx Lot of 502 errors after upgrading to PHP 5.6.4

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Andy, Dec 20, 2014.

Tags:
Previous Thread Next Thread
Loading...
Page 1 of 4
  1. Dec 20, 2014 #1
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    After upgrading to latest PHP today, my forum got lot of 502 errors
    Code:
    tail -n 100  /var/log/php-fpm/www-error.log
[20-Dec-2014 03:00:16] WARNING: [pool www] child 26361 exited on signal 6 (SIGABRT) after 0.464584 seconds from start
[20-Dec-2014 03:00:20] WARNING: [pool www] child 26540 exited on signal 6 (SIGABRT) after 3.603412 seconds from start
[20-Dec-2014 03:00:23] WARNING: [pool www] child 27937 exited on signal 6 (SIGABRT) after 3.662455 seconds from start
[20-Dec-2014 03:00:28] WARNING: [pool www] child 29486 exited on signal 6 (SIGABRT) after 4.958942 seconds from start
[20-Dec-2014 03:00:29] WARNING: [pool www] child 31451 exited on signal 6 (SIGABRT) after 0.386156 seconds from start
[20-Dec-2014 03:00:40] WARNING: [pool www] child 31604 exited on signal 6 (SIGABRT) after 11.043355 seconds from start
[20-Dec-2014 03:00:50] WARNING: [pool www] child 3452 exited on signal 6 (SIGABRT) after 10.443477 seconds from start
[20-Dec-2014 03:00:56] WARNING: [pool www] child 7480 exited on signal 6 (SIGABRT) after 5.702593 seconds from start
[20-Dec-2014 03:00:57] WARNING: [pool www] child 9665 exited on signal 6 (SIGABRT) after 1.345332 seconds from start
[20-Dec-2014 03:00:59] WARNING: [pool www] child 10180 exited on signal 6 (SIGABRT) after 1.466874 seconds from start
[20-Dec-2014 03:01:04] WARNING: [pool www] child 10742 exited on signal 6 (SIGABRT) after 5.184686 seconds from start
[20-Dec-2014 03:01:06] WARNING: [pool www] child 12759 exited on signal 6 (SIGABRT) after 2.074248 seconds from start
[20-Dec-2014 03:01:07] WARNING: [pool www] child 13569 exited on signal 6 (SIGABRT) after 1.348228 seconds from start
[20-Dec-2014 03:01:19] WARNING: [pool www] child 14092 exited on signal 6 (SIGABRT) after 11.464885 seconds from start
[20-Dec-2014 03:01:27] WARNING: [pool www] child 18537 exited on signal 6 (SIGABRT) after 8.428756 seconds from start
[20-Dec-2014 03:01:30] WARNING: [pool www] child 21829 exited on signal 6 (SIGABRT) after 2.837490 seconds from start
[20-Dec-2014 03:01:31] WARNING: [pool www] child 22945 exited on signal 6 (SIGABRT) after 0.451239 seconds from start
[20-Dec-2014 03:01:32] WARNING: [pool www] child 23121 exited on signal 6 (SIGABRT) after 1.592240 seconds from start
[20-Dec-2014 03:01:33] WARNING: [pool www] child 23737 exited on signal 6 (SIGABRT) after 0.767898 seconds from start
[20-Dec-2014 03:01:34] WARNING: [pool www] child 24035 exited on signal 6 (SIGABRT) after 0.918446 seconds from start
[20-Dec-2014 03:01:34] WARNING: [pool www] child 24390 exited on signal 6 (SIGABRT) after 0.051168 seconds from start
[20-Dec-2014 03:01:37] WARNING: [pool www] child 24411 exited on signal 6 (SIGABRT) after 2.931439 seconds from start
[20-Dec-2014 03:01:40] WARNING: [pool www] child 25547 exited on signal 6 (SIGABRT) after 2.834405 seconds from start
[20-Dec-2014 03:01:40] WARNING: [pool www] child 26644 exited on signal 6 (SIGABRT) after 0.523551 seconds from start
[20-Dec-2014 03:01:41] WARNING: [pool www] child 26847 exited on signal 6 (SIGABRT) after 0.820849 seconds from start
[20-Dec-2014 03:01:41] WARNING: [pool www] child 27167 exited on signal 6 (SIGABRT) after 0.108075 seconds from start
[20-Dec-2014 03:01:44] WARNING: [pool www] child 27209 exited on signal 6 (SIGABRT) after 2.612356 seconds from start
    Got lot of errors in the domain error log
    Code:
    2014/12/20 03:01:44 [error] 19062#0: *362516 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 37.150.45.19, server: www.quantnet.com, request: "GET /forum/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000",

     
  2. Dec 20, 2014 #2
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    try downgrading to PHP 5.6.3 and see

    btw, are you using default php ondemand ? or static or dynamic pm in php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf)

    Code:
    pm = ondemand
     
    Last edited: Dec 20, 2014
  3. Dec 20, 2014 #3
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    Actually I don't have this kind of error.
    And I really felt PHP 5.6.4 is much faster.
     
  4. Dec 20, 2014 #4
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    well the first set of warnings NOT errors is normal it's just saying php process exited after hitting pm.max_requests which is by default set to 500 to keep php memory management in check. It ain't an error really.

    yeah PHP 5.6.4 seems faster - this forum is running PHP-FPM 5.6.4 right now too
     
  5. Dec 21, 2014 #5
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    Yes, downgrading to 5.6.3 and I no longer have those warning/errors.
    Yes, I have pm=ondemand
     
  6. Dec 21, 2014 #6
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    Upgraded to 5.6.4 again and see the same errors again
    Code:
    2014/12/20 16:43:41 [error] 32684#0: *48998 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 135.23.111.105, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/mfe-programs-rankings/"
2014/12/20 16:43:44 [error] 32683#0: *49011 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 103.252.200.65, server: www.quantnet.com, request: "GET /threads/on-line-courses-on-financial-engineering.592/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.google.com.sg/"
2014/12/20 16:43:47 [error] 32684#0: *48950 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 67.85.109.197, server: www.quantnet.com, request: "POST /deferred.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/level-9.73/?prefix_id=47"
2014/12/20 16:43:56 [error] 32684#0: *49023 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.145.216.8, server: www.quantnet.com, request: "GET /css.php?css=uix,uiflex,EXTRA&style=29&dir=LTR&d=1419002755 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/threads/cqf-entrance-math-test.8955/"
2014/12/20 16:44:04 [error] 32684#0: *35808 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 58.182.92.47, server: www.quantnet.com, request: "POST /search/latest-posts.json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com", referrer: "https://www.quantnet.com/forum/"
2014/12/20 16:44:10 [error] 32683#0: *49034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 82.236.34.104, server: www.quantnet.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.quantnet.com"
     
  7. Dec 21, 2014 #7
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    is it any better if you change php-fpm.conf from ondemand to static ?
    Code:
    pm = static
     
  8. Dec 21, 2014 #8
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    did it and do a nprestart. I will see if it helps.
    Thanks
     
  9. Dec 21, 2014 #9
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    May I know your CPU core and threads?
     
  10. Dec 21, 2014 #10
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 8
    On-line CPU(s) list: 0-7
    Thread(s) per core: 2
    Core(s) per socket: 4
    Socket(s): 1
    NUMA node(s): 1
    Vendor ID: GenuineIntel
    CPU family: 6
    Model: 60
    Model name: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
    Stepping: 3
    CPU MHz: 799.992
    BogoMIPS: 6584.58
    Virtualization: VT-x
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 256K
    L3 cache: 8192K
    NUMA node0 CPU(s): 0-7
     
  11. Dec 21, 2014 #11
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    Also notice that the server load is higher, around 1.10 now. Before, it is never more than 0.50
     
  12. Dec 22, 2014 #12
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    Yeah I have this warnings also Hmmm .
     
  13. Dec 22, 2014 #13
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    and downgrading to PHP 5.6.3 doesn't have those errors ?
     
  14. Dec 22, 2014 #14
    Andy

    Andy Active Member

    540
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    10:02 PM
    change to pm=dynamic seems to help. I'm still on 5.6.4 and don't see those errors/warning anymore
     
  15. Dec 22, 2014 #15
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    I don't want to downgrade :)
     
  16. Dec 22, 2014 #16
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    so could be related to ondemand as PHP-FPM process would exit after completion which could cause 502 as Nginx can no longer communicate with PHP but it could be a bug in PHP 5.6.4 and ondemand as there is no problem with that in prior versions.

    can you guys post your contents of your existing php-fpm.conf at /usr/local/etc/php-fpm.conf (command shortcut = fpmconf) wrapped in CODE tags or post on gist.github.com etc
     
  17. Dec 22, 2014 #17
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    Here's mine:
    Code:
    ; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = warning
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/www-error.log
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
;include=/usr/local/nginx/conf/phpfpmd/*.conf

[www]
user = nginx
group = nginx

listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
;listen.backlog = -1

;listen = /tmp/php5-fpm.sock
;listen.owner = nginx
;listen.group = nginx
;listen.mode = 0666


pm = dynamic
pm.max_children = 16
pm.start_servers = 6
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 200


; PHP 5.3.9 setting
; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
pm.process_idle_timeout = 10s;

rlimit_files = 65536
rlimit_core = 0

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0
;request_slowlog_timeout = 0
slowlog = /var/log/php-fpm/www-slow.log

pm.status_path = /phpstatus
ping.path = /phpping
ping.response = pong

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
security.limit_extensions = .php .php3 .php4 .php5

; catch_workers_output = yes
php_admin_value[error_log] = /var/log/php-fpm/www-php.error.log
     
  18. Dec 22, 2014 #18
    eva2000

    eva2000 Administrator Staff Member

    53,554
    12,135
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,678
    Local Time:
    1:02 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    and your error log entries ? a small sample :)
     
  19. Dec 22, 2014 #19
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    Opz I already deleted the content.
    Same as andy on 1st post.
     
  20. Dec 22, 2014 #20
    rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    11:02 AM
    Mainline
    10.2
    I just change my pm.max_requests to 500 now.
     
Previous Thread Next Thread
Loading...
Page 1 of 4

.