PHP-FPM newrelic-php extension causing segfault errors in PHP-FPM

Just heads up for folks using newrelic and have the newrelic-php YUM package installed. It may been buggy with latest update causing PHP to segfault (signal 11) and PHP to crash.

I just experienced this myself on this forum only when i tried to search using search box or advanced search. I kept getting 502 bad gateway errors telling me PHP-FPM service was down or crashing and PHP was segfaulting and crashing. This was just after restarting PHP-FPM with latest libssh2 yum security update outlined at Security - Security Vulnerabilitys for libssh2 (CVE-2016-0787) & libssh (CVE-2016-0739) | Centmin Mod Community It seems only search box and admin.php for my forums were giving 502 bad gateway errors.

PHP-FPM error log /var/log/php-fpm/www-error.log showed that my PHP-FPM pools where segfaulting (exited on signal 11 (SIGSEGV))

Code (Text):

[10-Mar-2016 19:29:31] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[10-Mar-2016 19:30:58] WARNING: [pool www] child 18553 exited on signal 11 (SIGSEGV) after 34.715132 seconds from start
[10-Mar-2016 19:31:15] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload

I have newrelic-php extension installed on this forum as I use newrelic for server monitoring.

loaded php extension .ini settings files for forum

Code (Text):

php --ini
Configuration File (php.ini) Path: /usr/local/lib
Loaded Configuration File:         /usr/local/lib/php.ini
Scan for additional .ini files in: /etc/centminmod/php.d
Additional .ini files parsed:      /etc/centminmod/php.d/a_customphp.ini,
/etc/centminmod/php.d/curlcainfo.ini,
/etc/centminmod/php.d/custom_php2.ini,
/etc/centminmod/php.d/geoip.ini,
/etc/centminmod/php.d/igbinary.ini,
/etc/centminmod/php.d/imagick.ini,
/etc/centminmod/php.d/mailparse.ini,
/etc/centminmod/php.d/memcache.ini,
/etc/centminmod/php.d/memcached.ini,
/etc/centminmod/php.d/newrelic.ini,
/etc/centminmod/php.d/redis.ini,
/etc/centminmod/php.d/zendopcache.ini,

I commented out and unloaded newrelic extension by editing /etc/centminmod/php.d/newrelic.ini and placing semi-colon in front of extension load line and restarted PHP-FPM service and 502 bad gateway errors are gone now

Code (Text):

;extension = "newrelic.so"

checking php extensions loaded in php extension directory

Code (Text):

php-config --extension-dir
/usr/local/lib/php/extensions/no-debug-non-zts-20131226

Code (Text):

ls -lah /usr/local/lib/php/extensions/no-debug-non-zts-20131226
total 3.6M
drwxr-xr-x 2 root root 4.0K Mar 10 19:18 .
drwxr-xr-x 5 root root 4.0K Aug 28  2014 ..
-rwxr-xr-x 1 root root  64K Aug 28  2014 geoip.so
-rwxr-xr-x 1 root root 179K May 15  2015 igbinary.so
-rwxr-xr-x 1 root root 1.1M Aug 28  2015 imagick.so
-rwxr-xr-x 1 root root 216K Feb  4 22:17 mailparse.so
-rwxr-xr-x 1 root root 331K Nov 27 02:17 memcached.so
-rwxr-xr-x 1 root root 457K Nov 27 02:16 memcache.so
lrwxrwxrwx 1 root root   53 Mar 10 03:51 newrelic.so -> /usr/lib/newrelic-php5/agent/x64/newrelic-20131226.so
-rwxr-xr-x 1 root root 268K Mar 10 19:18 opcache.a
-rwxr-xr-x 1 root root 167K Mar 10 19:18 opcache.so
-rwxr-xr-x 1 root root 880K May 15  2015 redis.so

rechecking php error log last line entry's date timestamp and current server time, there hasn't been any segfault (signal 11) logged in the past 13 minutes now

Code (Text):

tail -1 /var/log/php-fpm/www-error.log; date
[10-Mar-2016 19:31:50] WARNING: [pool www5] child 18748 exited on signal 11 (SIGSEGV) after 2.840753 seconds from start
Thu Mar 10 19:44:50 UTC 2016

Code (Text):

php -v
PHP 5.6.19 (cli) (built: Mar 10 2016 19:18:28)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Code (Text):

php -m
[PHP Modules]
bcmath
bz2
calendar
Core
ctype
curl
date
dom
enchant
ereg
exif
filter
ftp
gd
geoip
gettext
gmp
hash
iconv
igbinary
imagick
imap
intl
json
libxml
mailparse
mbstring
mcrypt
memcache
memcached
mhash
mysql
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
pspell
readline
redis
Reflection
session
shmop
SimpleXML
snmp
soap
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

Checking yum history list for last 3 yum updates

Code (Text):

yum history list
Loaded plugins: fastestmirror, priorities, security
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   304 | root <root>              | 2016-03-10 18:47 | Update         |    1
   303 | root <root>              | 2016-03-10 03:51 | Update         |    4
   302 | root <root>              | 2016-03-08 04:17 | Update         |    3   

Seems yum transaction id 303 on March 10th, 2016 had a newrelic-php related package update as I use yum-cron to auto update yum packages

Code (Text):

yum history info 303               
Loaded plugins: fastestmirror, priorities, security
Transaction ID : 303
Begin time     : Thu Mar 10 03:51:40 2016
Begin rpmdb    : 652:e71992f2040633d105f5b7676c4030afe3752f32
End time       :            03:51:45 2016 (5 seconds)
End rpmdb      : 652:0142e52c717b7a846bae339e335df69401603551
User           : root <root>
Return-Code    : Success
Command Line   : -x MariaDB* -e 0 -d 0 -y shell /etc/yum/yum-daily.yum
Additional non-default information stored: 1
Transaction performed with:
    Installed     rpm-4.8.0-47.el6.x86_64                       @base
    Installed     yum-3.2.29-69.el6.centos.noarch               @base
    Installed     yum-metadata-parser-1.1.2-16.el6.x86_64       @anaconda-CentOS-201311272149.x86_64/6.5
    Installed     yum-plugin-fastestmirror-1.1.30-30.el6.noarch @base
Packages Altered:
    Updated newrelic-daemon-6.0.0.155-1.x86_64      @newrelic
    Update                  6.0.1.156-1.x86_64      @newrelic
    Updated newrelic-php5-6.0.0.155-1.x86_64        @newrelic
    Update                6.0.1.156-1.x86_64        @newrelic
    Updated newrelic-php5-common-6.0.0.155-1.noarch @newrelic
    Update                       6.0.1.156-1.noarch @newrelic
    Updated nss-util-3.19.1-2.el6_7.x86_64          @updates
    Update           3.19.1-5.el6_7.x86_64          @updates
history info

available newrelic-php modules in yum package

Code (Text):

rpm -ql newrelic-php5
/usr/lib/newrelic-php5/agent/x64
/usr/lib/newrelic-php5/agent/x64/newrelic-20060613-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20060613.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20090626-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20090626.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20100525-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20100525.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20121212-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20121212.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20131226-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20131226.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20151012-zts.so
/usr/lib/newrelic-php5/agent/x64/newrelic-20151012.so
/usr/lib/newrelic-php5/scripts/newrelic-iutil.x64

 

Yeah and report to newrelic tech support the issue too if you can, more folks reporting = faster the fix I guess

 

Looks like other newrelic-php extension users reporting same segfaults on their support forums at PHP Agent - New Relic Community Forum

• PHP Agent 6.0.1.156 Segfault - PHP Agent - New Relic Community Forum
• PHP Agent 6.0.0.155 Segfault - PHP Agent - New Relic Community Forum

One suggestion is to set newrelic.transaction_tracer.detail = 0

my newrelic ini settings config file has it commented out /etc/centminmod/php.d/newrelic.ini

Code (Text):

grep -C1 'newrelic.transaction_tracer.detail' /etc/centminmod/php.d/newrelic.ini
;
; Setting: newrelic.transaction_tracer.detail
; Type   : integer in the range 0-1
--
;
;newrelic.transaction_tracer.detail = 1

--
;          appear in transaction traces. This is only meaningful if you have
;          set newrelic.transaction_tracer.detail to 0. This can be a comma-
;          separated list of function or class method names.

setting it to = 0 doesn't help, still getting 502 bad gateway errors and segfaults in PHP error log

Code (Text):

newrelic.transaction_tracer.detail = 0

from /var/log/php-fpm/www-error.log

Code (Text):

tail -3 /var/log/php-fpm/www-error.log; date
[10-Mar-2016 21:32:38] WARNING: [pool www2] child 3977 exited on signal 11 (SIGSEGV) after 20.625303 seconds from start
[10-Mar-2016 21:32:38] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[10-Mar-2016 21:32:38] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
Thu Mar 10 21:34:55 UTC 2016

 

Submitted my newrelic logs to newrelic tech support. For now you can actually downgrade the newrelic-php yum packages

 

i'd still submit a ticket to newrelic as their devs are asking folks for newrelic debug logs so they can figure out the problem

 

@jeffwidman FYI newrelic-php updated agent apparently fixes segfaults PHP Agent 6.0.0.155 Segfault - PHP Agent - New Relic Community Forum

PHP Agent 6.1.0.157 | New Relic Documentation