Discover Centmin Mod today
Register Now

PHP-FPM newrelic-php extension causing segfault errors in PHP-FPM

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by eva2000, Mar 11, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    Just heads up for folks using newrelic and have the newrelic-php YUM package installed. It may been buggy with latest update causing PHP to segfault (signal 11) and PHP to crash.

    I just experienced this myself on this forum only when i tried to search using search box or advanced search. I kept getting 502 bad gateway errors telling me PHP-FPM service was down or crashing and PHP was segfaulting and crashing. This was just after restarting PHP-FPM with latest libssh2 yum security update outlined at Security - Security Vulnerabilitys for libssh2 (CVE-2016-0787) & libssh (CVE-2016-0739) | Centmin Mod Community It seems only search box and admin.php for my forums were giving 502 bad gateway errors.

    PHP-FPM error log /var/log/php-fpm/www-error.log showed that my PHP-FPM pools where segfaulting (exited on signal 11 (SIGSEGV))
    Code (Text):
    [10-Mar-2016 19:29:31] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
    [10-Mar-2016 19:30:58] WARNING: [pool www] child 18553 exited on signal 11 (SIGSEGV) after 34.715132 seconds from start
    [10-Mar-2016 19:31:15] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload


    I have newrelic-php extension installed on this forum as I use newrelic for server monitoring.

    loaded php extension .ini settings files for forum
    Code (Text):
    php --ini
    Configuration File (php.ini) Path: /usr/local/lib
    Loaded Configuration File:         /usr/local/lib/php.ini
    Scan for additional .ini files in: /etc/centminmod/php.d
    Additional .ini files parsed:      /etc/centminmod/php.d/a_customphp.ini,
    /etc/centminmod/php.d/curlcainfo.ini,
    /etc/centminmod/php.d/custom_php2.ini,
    /etc/centminmod/php.d/geoip.ini,
    /etc/centminmod/php.d/igbinary.ini,
    /etc/centminmod/php.d/imagick.ini,
    /etc/centminmod/php.d/mailparse.ini,
    /etc/centminmod/php.d/memcache.ini,
    /etc/centminmod/php.d/memcached.ini,
    /etc/centminmod/php.d/newrelic.ini,
    /etc/centminmod/php.d/redis.ini,
    /etc/centminmod/php.d/zendopcache.ini,


    I commented out and unloaded newrelic extension by editing /etc/centminmod/php.d/newrelic.ini and placing semi-colon in front of extension load line and restarted PHP-FPM service and 502 bad gateway errors are gone now
    Code (Text):
    ;extension = "newrelic.so"

    checking php extensions loaded in php extension directory
    Code (Text):
    php-config --extension-dir
    /usr/local/lib/php/extensions/no-debug-non-zts-20131226
    

    Code (Text):
    ls -lah /usr/local/lib/php/extensions/no-debug-non-zts-20131226
    total 3.6M
    drwxr-xr-x 2 root root 4.0K Mar 10 19:18 .
    drwxr-xr-x 5 root root 4.0K Aug 28  2014 ..
    -rwxr-xr-x 1 root root  64K Aug 28  2014 geoip.so
    -rwxr-xr-x 1 root root 179K May 15  2015 igbinary.so
    -rwxr-xr-x 1 root root 1.1M Aug 28  2015 imagick.so
    -rwxr-xr-x 1 root root 216K Feb  4 22:17 mailparse.so
    -rwxr-xr-x 1 root root 331K Nov 27 02:17 memcached.so
    -rwxr-xr-x 1 root root 457K Nov 27 02:16 memcache.so
    lrwxrwxrwx 1 root root   53 Mar 10 03:51 newrelic.so -> /usr/lib/newrelic-php5/agent/x64/newrelic-20131226.so
    -rwxr-xr-x 1 root root 268K Mar 10 19:18 opcache.a
    -rwxr-xr-x 1 root root 167K Mar 10 19:18 opcache.so
    -rwxr-xr-x 1 root root 880K May 15  2015 redis.so
    


    rechecking php error log last line entry's date timestamp and current server time, there hasn't been any segfault (signal 11) logged in the past 13 minutes now
    Code (Text):
    tail -1 /var/log/php-fpm/www-error.log; date
    [10-Mar-2016 19:31:50] WARNING: [pool www5] child 18748 exited on signal 11 (SIGSEGV) after 2.840753 seconds from start
    Thu Mar 10 19:44:50 UTC 2016


    Code (Text):
    php -v
    PHP 5.6.19 (cli) (built: Mar 10 2016 19:18:28)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    

    Code (Text):
    php -m
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    libxml
    mailparse
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    
    


    Checking yum history list for last 3 yum updates
    Code (Text):
    yum history list
    Loaded plugins: fastestmirror, priorities, security
    ID     | Login user               | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
       304 | root <root>              | 2016-03-10 18:47 | Update         |    1
       303 | root <root>              | 2016-03-10 03:51 | Update         |    4
       302 | root <root>              | 2016-03-08 04:17 | Update         |    3   


    Seems yum transaction id 303 on March 10th, 2016 had a newrelic-php related package update as I use yum-cron to auto update yum packages
    Code (Text):
    yum history info 303               
    Loaded plugins: fastestmirror, priorities, security
    Transaction ID : 303
    Begin time     : Thu Mar 10 03:51:40 2016
    Begin rpmdb    : 652:e71992f2040633d105f5b7676c4030afe3752f32
    End time       :            03:51:45 2016 (5 seconds)
    End rpmdb      : 652:0142e52c717b7a846bae339e335df69401603551
    User           : root <root>
    Return-Code    : Success
    Command Line   : -x MariaDB* -e 0 -d 0 -y shell /etc/yum/yum-daily.yum
    Additional non-default information stored: 1
    Transaction performed with:
        Installed     rpm-4.8.0-47.el6.x86_64                       @base
        Installed     yum-3.2.29-69.el6.centos.noarch               @base
        Installed     yum-metadata-parser-1.1.2-16.el6.x86_64       @anaconda-CentOS-201311272149.x86_64/6.5
        Installed     yum-plugin-fastestmirror-1.1.30-30.el6.noarch @base
    Packages Altered:
        Updated newrelic-daemon-6.0.0.155-1.x86_64      @newrelic
        Update                  6.0.1.156-1.x86_64      @newrelic
        Updated newrelic-php5-6.0.0.155-1.x86_64        @newrelic
        Update                6.0.1.156-1.x86_64        @newrelic
        Updated newrelic-php5-common-6.0.0.155-1.noarch @newrelic
        Update                       6.0.1.156-1.noarch @newrelic
        Updated nss-util-3.19.1-2.el6_7.x86_64          @updates
        Update           3.19.1-5.el6_7.x86_64          @updates
    history info


    available newrelic-php modules in yum package
    Code (Text):
    rpm -ql newrelic-php5
    /usr/lib/newrelic-php5/agent/x64
    /usr/lib/newrelic-php5/agent/x64/newrelic-20060613-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20060613.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20090626-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20090626.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20100525-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20100525.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20121212-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20121212.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20131226-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20131226.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20151012-zts.so
    /usr/lib/newrelic-php5/agent/x64/newrelic-20151012.so
    /usr/lib/newrelic-php5/scripts/newrelic-iutil.x64
    

     
    Last edited: Mar 13, 2016
  2. pamamolf

    pamamolf Well-Known Member

    2,533
    231
    63
    May 31, 2014
    Ratings:
    +394
    Local Time:
    9:09 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    So we disable it and we wait for a fix from newrelic ?
     
  3. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    Yeah and report to newrelic tech support the issue too if you can, more folks reporting = faster the fix I guess :)
     
    • Agree Agree x 1
  4. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    Looks like other newrelic-php extension users reporting same segfaults on their support forums at PHP Agent - New Relic Community Forum
    One suggestion is to set newrelic.transaction_tracer.detail = 0

    my newrelic ini settings config file has it commented out /etc/centminmod/php.d/newrelic.ini
    Code (Text):
    grep -C1 'newrelic.transaction_tracer.detail' /etc/centminmod/php.d/newrelic.ini
    ;
    ; Setting: newrelic.transaction_tracer.detail
    ; Type   : integer in the range 0-1
    --
    ;
    ;newrelic.transaction_tracer.detail = 1
    
    --
    ;          appear in transaction traces. This is only meaningful if you have
    ;          set newrelic.transaction_tracer.detail to 0. This can be a comma-
    ;          separated list of function or class method names.


    setting it to = 0 doesn't help, still getting 502 bad gateway errors and segfaults in PHP error log
    Code (Text):
    newrelic.transaction_tracer.detail = 0

    from /var/log/php-fpm/www-error.log
    Code (Text):
    tail -3 /var/log/php-fpm/www-error.log; date
    [10-Mar-2016 21:32:38] WARNING: [pool www2] child 3977 exited on signal 11 (SIGSEGV) after 20.625303 seconds from start
    [10-Mar-2016 21:32:38] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
    [10-Mar-2016 21:32:38] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
    Thu Mar 10 21:34:55 UTC 2016
     
    Last edited: Mar 11, 2016
  5. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    Submitted my newrelic logs to newrelic tech support. For now you can actually downgrade the newrelic-php yum packages

     
    • Informative Informative x 1
  6. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    11:09 PM
    Alas, I thought my sites were working fine with this new version of the NR agent, but turns out a few of them have specific pages that are crashing and I had to revert. Thanks for posting about this George--it saved me a ton of time debugging when I started getting the error reports.
     
    • Like Like x 1
  7. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    i'd still submit a ticket to newrelic as their devs are asking folks for newrelic debug logs so they can figure out the problem
     
  8. eva2000

    eva2000 Administrator Staff Member

    29,039
    6,589
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,784
    Local Time:
    4:09 PM
    Nginx 1.13.x
    MariaDB 5.5
    @jeffwidman FYI newrelic-php updated agent apparently fixes segfaults PHP Agent 6.0.0.155 Segfault - PHP Agent - New Relic Community Forum

    PHP Agent 6.1.0.157 | New Relic Documentation
     
    • Like Like x 1