Learn about Centmin Mod LEMP Stack today
Register Now

PHP-FPM Upgrading to PHP 7.4 causes downtime

Discussion in 'Install & Upgrades or Pre-Install Questions' started by jcat, May 13, 2020.

  1. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit ?
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: nginx/1.17.10
    • PHP Version Installed: 7.3
    • MariaDB MySQL Version Installed: 10.3.23-MariaDB
    • When was last time updated Centmin Mod code base ? : just before running php upgrade
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      NGXDYNAMIC_BROTLI=y
      NGINX_LIBBROTLI=y
      ZSTD_LOGROTATE_NGINX=y
      ZSTD_LOGROTATE_PHPFPM=y
      MARIADB_INSTALLTENTHREE=y
      PHP_BROTLI=y
      PHP_LZFOUR=y
      PHP_LZF=y
      PHP_ZSTD=y
      PHPFINFO=y
      DISABLE_IPVSIX=y
      LETSENCRYPT_DETECT='y'
      DUALCERTS='y'
      


    Has the compile process changed for PHP 7.4? We noticed everytime now when running an upgrade to PHP 7.4 that PHP fails during the process and remains down for roughly 15 minutes.

    Code:
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: php-fpm.service failed.
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: Unit php-fpm.service entered failed state.
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: Failed to start PHP FastCGI Process Manager.
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: php-fpm.service: main process exited, code=exited, status=127/n/a
    May 13 06:47:26 13156.bigscoots-wpo.com php-fpm[18356]: /usr/local/sbin/php-fpm: error while loading shared libraries: libicuio.so.62: cannot open shared object file: No suc
    h file or directory
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: Starting PHP FastCGI Process Manager...
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: Stopped PHP FastCGI Process Manager.
    May 13 06:47:26 13156.bigscoots-wpo.com systemd[1]: Stopping PHP FastCGI Process Manager...
    
    Any ideas, making it painful to do upgrades throughout the day
     
  2. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    what's output for
    Code (Text):
    yum history list libicu62-devel -q
     
  3. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    also how in sync is your centmin mod 123.09beta01 code you're using as I recall you have modified it for your own needs ? you can try to compare/diff 123.09beta01 official code base at centminmod/centminmod with your copy to see if you missed any changes you need to sync up
     
  4. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Code:
    [23:27][root@13156.bigscoots-wpo.com ~]# yum history list libicu62-devel -q
    ID     | Command line             | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
        46 | -y remove libicu62 --ena | 2020-05-13 06:45 | Erase          |    2
        44 | update --disableplugin=p | 2020-05-13 06:03 | Update         |   21 EE
        29 | -q -y install libicu62 l | 2020-04-28 17:16 | Install        |    2
    
    We don't actually modify the core anymore and that was primarily to modify the wordpress install routine to stop certain plugins from being installed, etc, now we just run a post script which makes adjustments to the install after wordpress is installed.
     
  5. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    looks like yum transaction id 46 removed libicu62

    what does output for below look like ?
    Code (Text):
    yum history info 46


    seems to be related to line 728 of inc/php_configure.inc from 123.09beta01 github sourcegraph search shown here

    It's a routine which is meant for PHP 7.1 and lower to use libicu native v50 by removing libicu62 which is libicu v62. So not sure why it's being triggered for a PHP 7.4 upgrade

    this is what I see on PHP 7.4.5 to 7.4.6 upgrade server only last month April 16 shows such entries as I regularly test PHP 5.6, 7.0 and 7.1 for backported security fix updates
    Code (Text):
    yum history list libicu62-devel -q
    ID     | Command line             | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
       665 | -y remove libicu62 --ena | 2020-04-16 03:00 | Erase          |    2   
       664 | -y swap libicu-devel lib | 2020-04-16 02:19 | E, I           |    2   
       661 | -y remove libicu62 --ena | 2020-04-16 02:03 | Erase          |    2   
       660 | -y swap libicu-devel lib | 2020-04-16 01:50 | E, I           |    2   
       657 | -y remove libicu62 --ena | 2020-04-16 01:29 | Erase          |    2   
       656 | -y swap libicu-devel lib | 2020-04-16 01:26 | E, I           |    3   
       654 | -y remove libicu62 --ena | 2020-04-16 00:47 | Erase          |    4   
       653 | -y swap libicu-devel lib | 2020-04-16 00:15 | E, I           |    2   
       621 | -y remove libicu62 --ena | 2020-01-22 09:46 | Erase          |    9   
       620 | -y swap libicu-devel lib | 2020-01-14 18:39 | E, I           |    2 
    
     
  6. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Code:
    [08:04][root@13156.bigscoots-wpo.com ~]# yum history info 46
    Loaded plugins: fastestmirror, priorities, versionlock
    Transaction ID : 46
    Begin time     : Wed May 13 06:45:50 2020
    Begin rpmdb    : 687:6a2101406bcaa1b28856b70becd4475b614c0023
    End time       :            06:45:51 2020 (1 seconds)
    End rpmdb      : 685:4ac77f53f3dc3ad1fcd1ac2f8d6e3d3973b9678e
    User           : root <root>
    Return-Code    : Success
    Command Line   : -y remove libicu62 --enablerepo=remi
    Transaction performed with:
        Installed     rpm-4.11.3-43.el7.x86_64                      @base
        Installed     yum-3.4.3-167.el7.centos.noarch               @base
        Installed     yum-plugin-fastestmirror-1.1.31-53.el7.noarch @base
    Packages Altered:
        Erase libicu62-62.2-1.el7.remi.x86_64       @remi
        Erase libicu62-devel-62.2-1.el7.remi.x86_64 @remi
    history info
    
    Yeah I was looking at the code as well but based on the comment its affecting all versions it seems as the one I tested on this last time was using PHP 7.3.
     
  7. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    So I just set
    Code:
    PHP_VERSION='7.4.5'
    in
    Code:
    /usr/local/src/centminmod/centmin.sh
    and then upgraded a random server from 7.3 to 7.4.5 and this time it didn't happen

    Code:
    [09:08][root@11426.bigscoots-wpo.com ~]# yum history list libicu62-devel -q
    ID     | Command line             | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
        30 | -q -y install libicu62 l | 2019-10-16 09:40 | Install        |    2
    
     
  8. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    what does transaction 29 look like
    Code (Text):
    yum history info 29

    for server with
    Code (Text):
    [23:27][root@13156.bigscoots-wpo.com ~]# yum history list libicu62-devel -q
    ID     | Command line             | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
       46 | -y remove libicu62 --ena | 2020-05-13 06:45 | Erase          |    2
       44 | update --disableplugin=p | 2020-05-13 06:03 | Update         |   21 EE
       29 | -q -y install libicu62 l | 2020-04-28 17:16 | Install        |    2
    
     
  9. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Code:
    [13:29][root@13156.bigscoots-wpo.com ~]# yum history info 29
    Loaded plugins: fastestmirror, priorities, versionlock
    Transaction ID : 29
    Begin time     : Tue Apr 28 17:16:17 2020
    Begin rpmdb    : 651:f73e17bf29b1836bff0349d8d7f8e75c5bc1dec2
    End time       :            17:16:19 2020 (2 seconds)
    End rpmdb      : 653:3d36bdcbc9de6cef87968d314617f27dd2ad21a0
    User           : root <root>
    Return-Code    : Success
    Command Line   : -q -y install libicu62 libicu62-devel --enablerepo=remi --disablerepo=rpmforge,epel
    Transaction performed with:
        Installed     rpm-4.11.3-43.el7.x86_64                      @base
        Installed     yum-3.4.3-167.el7.centos.noarch               @base
        Installed     yum-plugin-fastestmirror-1.1.31-53.el7.noarch @base
        Installed     yum-plugin-versionlock-1.1.31-53.el7.noarch   @base
    Packages Altered:
        Install libicu62-62.1-3.el7.remi.x86_64       @remi
        Install libicu62-devel-62.1-3.el7.remi.x86_64 @remi
    history info
    
     
  10. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    FYI you do not need to set PHP_VERSION in centmin.sh to update, just run cmupdate and then centmin.sh menu option 5 and specify PHP version = 7.4.5 when prompted
     
  11. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Only time both libicu62 and libicu62-devel are installed at same time is when NEWLIBICU='y' is detected and when /usr/bin/icu-config didn't exist (installed from either libicu-devel or libicu62-devel)
    Code (Text):
    yum -q provides /usr/bin/icu-config --enablerepo=remi
    libicu-devel-50.2-3.el7.x86_64 : Development files for International Components for Unicode
    Repo        : base
    Matched from:
    Filename    : /usr/bin/icu-config
    
    libicu-devel-50.2-4.el7_7.x86_64 : Development files for International Components for Unicode
    Repo        : updates
    Matched from:
    Filename    : /usr/bin/icu-config
    
    libicu62-devel-62.1-3.el7.remi.x86_64 : Development files for International Components for Unicode
    Repo        : remi
    Matched from:
    Filename    : /usr/bin/icu-config
    
    libicu62-devel-62.2-1.el7.remi.x86_64 : Development files for International Components for Unicode
    Repo        : remi
    Matched from:
    Filename    : /usr/bin/icu-config
    
    libicu-devel-50.2-4.el7_7.x86_64 : Development files for International Components for Unicode
    Repo        : @updates
    Matched from:
    Filename    : /usr/bin/icu-config
    

    libicu-devel would of been installed with PHP versions 7.2 and below (NEWLIBICU='n') while libicu62-devel would be installed with PHP versions 7.3 and above (NEWLIBICU='y')

    you can check libicu-devel's yum install history via
    Code (Text):
    yum -q history list libicu-devel

    example
    Code (Text):
    yum -q history list libicu-devel
    ID     | Command line             | Date and time    | Action(s)      | Altered
    -------------------------------------------------------------------------------
       666 | -q -y install libicu-dev | 2020-04-16 03:00 | Install        |    1   
       664 | -y swap libicu-devel lib | 2020-04-16 02:19 | E, I           |    2   
       662 | -q -y install libicu-dev | 2020-04-16 02:03 | Install        |    1   
       660 | -y swap libicu-devel lib | 2020-04-16 01:50 | E, I           |    2   
       658 | -q -y install libicu-dev | 2020-04-16 01:29 | Install        |    1   
       656 | -y swap libicu-devel lib | 2020-04-16 01:26 | E, I           |    3   
       655 | -q -y install libicu-dev | 2020-04-16 00:47 | Install        |    1   
       653 | -y swap libicu-devel lib | 2020-04-16 00:15 | E, I           |    2   
       647 | update --disableplugin=p | 2020-04-01 23:48 | I, U           |   72 EE
       622 | -q -y install libicu-dev | 2020-01-22 09:46 | Install        |    1   
       620 | -y swap libicu-devel lib | 2020-01-14 18:39 | E, I           |    2   
       556 | update --disableplugin=p | 2019-09-19 05:56 | E, I, O, U     |  329 EE
       415 | update --disableplugin=p | 2018-12-25 22:04 | E, I, O, U     |  352 EE
        21 | -q -y install libicu lib | 2017-04-04 07:36 | Install        |    2 
    
     
  12. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Yeah, no we never do, I just figured I test doing that in this situation incase for some reason that was the cause, but you can ignore as I tested on another server but php-fpm still went down this time around.
     
  13. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Guess I am confused what we should be looking for moving forward to prevent the downtime.

    Example:

    Code:
    [13:45][root@12399.bigscoots-wpo.com public]# yum list installed |grep libicu
    libicu62.x86_64                       62.1-3.el7.remi            @remi
    libicu62-devel.x86_64                 62.1-3.el7.remi            @remi
    [13:45][root@12399.bigscoots-wpo.com public]# php -v
    PHP 7.3.14 (cli) (built: Feb  5 2020 19:57:59) ( NTS )
    
    What's needed in this case prior to running the PHP upgrade
     
  14. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Probably need full php upgrade logs to further investigate as per PHP-FPM - How to troubleshoot PHP installs/upgrades, you can zip the log up and sent me a private message attached zipped up log if you want
    should be nothing required and should work - never had a PHP-FPM upgrade with 15 minutes downtime even if libicu library was missing as it would of a temporarily state resolved once it was installed. Only thing i can think of is if yum/network was slow and maybe hung between removal of libicu old devel library package and took 15 minutes to get updated/reinstalled. How long is the total php upgrade time for the servers ? Should be listed at end of centmin.sh menu option 5 runs

    i.e. for PHP 7.4.6 upgrade
    Code (Text):
    Usage: /usr/local/bin/php-config [OPTION]
    Options:
      --prefix            [/usr/local]
      --includes          [-I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib]
      --ldflags           [ -L/usr/local/lib64]
      --libs              [-lcrypt  -lc-client  -ltidy -largon2 -lresolv -lcrypt -lncurses -laspell -lpspell -lrt -lldap -llber -lstdc++ -lcrypt -lpam -lgmp -lbz2 -lrt -lm -ldl  -lsystemd -lxml2 -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lssl -lcrypto -lsqlite3 -lz -lcurl -lxml2 -lenchant -lgmodule-2.0 -lglib-2.0 -lffi -lssl -lcrypto -lz -lpng15 -lwebp -ljpeg -lXpm -lX11 -lfreetype -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lssl -lcrypto -licuio -licui18n -licuuc -licudata -lonig -lsqlite3 -ledit -ltinfo -lxml2 -lnetsnmp -lssl -lssl -lcrypto -lm -lxml2 -lsodium -lcrypt -largon2 -lxml2 -lxml2 -lxml2 -lxml2 -lxslt -lxml2 -lexslt -lxslt -lxml2 -lzip -lz -lssl -lcrypto -lcrypt ]
      --extension-dir     [/usr/local/lib/php/extensions/no-debug-non-zts-20190902]
      --include-dir       [/usr/local/include/php]
      --man-dir           [/usr/local/php/man]
      --php-binary        [/usr/local/bin/php]
      --php-sapis         [ cli embed fpm phpdbg cgi]
      --ini-path          [/usr/local/lib]
      --ini-dir           [/etc/centminmod/php.d]
      --configure-options [--enable-fpm --enable-opcache --enable-intl --enable-pcntl --with-mcrypt --with-snmp --enable-embed=shared --with-mhash --with-zlib --with-gettext --enable-exif --with-zip --with-libzip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --enable-gd --with-xmlrpc --enable-bcmath --enable-calendar --enable-ftp --enable-gd-native-ttf --with-freetype --with-jpeg --with-png-dir=/usr --with-xpm --with-webp --with-t1lib=/usr --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --with-ldap --with-ldap-sasl --with-password-argon2=/usr/local --with-sodium=/usr/local --with-config-file-scan-dir=/etc/centminmod/php.d --with-fpm-systemd --with-ffi --with-xsl PKG_CONFIG_PATH=/opt/rh/devtoolset-9/root/usr/lib64/pkgconfig ICU_CFLAGS=-I/usr/include ICU_LIBS=-L/usr/lib64 -licuio -licui18n -licuuc -licudata LIBSODIUM_CFLAGS=-I/usr/local/include LIBSODIUM_LIBS=-L/usr/local/lib64 -lsodium LIBZIP_CFLAGS=-I/usr/local/include LIBZIP_LIBS=-L/usr/local/lib64 -lzip]
      --version           [7.4.6]
      --vernum            [70406]
    ***************************************************
    * PHP compiled with CPU_MARCH=haswell
    * PHP upgraded and auto reinstalled PHP extensions
    ***************************************************
       ____   _   _  ____    _   _                                _            _ 
      |  _ \ | | | ||  _ \  | | | | _ __    __ _  _ __  __ _   __| |  ___   __| |
      | |_) || |_| || |_) | | | | || '_ \  / _` || '__|/ _` | / _` | / _ \ / _` |
      |  __/ |  _  ||  __/  | |_| || |_) || (_| || |  | (_| || (_| ||  __/| (_| |
      |_|    |_| |_||_|      \___/ | .__/  \__, ||_|   \__,_| \__,_| \___| \__,_|
                                   |_|     |___/                                 
    Total PHP Upgrade Time: 146.050652049 seconds
    

    where
    Code (Text):
    Total PHP Upgrade Time: 146.050652049 seconds
     
  15. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    On other think you can try to see if it's related is edit /usr/lib/systemd/system/php-fpm.service change TYPE=notify to TYPE=forking and then run
    Code (Text):
    systemctl daemon-reload
    fpmrestart

    then try PHP 7.4.5 or 7.4.6 update
     
  16. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  17. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Ran several upgrades, all look to work great! That commit definitely seems to of fixed it.

    Also, sorry if this is super picky but can reduce 502's while people are on the site if they are not necessary, there are 3 PHP-FPM restarts instead of reloads:

    Code:
    Copying mailparse.ini > /etc/centminmod/php.d/mailparse.ini
    
    service php-fpm restart
    
    Code:
    Copying imagick.ini > /etc/centminmod/php.d/imagick.ini
    
    service php-fpm restart
    
    Are these necessary during the upgrade or would reload's work just the same? The 3rd restart happens at the very end which seems fine.
     
  18. pamamolf

    pamamolf Premium Member Premium Member

    3,774
    366
    83
    May 31, 2014
    Ratings:
    +706
    Local Time:
    5:25 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Any optimizations if possible for less restarts on Nginx and Php upgrades will be great !!!
     
  19. eva2000

    eva2000 Administrator Staff Member

    44,186
    10,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,572
    Local Time:
    12:25 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    The restarts I removed in this latest update are only related to php-fpm configure and make compilation stages. But some php extension's restarts are required as it's need in php-fpm routine to detect if PHP extensions are loaded (if/else logic so do XX if php extension loaded or do YY if php extension isn't loaded) and that only happens after a restart. Will see if there's a better way but will still need to keep a way of detecting if a php extension has been loaded after it's been installed. Note loaded and installed aren't the same thing, a php extension can be installed and be either loaded or not loaded.
     
  20. jcat

    jcat Member

    125
    18
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +48
    Local Time:
    10:25 AM
    Yeah makes sense, I was trying to determine that last night in terms of when a restart / reload is truly needed. I know you and your magical powers will find a way :D