PHP-FPM Strange PHP-FPM problem after PHP rebuild

We have a website with an old XenForo 1 installation with Centminmod 130.00beta01 at a dedicated Rocky 8.10 server (only Centminmod with Nginx 1.25.4 and PHP-FPMD 7.2.34 running beside Redis and Memcached). That setup runs smoothly without problems since years with occasional updates for Nginx and recompile of PHP 7.2.34 using Centminmod.

Last week we decided to upgrade the server to Centminmod 140.00beta01 which could be done without any problems. We also upgraded Nginx to 1.27.1, also without any problems.

But after we recompiled PHP 7.2.34 (the same version we used since many years) to catch any new patches (we did that without any problems several times before), the server began to make weird things. It started to raise CPU usage exponentially until the server breaks down. After a restart of the server we recognised that some (not all) XenForo thread and gallery pages could not finish to be shown at the visitors browser and timed out at PHP-FPM with one of the the following 2 errors:

Code (Text):

Maximum execution time of 60 seconds exceeded in /library/XenForo/Template/Abstract.php on line 366

Code (Text):

Maximum execution time of 60 seconds exceeded in /library/XenForo/Helper/String.php on line 519

Some thread and gallery pages show without delay and some of them end with that error. The server builds up load until the CPU gets overwhelmed and the server crashes.

We tried to find the bug without any success. We tried to upgrade PHP 7.2.34 to PHP 7.3.33 but the problem persists with exactly the same behaviour.

Only a downgrade to PHP 7.1.33 solved the problem and Xenforo runs without problems again.

We could not find the exact reason for our problems. It seems that the recompile of PHP 7.2.34 under Centminmod 140.00beta01 introduced some kind of incompatibility with XenForo 1. Perhaps an updated module or a newly introduced patch? Because the upgrade to PHP 7.3.33 did not solve the issue whereas the downgrade to PHP 7.1.33 does the incompatible module or patch seems to be applied to PHP 7.2 upwards only.

Does anyone has any clue or a tip what the problem could have been?
@eva2000 perhaps?

Thanks for any help.

The relevant custom config vars for our installation, all the installed PHP modules are Centminmod standard:

Code (Text):

PHP_PGO='y'
MARCH_TARGETNATIVE='n'
PHPFINFO='y'

---

 

Thank you. I'll try that within the next maintenance window and report if successful.

 

Coming back to this thread after a while with no work at that server.

I now tried to recompile PHP 7.2.34 at that server (centminmod beta 140.0 current), using "PHP_PGO='n' as recommended by you. It failed to compile.

I also tried with "DEVTOOLSETTWELVE='y' && DEVTOOLSETTHIRTEEN='n'" with no luck and resulting in the same error.

Here is a part of the error log:

Code (Text):

Using GCC 8

export CFLAGS="-O3 -m64 -march=x86-64-v3 -pipe -Wno-deprecated-declarations -fstack-protector-strong -fPIC"
export CPPFLAGS=" -D_FORTIFY_SOURCE=2"
export ICU_CFLAGS='-fPIC'
export LIBSODIUM_CFLAGS='-fPIC'
export LIBZIP_CFLAGS="-I/opt/remi/libzip/include -fPIC"
export PHP_LDFLAGS=" -Wl,-z,relro,-z,now -pie -Wl,-rpath,/opt/remi/libzip/lib64"
2
LD_LIBRARY_PATH=/opt/remi/libzip/lib64:/usr/local/lib64 PKG_CONFIG_PATH="/opt/remi/libzip/lib64/pkgconfig:/usr/local/lib64/pkgconfig:" ../configure --enable-fpm --enable-opcache --enable-intl --enable-pcntl --with-mcrypt --with-snmp --enable-embed=shared --with-mhash --with-zlib --with-gettext --enable-exif --enable-zip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --with-gd --with-xmlrpc --enable-bcmath --enable-calendar  --enable-ftp --enable-gd-native-ttf --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr --with-xpm-dir=/usr --with-webp-dir=/usr --with-t1lib=/usr --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --with-ldap --with-ldap-sasl --with-pic --with-config-file-scan-dir=/etc/centminmod/php.d --with-fpm-systemd --with-xsl
configure: WARNING: unrecognized options: --with-mcrypt, --with-mysql, --enable-gd-native-ttf, --with-t1lib, --with-mysql
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
../build/shtool: line 509: warning: command substitution: ignored null byte in input
../build/shtool: line 510: warning: command substitution: ignored null byte in input
../build/shtool: line 509: warning: command substitution: ignored null byte in input
../build/shtool: line 510: warning: command substitution: ignored null byte in input
checking whether the C compiler works... no
configure: error: in `/svr-setup/php-7.2.34/fpm-build':
configure: error: C compiler cannot create executables
See `config.log' for more details
sed: can't read Makefile: No such file or directory
sed: can't read Makefile: No such file or directory

Sun Nov 10 10:20:36 UTC 2024
Error: , PHP configure failed (140.00beta01.b158)

 

This was the first attempt since then. Nothing done at the server since then except OS and centminmod updates.

Noticed that as well. Right before trying to update PHP I updated Nginx to 1.27.2 successfully.

Nothing in custom_config except the settings mentioned. At first try "PHP_PGO" was off and at second try "DEVTOOLSETTWELVE" was on and "DEVTOOLSETTHIRTEEN" was off.

GCC 8 seems to be the default compiler at Alma 8.

 

Sent via PN.

 

Coming back to my very strange problem from the first post in this thread.

I setup a copy of the production server with that problem as a test server and started a research mission on why it was not possible to install a fully functional PHP 7.2 at Rocky 8 with Centminmod.

Using the most recent Centminmod version PHP 7.2 compiled without problems after forcing GCC 12, but the problems from the first post (high CPU load, timeouts at some pages) occurred again.

I found out that the whole problem was caused by the PHP opcache module. If opcache was disabled everything worked without problems. With opcache enabled the problems started again. But opcache is a must, so I had to solve this mystery.

Compiling PHP 7.2 with GCC 13 showed exactly the same errors.

However compiling PHP 7.2 with GCC 11 works well and opcache does what is expected without high CPU loads and timeouts.

So, if anyone uses Rocky 8 and needs to run the old PHP 7 with it (e.g. when still using XF 1.5), use

Code (Text):

DEVTOOLSETELEVEN='y'

in your custom_config to avoid any problems.
I don't know and didn't find any information about why PHP opcache does not compile correctly with GCC > 11.

But for now the problem has been solved.

 

I tested with different CPU too. The production server, were that problem occurred first, runs with AMD EPYC. The test server, which showed the same problem, runs with Intel Xeon Scalable CPUs. So I doubt the CPU makes a difference.

I also tested different settings with Zend Opcache. Any setting I tried did not help. So I stayed with your default CMM settings.

I did not try it with Alma 8, because we run Rocky 8/9 at all our servers and do not use Alma 8. But yes, there may be a difference between Alma and Rocky, as I know you prefer Alma and may use that when working at Centminmod.