Sysadmin MMonit auto restart nginx/php-fpm?

Haven't used monit for ages. But did have custom RPM monit RPMbuilt (CentOS version is too old) with installer for detecting services and adding configurations to /etc/monit.d/ directory. What do you currently use config wise?

 

Maybe a better way Set up self-healing services with systemd. Note though current 123.09beta01's Nginx service doesn't use systemd service file but /etc/init.d/nginx so the articles outlined method won't work on Nginx in Centmin Mod 123.09beta01 on CentOS 7. The CentOS 8/AlmaLinux 8/Rocky Linux 8+ branch I am testing switches Nginx service to systemd though and no longer uses /etc/init.d/nginx. I'd backported Nginx systemd routine to CentOS 7 123.09beta01 but it's disabled by default for testing right now.

But you can see an example of failed Nginx or PHP-FPM service being auto restarted after 5s on Alma Linux 8's Centmin Mod test server

When I manually killed Nginx or PHP-FPM process, the status checks have Active: activating (auto-restart) and after configured 5 seconds, auto restarted the services.

Code (Text):

kill -9 $(pidof nginx)

service nginx status
Redirecting to /bin/systemctl status nginx.service
● nginx.service - Centmin Mod NGINX Server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/nginx.service.d
          └─failure-restart.conf, openfileslimit.conf
   Active: activating (auto-restart) (Result: signal) since Mon 2022-04-18 16:16:57 EDT; 4s ago
  Process: 1368276 ExecStart=/usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf (code=exited, status=0/SUCCESS)
  Process: 1368274 ExecStartPre=/usr/local/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Main PID: 1368277 (code=killed, signal=KILL)

service nginx status
Redirecting to /bin/systemctl status nginx.service
● nginx.service - Centmin Mod NGINX Server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/nginx.service.d
          └─failure-restart.conf, openfileslimit.conf
   Active: active (running) since Mon 2022-04-18 16:17:03 EDT; 895ms ago
  Process: 1368336 ExecStart=/usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf (code=exited, status=0/SUCCESS)
  Process: 1368334 ExecStartPre=/usr/local/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Main PID: 1368337 (nginx)
   Tasks: 3 (limit: 4981)
   Memory: 112.5M
   CGroup: /system.slice/nginx.service
          ├─1368337 nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
          ├─1368338 nginx: worker process
          └─1368339 nginx: worker process

Apr 18 16:17:02 almalinux-8 systemd[1]: Starting Centmin Mod NGINX Server...
Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
Apr 18 16:17:03 almalinux-8 systemd[1]: Started Centmin Mod NGINX Server.

Code (Text):

kill -9 $(pidof php-fpm)

service php-fpm status
Redirecting to /bin/systemctl status php-fpm.service
● php-fpm.service - PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/php-fpm.service.d
          └─failure-restart.conf, limit.conf
   Active: activating (auto-restart) (Result: signal) since Mon 2022-04-18 16:21:08 EDT; 4s ago
  Process: 1368530 ExecStart=/usr/local/sbin/php-fpm --daemonize --fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid (code=exited, status=0/SUCCESS)
 Main PID: 1368531 (code=killed, signal=KILL)

Apr 18 16:21:08 almalinux-8 systemd[1]: php-fpm.service: Main process exited, code=killed, status=9/KILL
Apr 18 16:21:08 almalinux-8 systemd[1]: php-fpm.service: Failed with result 'signal'.

service php-fpm status
Redirecting to /bin/systemctl status php-fpm.service
● php-fpm.service - PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/php-fpm.service.d
          └─failure-restart.conf, limit.conf
   Active: active (running) since Mon 2022-04-18 16:21:13 EDT; 670ms ago
  Process: 1368608 ExecStart=/usr/local/sbin/php-fpm --daemonize --fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid (code=exited, status=0/SUCCESS)
 Main PID: 1368609 (php-fpm)
   Tasks: 1 (limit: 4981)
   Memory: 11.6M
   CGroup: /system.slice/php-fpm.service
          └─1368609 php-fpm: master process (/usr/local/etc/php-fpm.conf)

Apr 18 16:21:13 almalinux-8 systemd[1]: Starting PHP FastCGI Process Manager...
Apr 18 16:21:13 almalinux-8 systemd[1]: Started PHP FastCGI Process Manager.

Contents of override systemd files for PHP-FPM /etc/systemd/system/php-fpm.service.d/failure-restart.conf

Code (Text):

[Unit]
StartLimitIntervalSec=30
StartLimitBurst=2

[Service]
Restart=on-failure
RestartSec=5s

and Nginx /etc/systemd/system/nginx.service.d/failure-restart.conf

Code (Text):

[Unit]
StartLimitIntervalSec=30
StartLimitBurst=2

[Service]
Restart=on-failure
RestartSec=5s

check Nginx systemd logging shows after Nginx entered failed signal state, it waited 5 seconds before auto restarting Nginx service

Code (Text):

journalctl -u nginx --no-pager

Apr 18 16:16:57 almalinux-8 systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
Apr 18 16:16:57 almalinux-8 systemd[1]: nginx.service: Failed with result 'signal'.
Apr 18 16:17:02 almalinux-8 systemd[1]: nginx.service: Service RestartSec=5s expired, scheduling restart.
Apr 18 16:17:02 almalinux-8 systemd[1]: nginx.service: Scheduled restart job, restart counter is at 2.
Apr 18 16:17:02 almalinux-8 systemd[1]: Stopped Centmin Mod NGINX Server.
Apr 18 16:17:02 almalinux-8 systemd[1]: Starting Centmin Mod NGINX Server...
Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
Apr 18 16:17:03 almalinux-8 systemd[1]: Started Centmin Mod NGINX Server.

Though I have rarely seen Nginx and PHP-FPM crash or be killed on a properly tuned server. So rather than auto restarted failed services, have an uptime monitor to notify of failures so you can investigate.