Learn about Centmin Mod LEMP Stack today
Become a Member

Sysadmin MMonit auto restart nginx/php-fpm?

Discussion in 'System Administration' started by Matt Williams, Apr 18, 2022.

  1. Matt Williams

    Matt Williams WordPress Fanatic Premium Member

    527
    102
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +155
    Local Time:
    6:09 AM
    latest
    10
    Anyone have a good, working mmonit config file they can share? Just need an example to go by...

     
  2. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    8:09 PM
    Nginx 1.21.x
    MariaDB 10.x
    Haven't used monit for ages. But did have custom RPM monit RPMbuilt (CentOS version is too old) with installer for detecting services and adding configurations to /etc/monit.d/ directory. What do you currently use config wise?
     
  3. Matt Williams

    Matt Williams WordPress Fanatic Premium Member

    527
    102
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +155
    Local Time:
    6:09 AM
    latest
    10
    I don't currently use it but was looking for a service to auto restart nginx/php-fpm etc.. upon a failure and I know mmonit does just this. I had it setup before and worked good but just can't remember my old config I used with it. I have so many txt files on my computer with old stuff but none with this lol!
     
  4. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    8:09 PM
    Nginx 1.21.x
    MariaDB 10.x
    Maybe a better way Set up self-healing services with systemd. Note though current 123.09beta01's Nginx service doesn't use systemd service file but /etc/init.d/nginx so the articles outlined method won't work on Nginx in Centmin Mod 123.09beta01 on CentOS 7. The CentOS 8/AlmaLinux 8/Rocky Linux 8+ branch I am testing switches Nginx service to systemd though and no longer uses /etc/init.d/nginx. I'd backported Nginx systemd routine to CentOS 7 123.09beta01 but it's disabled by default for testing right now.

    But you can see an example of failed Nginx or PHP-FPM service being auto restarted after 5s on Alma Linux 8's Centmin Mod test server

    When I manually killed Nginx or PHP-FPM process, the status checks have Active: activating (auto-restart) and after configured 5 seconds, auto restarted the services.
    Code (Text):
    kill -9 $(pidof nginx)
    
    service nginx status
    Redirecting to /bin/systemctl status nginx.service
    ● nginx.service - Centmin Mod NGINX Server
       Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/nginx.service.d
              └─failure-restart.conf, openfileslimit.conf
       Active: activating (auto-restart) (Result: signal) since Mon 2022-04-18 16:16:57 EDT; 4s ago
      Process: 1368276 ExecStart=/usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf (code=exited, status=0/SUCCESS)
      Process: 1368274 ExecStartPre=/usr/local/sbin/nginx -t (code=exited, status=0/SUCCESS)
     Main PID: 1368277 (code=killed, signal=KILL)
    
    service nginx status
    Redirecting to /bin/systemctl status nginx.service
    ● nginx.service - Centmin Mod NGINX Server
       Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/nginx.service.d
              └─failure-restart.conf, openfileslimit.conf
       Active: active (running) since Mon 2022-04-18 16:17:03 EDT; 895ms ago
      Process: 1368336 ExecStart=/usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf (code=exited, status=0/SUCCESS)
      Process: 1368334 ExecStartPre=/usr/local/sbin/nginx -t (code=exited, status=0/SUCCESS)
     Main PID: 1368337 (nginx)
       Tasks: 3 (limit: 4981)
       Memory: 112.5M
       CGroup: /system.slice/nginx.service
              ├─1368337 nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
              ├─1368338 nginx: worker process
              └─1368339 nginx: worker process
    
    Apr 18 16:17:02 almalinux-8 systemd[1]: Starting Centmin Mod NGINX Server...
    Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    Apr 18 16:17:03 almalinux-8 systemd[1]: Started Centmin Mod NGINX Server.
    


    Code (Text):
    kill -9 $(pidof php-fpm)
    
    service php-fpm status
    Redirecting to /bin/systemctl status php-fpm.service
    ● php-fpm.service - PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/php-fpm.service.d
              └─failure-restart.conf, limit.conf
       Active: activating (auto-restart) (Result: signal) since Mon 2022-04-18 16:21:08 EDT; 4s ago
      Process: 1368530 ExecStart=/usr/local/sbin/php-fpm --daemonize --fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid (code=exited, status=0/SUCCESS)
     Main PID: 1368531 (code=killed, signal=KILL)
    
    Apr 18 16:21:08 almalinux-8 systemd[1]: php-fpm.service: Main process exited, code=killed, status=9/KILL
    Apr 18 16:21:08 almalinux-8 systemd[1]: php-fpm.service: Failed with result 'signal'.
    
    service php-fpm status
    Redirecting to /bin/systemctl status php-fpm.service
    ● php-fpm.service - PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/php-fpm.service.d
              └─failure-restart.conf, limit.conf
       Active: active (running) since Mon 2022-04-18 16:21:13 EDT; 670ms ago
      Process: 1368608 ExecStart=/usr/local/sbin/php-fpm --daemonize --fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid (code=exited, status=0/SUCCESS)
     Main PID: 1368609 (php-fpm)
       Tasks: 1 (limit: 4981)
       Memory: 11.6M
       CGroup: /system.slice/php-fpm.service
              └─1368609 php-fpm: master process (/usr/local/etc/php-fpm.conf)
    
    Apr 18 16:21:13 almalinux-8 systemd[1]: Starting PHP FastCGI Process Manager...
    Apr 18 16:21:13 almalinux-8 systemd[1]: Started PHP FastCGI Process Manager.
    

    Contents of override systemd files for PHP-FPM /etc/systemd/system/php-fpm.service.d/failure-restart.conf
    Code (Text):
    [Unit]
    StartLimitIntervalSec=30
    StartLimitBurst=2
    
    [Service]
    Restart=on-failure
    RestartSec=5s
    

    and Nginx /etc/systemd/system/nginx.service.d/failure-restart.conf
    Code (Text):
    [Unit]
    StartLimitIntervalSec=30
    StartLimitBurst=2
    
    [Service]
    Restart=on-failure
    RestartSec=5s
    


    check Nginx systemd logging shows after Nginx entered failed signal state, it waited 5 seconds before auto restarting Nginx service
    Code (Text):
    journalctl -u nginx --no-pager
    
    Apr 18 16:16:57 almalinux-8 systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
    Apr 18 16:16:57 almalinux-8 systemd[1]: nginx.service: Failed with result 'signal'.
    Apr 18 16:17:02 almalinux-8 systemd[1]: nginx.service: Service RestartSec=5s expired, scheduling restart.
    Apr 18 16:17:02 almalinux-8 systemd[1]: nginx.service: Scheduled restart job, restart counter is at 2.
    Apr 18 16:17:02 almalinux-8 systemd[1]: Stopped Centmin Mod NGINX Server.
    Apr 18 16:17:02 almalinux-8 systemd[1]: Starting Centmin Mod NGINX Server...
    Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    Apr 18 16:17:03 almalinux-8 nginx[1368334]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    Apr 18 16:17:03 almalinux-8 systemd[1]: Started Centmin Mod NGINX Server.
    


    Though I have rarely seen Nginx and PHP-FPM crash or be killed on a properly tuned server. So rather than auto restarted failed services, have an uptime monitor to notify of failures so you can investigate.
     
    Last edited: Apr 19, 2022