Want more timely Centmin Mod News Updates?
Become a Member

PHP-FPM What's the reason fpmstop to be very slow?

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by rdan, Nov 24, 2018.

Tags:
  1. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed:123.09beta01
    • PHP Version Installed: 7.2.12
    This is a brand new server/new CMM installed/no visitors.
    Other servers with similar specs and has significant amount of traffic is instant.
     
  2. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    have you modified /usr/local/etc/php-fpm.conf at all ? server specs ? number of cpu cores/memory ?

    output for command
    Code (Text):
    ps xao pid,ppid,command | grep 'php-fpm[:]'
    
     
  3. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Yes with dynamic mode.
    Hm, I already reinstall the server.

    I will try to re apply my custom settings and observe again.
     
  4. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Just an update, this only happens when I use dynamic mode with this formula.
    While all my other server have this config has no problem.
     
  5. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    I still have this issue.

    Old vs New server.
    Both have same CPU, RAM, Disk on NVMe is almost the same.

    With this config:
    Code:
    pm = dynamic
    pm.max_children = cpu x 4
    pm.start_servers = # of cpu
    pm.min_spare_servers = cpu / 2
    pm.max_spare_servers = cpu x 2
    pm.max_requests = 2000
    Old server restarts quick, but New server takes several seconds.
     
  6. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Old server even have more child process running at 16.
    New server has no visitor.
    So has only 8 child running.
     
  7. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    still only problem with dynamic process management instead of on demand or static ?
     
  8. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Yes, only ondemand will restart/stop quickly.
     
  9. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    how slow is slow ? run this debug command to stop php-fpm service and post it's output in CODE or CODEB bbcode tags. You can start php-fpm service again after running this command
    Code (Text):
    time bash -x /etc/init.d/php-fpm stop
    


    Edit:

    If the time is around ~10 seconds, it means it's related to php-fpm setting process_control_timeout = 10s in /usr/local/etc/php-fpm.conf

    PHP: Configuration - Manual
    This ensures child PHP-FPM processes complete any work they have been exiting and was old way for init.d CentOS php-fpm startup service script. So if you set process_control_timeout = 5s it will lower time to wait.

    Then on CentOS 7 do
    Code (Text):
    systemctl daemon-reload
    fpmrestart
    


    Now on the other hand, if you want to try out CentOS 7 php-fpm systemd startup service script I have been developing, you can run this command and it will replace /etc/init.d/php-fpm with systemd based service /usr/lib/systemd/system/php-fpm.service and update all cmd shortcuts i.e. fpmrestart, fpmstop, fpmstart, fpmreload, fpmstatus appropriately.
    Code (Text):
    curl -sL https://gist.githubusercontent.com/centminmod/a84a86775925ea558fece1f0ac130be3/raw/php-systemd.sh | bash

    example
    Code (Text):
    fpmstatus
    Redirecting to /bin/systemctl status php-fpm.service
    ● php-fpm.service - PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/php-fpm.service.d
               └─limit.conf
       Active: active (running) since Fri 2019-01-25 18:14:31 UTC; 2s ago
     Main PID: 2447 (php-fpm)
       CGroup: /system.slice/php-fpm.service
               ├─2447 php-fpm: master process (/usr/local/etc/php-fpm.conf)
               ├─2448 php-fpm: pool www
               ├─2449 php-fpm: pool www
               ├─2450 php-fpm: pool www
               ├─2451 php-fpm: pool www
               ├─2452 php-fpm: pool www
               ├─2453 php-fpm: pool www
               ├─2454 php-fpm: pool www
               └─2455 php-fpm: pool www
    
    Jan 25 18:14:31 test.com systemd[1]: Stopped PHP FastCGI Process Manager.
    Jan 25 18:14:31 test.com systemd[1]: Starting PHP FastCGI Process Manager...
    Jan 25 18:14:31 test.com systemd[1]: Started PHP FastCGI Process Manager.
    

    experimental so test and let me know over time how it goes.
     
    Last edited: Jan 26, 2019
    • Informative Informative x 1
  10. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Code:
    # time bash -x /etc/init.d/php-fpm stop
    + '[' -f /proc/user_beancounters ']'
    + '[' '!' -d /var/run/php-fpm ']'
    + prefix=/usr/local
    + exec_prefix=/usr/local
    + php_fpm_BIN=/usr/local/sbin/php-fpm
    + php_fpm_CONF=/usr/local/etc/php-fpm.conf
    + php_fpm_PID=/var/run/php-fpm/php-fpm.pid
    + php_opts='--fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid'
    + case "$1" in
    + echo -n 'Gracefully shutting down php-fpm '
    Gracefully shutting down php-fpm + '[' '!' -r /var/run/php-fpm/php-fpm.pid ']'
    ++ cat /var/run/php-fpm/php-fpm.pid
    + kill -QUIT 14501
    + wait_for_pid removed /var/run/php-fpm/php-fpm.pid
    + try=0
    + test 0 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 0 + 1
    + try=1
    + sleep 1
    + test 1 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 1 + 1
    + try=2
    + sleep 1
    + test 2 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 2 + 1
    + try=3
    + sleep 1
    + test 3 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 3 + 1
    + try=4
    + sleep 1
    + test 4 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 4 + 1
    + try=5
    + sleep 1
    + test 5 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 5 + 1
    + try=6
    + sleep 1
    + test 6 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 6 + 1
    + try=7
    + sleep 1
    + test 7 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 7 + 1
    + try=8
    + sleep 1
    + test 8 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 8 + 1
    + try=9
    + sleep 1
    + test 9 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 9 + 1
    + try=10
    + sleep 1
    + test 10 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + try=
    + break
    + '[' -n '' ']'
    + echo ' done'
     done
    
    real    0m10.083s
    user    0m0.035s
    sys    0m0.049s
     
  11. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    But why on old server install, config is 100% same including this, yet restart very quick?
    This only happens server I installed since late last year.
     
  12. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Yeah, it's a little bit okay from 10s to 5s :D.
    Code:
    # time bash -x /etc/init.d/php-fpm stop
    + '[' -f /proc/user_beancounters ']'
    + '[' '!' -d /var/run/php-fpm ']'
    + prefix=/usr/local
    + exec_prefix=/usr/local
    + php_fpm_BIN=/usr/local/sbin/php-fpm
    + php_fpm_CONF=/usr/local/etc/php-fpm.conf
    + php_fpm_PID=/var/run/php-fpm/php-fpm.pid
    + php_opts='--fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid'
    + case "$1" in
    + echo -n 'Gracefully shutting down php-fpm '
    Gracefully shutting down php-fpm + '[' '!' -r /var/run/php-fpm/php-fpm.pid ']'
    ++ cat /var/run/php-fpm/php-fpm.pid
    + kill -QUIT 29573
    + wait_for_pid removed /var/run/php-fpm/php-fpm.pid
    + try=0
    + test 0 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 0 + 1
    + try=1
    + sleep 1
    + test 1 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 1 + 1
    + try=2
    + sleep 1
    + test 2 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 2 + 1
    + try=3
    + sleep 1
    + test 3 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 3 + 1
    + try=4
    + sleep 1
    + test 4 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 4 + 1
    + try=5
    + sleep 1
    + test 5 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + try=
    + break
    + '[' -n '' ']'
    + echo ' done'
     done
    
    real    0m5.051s
    user    0m0.021s
    sys    0m0.032s
     
  13. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    how old is old ? more than 2-5yrs ? that is time since last time /etc/init.d/php-fpm was modified according to github blame info at centminmod/centminmod which lists the time for each commit for each piece of code.
     
  14. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    This doesn't run on my install by the way, and
     
  15. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Installed March 2018, so less than 1 year.
     
  16. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    On both server, this file is also 100% the same.
     
  17. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  18. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    then strange it is.
     
  19. rdan

    rdan Premium Member Premium Member

    4,501
    1,079
    113
    May 25, 2014
    Ratings:
    +1,576
    Local Time:
    7:32 PM
    Mainline
    10.2
    Not sure what else I need to investigate.
     
  20. eva2000

    eva2000 Administrator Staff Member

    38,028
    8,352
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,865
    Local Time:
    9:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
..