Discover Centmin Mod today
Register Now

PHP-FPM What's the reason fpmstop to be very slow?

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by rdan, Nov 24, 2018.

Tags:
  1. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed:123.09beta01
    • PHP Version Installed: 7.2.12
    This is a brand new server/new CMM installed/no visitors.
    Other servers with similar specs and has significant amount of traffic is instant.
     
  2. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    have you modified /usr/local/etc/php-fpm.conf at all ? server specs ? number of cpu cores/memory ?

    output for command
    Code (Text):
    ps xao pid,ppid,command | grep 'php-fpm[:]'
    
     
  3. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Yes with dynamic mode.
    Hm, I already reinstall the server.

    I will try to re apply my custom settings and observe again.
     
  4. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Just an update, this only happens when I use dynamic mode with this formula.
    While all my other server have this config has no problem.
     
  5. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    I still have this issue.

    Old vs New server.
    Both have same CPU, RAM, Disk on NVMe is almost the same.

    With this config:
    Code:
    pm = dynamic
    pm.max_children = cpu x 4
    pm.start_servers = # of cpu
    pm.min_spare_servers = cpu / 2
    pm.max_spare_servers = cpu x 2
    pm.max_requests = 2000
    Old server restarts quick, but New server takes several seconds.
     
  6. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Old server even have more child process running at 16.
    New server has no visitor.
    So has only 8 child running.
     
  7. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    still only problem with dynamic process management instead of on demand or static ?
     
  8. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Yes, only ondemand will restart/stop quickly.
     
  9. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    how slow is slow ? run this debug command to stop php-fpm service and post it's output in CODE or CODEB bbcode tags. You can start php-fpm service again after running this command
    Code (Text):
    time bash -x /etc/init.d/php-fpm stop
    


    Edit:

    If the time is around ~10 seconds, it means it's related to php-fpm setting process_control_timeout = 10s in /usr/local/etc/php-fpm.conf

    PHP: Configuration - Manual
    This ensures child PHP-FPM processes complete any work they have been exiting and was old way for init.d CentOS php-fpm startup service script. So if you set process_control_timeout = 5s it will lower time to wait.

    Then on CentOS 7 do
    Code (Text):
    systemctl daemon-reload
    fpmrestart
    


    Now on the other hand, if you want to try out CentOS 7 php-fpm systemd startup service script I have been developing, you can run this command and it will replace /etc/init.d/php-fpm with systemd based service /usr/lib/systemd/system/php-fpm.service and update all cmd shortcuts i.e. fpmrestart, fpmstop, fpmstart, fpmreload, fpmstatus appropriately.
    Code (Text):
    curl -sL https://gist.githubusercontent.com/centminmod/a84a86775925ea558fece1f0ac130be3/raw/php-systemd.sh | bash

    example
    Code (Text):
    fpmstatus
    Redirecting to /bin/systemctl status php-fpm.service
    ● php-fpm.service - PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/php-fpm.service.d
               └─limit.conf
       Active: active (running) since Fri 2019-01-25 18:14:31 UTC; 2s ago
     Main PID: 2447 (php-fpm)
       CGroup: /system.slice/php-fpm.service
               ├─2447 php-fpm: master process (/usr/local/etc/php-fpm.conf)
               ├─2448 php-fpm: pool www
               ├─2449 php-fpm: pool www
               ├─2450 php-fpm: pool www
               ├─2451 php-fpm: pool www
               ├─2452 php-fpm: pool www
               ├─2453 php-fpm: pool www
               ├─2454 php-fpm: pool www
               └─2455 php-fpm: pool www
    
    Jan 25 18:14:31 test.com systemd[1]: Stopped PHP FastCGI Process Manager.
    Jan 25 18:14:31 test.com systemd[1]: Starting PHP FastCGI Process Manager...
    Jan 25 18:14:31 test.com systemd[1]: Started PHP FastCGI Process Manager.
    

    experimental so test and let me know over time how it goes.
     
    Last edited: Jan 26, 2019
    • Informative Informative x 1
  10. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Code:
    # time bash -x /etc/init.d/php-fpm stop
    + '[' -f /proc/user_beancounters ']'
    + '[' '!' -d /var/run/php-fpm ']'
    + prefix=/usr/local
    + exec_prefix=/usr/local
    + php_fpm_BIN=/usr/local/sbin/php-fpm
    + php_fpm_CONF=/usr/local/etc/php-fpm.conf
    + php_fpm_PID=/var/run/php-fpm/php-fpm.pid
    + php_opts='--fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid'
    + case "$1" in
    + echo -n 'Gracefully shutting down php-fpm '
    Gracefully shutting down php-fpm + '[' '!' -r /var/run/php-fpm/php-fpm.pid ']'
    ++ cat /var/run/php-fpm/php-fpm.pid
    + kill -QUIT 14501
    + wait_for_pid removed /var/run/php-fpm/php-fpm.pid
    + try=0
    + test 0 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 0 + 1
    + try=1
    + sleep 1
    + test 1 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 1 + 1
    + try=2
    + sleep 1
    + test 2 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 2 + 1
    + try=3
    + sleep 1
    + test 3 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 3 + 1
    + try=4
    + sleep 1
    + test 4 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 4 + 1
    + try=5
    + sleep 1
    + test 5 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 5 + 1
    + try=6
    + sleep 1
    + test 6 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 6 + 1
    + try=7
    + sleep 1
    + test 7 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 7 + 1
    + try=8
    + sleep 1
    + test 8 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 8 + 1
    + try=9
    + sleep 1
    + test 9 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 9 + 1
    + try=10
    + sleep 1
    + test 10 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + try=
    + break
    + '[' -n '' ']'
    + echo ' done'
     done
    
    real    0m10.083s
    user    0m0.035s
    sys    0m0.049s
     
  11. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    But why on old server install, config is 100% same including this, yet restart very quick?
    This only happens server I installed since late last year.
     
  12. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Yeah, it's a little bit okay from 10s to 5s :D.
    Code:
    # time bash -x /etc/init.d/php-fpm stop
    + '[' -f /proc/user_beancounters ']'
    + '[' '!' -d /var/run/php-fpm ']'
    + prefix=/usr/local
    + exec_prefix=/usr/local
    + php_fpm_BIN=/usr/local/sbin/php-fpm
    + php_fpm_CONF=/usr/local/etc/php-fpm.conf
    + php_fpm_PID=/var/run/php-fpm/php-fpm.pid
    + php_opts='--fpm-config /usr/local/etc/php-fpm.conf --pid /var/run/php-fpm/php-fpm.pid'
    + case "$1" in
    + echo -n 'Gracefully shutting down php-fpm '
    Gracefully shutting down php-fpm + '[' '!' -r /var/run/php-fpm/php-fpm.pid ']'
    ++ cat /var/run/php-fpm/php-fpm.pid
    + kill -QUIT 29573
    + wait_for_pid removed /var/run/php-fpm/php-fpm.pid
    + try=0
    + test 0 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 0 + 1
    + try=1
    + sleep 1
    + test 1 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 1 + 1
    + try=2
    + sleep 1
    + test 2 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 2 + 1
    + try=3
    + sleep 1
    + test 3 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 3 + 1
    + try=4
    + sleep 1
    + test 4 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + echo -n .
    .++ expr 4 + 1
    + try=5
    + sleep 1
    + test 5 -lt 35
    + case "$1" in
    + '[' '!' -f /var/run/php-fpm/php-fpm.pid ']'
    + try=
    + break
    + '[' -n '' ']'
    + echo ' done'
     done
    
    real    0m5.051s
    user    0m0.021s
    sys    0m0.032s
     
  13. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    how old is old ? more than 2-5yrs ? that is time since last time /etc/init.d/php-fpm was modified according to github blame info at centminmod/centminmod which lists the time for each commit for each piece of code.
     
  14. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    This doesn't run on my install by the way, and
     
  15. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Installed March 2018, so less than 1 year.
     
  16. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    On both server, this file is also 100% the same.
     
  17. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  18. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    then strange it is.
     
  19. rdan

    rdan Well-Known Member

    4,524
    1,085
    113
    May 25, 2014
    Ratings:
    +1,583
    Local Time:
    3:49 PM
    Mainline
    10.2
    Not sure what else I need to investigate.
     
  20. eva2000

    eva2000 Administrator Staff Member

    39,800
    8,778
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,511
    Local Time:
    5:49 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
..