Welcome to Centmin Mod Community
Register Now

PHP-FPM PHP Update hangs on...

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by RoldanLT, Apr 14, 2017.

  1. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • PHP Version Installed: 7.1.x
    Code:
      ____   _   _  ____        ____                __  _                         
     |  _ \ | | | ||  _ \  _   / ___| ___   _ __   / _|(_)  __ _  _   _  _ __  ___
     | |_) || |_| || |_) |(_) | |    / _ \ | '_ \ | |_ | | / _` || | | || '__|/ _ \
     |  __/ |  _  ||  __/  _  | |___| (_) || | | ||  _|| || (_| || |_| || |  |  __/
     |_|    |_| |_||_|    (_)  \____|\___/ |_| |_||_|  |_| \__, | \__,_||_|   \___|
                                                           |___/                   
    /opt/rh/devtoolset-4/root/usr/bin/gcc
    /opt/rh/devtoolset-4/root/usr/bin/g++
    centos 7 set devtoolset-4
    ccache /opt/rh/devtoolset-4/root/usr/bin/gcc
    ccache /opt/rh/devtoolset-4/root/usr/bin/g++
    /opt/rh/devtoolset-4/root/usr/bin/gcc
    /opt/rh/devtoolset-4/root/usr/bin/g++
    centos 7 set devtoolset-4
    ccache /opt/rh/devtoolset-4/root/usr/bin/gcc
    ccache /opt/rh/devtoolset-4/root/usr/bin/g++
    gcc (GCC) 5.3.1 20160406 (Red Hat 5.3.1-6)
    Copyright (C) 2015 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
      -m64                                [enabled]
      -m80387                             [enabled]
      -m96bit-long-double                 [enabled]
      -mabm                               [enabled]
      -maes                               [enabled]
      -malign-stringops                   [enabled]
      -march=                             haswell
      -mavx                               [enabled]
      -mavx2                              [enabled]
      -mbmi                               [enabled]
      -mbmi2                              [enabled]
      -mcx16                              [enabled]
      -mf16c                              [enabled]
      -mfancy-math-387                    [enabled]
      -mfentry                            [enabled]
      -mfma                               [enabled]
      -mfp-ret-in-387                     [enabled]
      -mfpmath=                           387
      -mfsgsbase                          [enabled]
      -mfxsr                              [enabled]
      -mglibc                             [enabled]
      -mhard-float                        [enabled]
      -mieee-fp                           [enabled]
      -mlong-double-80                    [enabled]
      -mlzcnt                             [enabled]
      -mmmx                               [enabled]
      -mmovbe                             [enabled]
      -mpclmul                            [enabled]
      -mpopcnt                            [enabled]
      -mpush-args                         [enabled]
      -mrdrnd                             [enabled]
      -mred-zone                          [enabled]
      -msahf                              [enabled]
      -msse                               [enabled]
      -msse2                              [enabled]
      -msse3                              [enabled]
      -msse4                              [enabled]
      -msse4.1                            [enabled]
      -msse4.2                            [enabled]
      -mssse3                             [enabled]
      -mstackrealign                      [enabled]
      -mtls-direct-seg-refs               [enabled]
      -mtune-ctrl=                       
      -mtune=                             haswell
      -mxsave                             [enabled]
      -mxsaveopt                          [enabled]
      Valid arguments to -mfpmath=:
    --enable-huge-code-pages
    
    set vm.nr.hugepages in /etc/sysctl.conf
    
    #sysctl --system also hangs..
     
  2. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    Code:
    # cat /etc/sysctl.d/101-sysctl.conf
    # centminmod added
    fs.nr_open=12000000
    fs.file-max=9000000
    net.core.wmem_max=16777216
    net.core.rmem_max=16777216
    net.ipv4.tcp_rmem=8192 87380 16777216                                         
    net.ipv4.tcp_wmem=8192 65536 16777216
    net.core.netdev_max_backlog=8192
    net.core.somaxconn=8151
    net.core.optmem_max=8192
    net.ipv4.tcp_fin_timeout=10
    net.ipv4.tcp_keepalive_intvl=30
    net.ipv4.tcp_keepalive_probes=3
    net.ipv4.tcp_keepalive_time=240
    net.ipv4.tcp_max_syn_backlog=8192
    net.ipv4.tcp_sack=1
    net.ipv4.tcp_syn_retries=3
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_tw_recycle = 0
    net.ipv4.tcp_tw_reuse = 0
    net.ipv4.tcp_max_tw_buckets = 1440000
    vm.swappiness=10
    vm.min_free_kbytes=65536
    net.ipv4.ip_local_port_range=1024 65535
    net.ipv4.tcp_slow_start_after_idle=0
    net.ipv4.tcp_limit_output_bytes=65536
    net.ipv4.tcp_rfc1337=1
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.all.log_martians = 1
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.conf.default.log_martians = 1
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.secure_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.icmp_ignore_bogus_error_responses = 1
    net.netfilter.nf_conntrack_helper=0
    net.netfilter.nf_conntrack_max = 524288
    net.netfilter.nf_conntrack_tcp_timeout_established = 28800
    net.netfilter.nf_conntrack_generic_timeout = 60
    net.ipv4.tcp_challenge_ack_limit = 999999999
     
  3. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    To troubleshoot, you need to check the the php upgrade logs at /root/centminlogs and instructions under Sharing logs and errors heading for using Pastebin.com or Gists to share a sanitised version of the contents of the nginx_upgrade and php_upgrade logs. You can see full details at How to troubleshoot Centmin Mod initial install issues

    if you type this command it lists all logs in date ascending order so latest log at bottom
    Code (Text):
    ls -lArt /root/centminlogs

    so copy the entire contents of latest php_upgrade log to gist.github.com or pastebin.com

    you can use grep to filter the logs, i.e. look for nginx_upgrade or php in log name
    Code (Text):
    ls -lahrt /root/centminlogs/ | egrep 'nginx_upgrade|php_upgrade'
    

    Code (Text):
    ls -lahrt /root/centminlogs/ | egrep 'nginx_upgrade|php_upgrade'
    -rw-r--r--  1 root root  2.6M Oct  3 03:48 centminmod_1.2.3-eva2000.09.001_031016-034409_php_upgrade.log
    -rw-r--r--  1 root root  664K Oct 12 09:24 centminmod_1.2.3-eva2000.09.001_121016-092406_nginx_upgrade.log
    

    So the last php upgrade log was named centminmod_1.2.3-eva2000.09.001_031016-034409_php_upgrade.log and located at /root/centminlogs/centminmod_1.2.3-eva2000.09.001_031016-034409_php_upgrade.log
    then use cat command to output the contents of that log.

    clear your ssh window buffer/screen and type
    Code (Text):
    cat /root/centminlogs/centminmod_1.2.3-eva2000.09.001_031016-034409_php_upgrade.log

    then select and copy and paste output to pastebin.com or gist.github.com file to share. If your SSH client's scroll buffer isn't large enough using cat might not output the entire log file contents, so you may need to download the log and use local text editor to open and copy and paste.

    so only need content of a specific log, in this case most recent php_upgrade.log logs
     
  4. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    I think it's something to do with this:
    Both:
    #sysctl --system
    #sysctl -p

    Hang...

    upload_2017-4-14_5-40-40.png

    upload_2017-4-14_5-40-56.png
     
  5. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    I think Centmin added vm.nr_hugepages=15344:
    Code:
    # cat /etc/sysctl.d/99-sysctl.conf
    # System default settings live in /usr/lib/sysctl.d/00-system.conf.
    # To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
    #
    # For more information, see sysctl.conf(5) and sysctl.d(5).
    
    # Disable IPv6 autoconf
    net.ipv6.conf.all.autoconf = 0
    net.ipv6.conf.default.autoconf = 0
    net.ipv6.conf.eth0.autoconf = 0
    net.ipv6.conf.all.accept_ra = 0
    net.ipv6.conf.default.accept_ra = 0
    net.ipv6.conf.eth0.accept_ra = 0
    #vm.nr_hugepages=15344
    
    I commented out #vm.nr_hugepages=15344 and WORKS now.,
     
    • Informative Informative x 1
  6. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
  7. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    strange just tested and works for me on OVH CentOS 7.3
    Code (Text):
    php -v
    PHP 7.1.4 (cli) (built: Apr 13 2017 21:43:15) ( NTS )
    Copyright (c) 1997-2017 The PHP Group
    Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
        with Zend OPcache v7.1.4, Copyright (c) 1999-2017, by Zend Technologies
    

    Code (Text):
    cat /etc/sysctl.conf
    # sysctl settings are defined through files in
    # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
    #
    # Vendors settings live in /usr/lib/sysctl.d/.
    # To override a whole file, create a new file with the same in
    # /etc/sysctl.d/ and put new settings there. To override
    # only specific settings, add a file with a lexically later
    # name in /etc/sysctl.d/ and put new settings there.
    #
    # For more information, see sysctl.conf(5) and sysctl.d(5).
    
    # Disable IPv6 autoconf
    net.ipv6.conf.all.autoconf = 0
    net.ipv6.conf.default.autoconf = 0
    net.ipv6.conf.eth0.autoconf = 0
    net.ipv6.conf.all.accept_ra = 0
    net.ipv6.conf.default.accept_ra = 0
    net.ipv6.conf.eth0.accept_ra = 0
    vm.nr_hugepages=1888
    

    Code (Text):
    cat /etc/sysctl.d/99-sysctl.conf
    # sysctl settings are defined through files in
    # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
    #
    # Vendors settings live in /usr/lib/sysctl.d/.
    # To override a whole file, create a new file with the same in
    # /etc/sysctl.d/ and put new settings there. To override
    # only specific settings, add a file with a lexically later
    # name in /etc/sysctl.d/ and put new settings there.
    #
    # For more information, see sysctl.conf(5) and sysctl.d(5).
    
    # Disable IPv6 autoconf 
    net.ipv6.conf.all.autoconf = 0
    net.ipv6.conf.default.autoconf = 0
    net.ipv6.conf.eth0.autoconf = 0
    net.ipv6.conf.all.accept_ra = 0
    net.ipv6.conf.default.accept_ra = 0
    net.ipv6.conf.eth0.accept_ra = 0
    vm.nr_hugepages=1888
    
     
  8. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    Yeah all other 5 servers I got works fine on PHP upgrade.
    Just this single server fail/hang.

    #vm.nr_hugepages=15344, seems to high value though?
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    it's auto optimised and calculated based on server specs via tools/hptweaks.sh script.

    if you lower the value to 1888 does it work for sysctl -p ?
     
  10. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    also what's output of this command on all your servers
    Code (Text):
    cat /sys/devices/system/node/node*/meminfo | fgrep Huge
    

    and
    Code (Text):
    numactl --hardware
    
     
  11. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    Output from the server I got this issue:

     
  12. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    looks like i need to take into account numa relevant multiple cpu socket node setups as to amount of free memory available.

    what's output for
    Code (Text):
    free -m
    

    Code (Text):
    cat /proc/meminfo
    
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
  14. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    Code:
    # free -m
                  total        used        free      shared  buff/cache   available
    Mem:         257858       18958       84974         841      153925      236786
    Swap:          4998           3        4995
    
     
  15. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    to test it out the fix, edit /etc/sysctl.conf and remove your vm.nr_hugepages line and re-run manually, tools/hptweaks.sh to see if it hangs and then check /etc/sysctl.conf to see the updated formula calculated value for vm.nr_hugepages
     
  16. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
    PHP upgrade works fine now.
    Thanks a lot Eva!
     
    • Like Like x 1
  17. RoldanLT

    RoldanLT Well-Known Member

    3,978
    965
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,329
    Local Time:
    9:25 AM
    1.11
    10.2
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5
    excellent :)
     
  19. pamamolf

    pamamolf Well-Known Member

    2,821
    253
    83
    May 31, 2014
    Ratings:
    +447
    Local Time:
    3:25 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    So do we have to manually fix that on our servers ?
     
  20. eva2000

    eva2000 Administrator Staff Member

    30,956
    6,917
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,418
    Local Time:
    11:25 AM
    Nginx 1.13.x
    MariaDB 5.5