Learn about Centmin Mod LEMP Stack today
Become a Member

Beta Branch Transparent Huge Pages Tweaks tools/hptweaks.sh

Discussion in 'Beta release code' started by eva2000, Dec 18, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    For dynamic tuning and detection of kernel's transparent huge pages support and memory and nr.hugepages count, I wrote a standalone shell script tools/hptweaks.sh for testing in Centmin Mod 123.09beta01 branch for now.

    This script came about due to @Ahmad testing with PHP 7 which allows Zend Opcache to support transparent pages. However, default CentOS 6 and 7 builds do not set the nr.hugepages count so even if kernel has enabled transparent huge pages, there is no memory allocated to the huge pages pool. This script tries to dynamically determine an optimal nr.hugepages count and max locked memory system limit based on the amount of free memory available on the system (calculated based on 12.5% of free available memory).

    To be able to access tools/hptweaks.sh, you need to be using Centmin Mod 123.09beta01 branch and be up to date. To update your Centmin Mod builds follow instructions at centminmod.com/upgrade.html and below:
    The actual code for tools/hptweaks.sh at centminmod/hptweaks.sh at 123.09beta01 · centminmod/centminmod · GitHub

     
    Last edited: Dec 18, 2015
  2. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    To test tools/hptweaks.sh, you need to be on latest Centmin Mod 123.09beta01 branch and run the following commands

    Code:
    cd /usr/local/src/centminmod/tools
    ./hptweaks.sh
    then please share the output from the script run here in CODE tags
     
  3. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    Trying it on the same server that was having the issue (of course set it back to 0 and removed it from sysctl.conf first).
    Script does not output anything.
    Code:
    [root@localhost tools]# ./hptweaks.sh 
    
    [root@localhost tools]# cat /etc/sysctl.conf
    # Kernel sysctl configuration file for Red Hat Linux
    #
    # For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
    # sysctl.conf(5) for more details.
    
    # Controls IP packet forwarding
    net.ipv4.ip_forward = 0
    
    # Controls source route verification
    net.ipv4.conf.default.rp_filter = 1
    
    # Do not accept source routing
    net.ipv4.conf.default.accept_source_route = 0
    
    # Controls the System Request debugging functionality of the kernel
    kernel.sysrq = 0
    
    # Controls whether core dumps will append the PID to the core filename.
    # Useful for debugging multi-threaded applications.
    kernel.core_uses_pid = 1
    
    # Controls the use of TCP syncookies
    net.ipv4.tcp_syncookies = 1
    
    # Controls the default maxmimum size of a mesage queue
    kernel.msgmnb = 65536
    
    # Controls the maximum size of a message, in bytes
    kernel.msgmax = 65536
    
    # Controls the maximum shared segment size, in bytes
    kernel.shmmax = 68719476736
    
    # Controls the maximum number of shared memory segments, in pages
    kernel.shmall = 4294967296
    # centminmod added
    fs.nr_open=12000000
    fs.file-max=9000000
    net.core.wmem_max=16777216
    net.core.rmem_max=16777216
    net.ipv4.tcp_rmem=8192 87380 16777216   
    net.ipv4.tcp_wmem=8192 65536 16777216
    net.core.netdev_max_backlog=8192
    net.core.somaxconn=8150
    net.core.optmem_max=8192
    net.ipv4.tcp_fin_timeout=10
    net.ipv4.tcp_keepalive_intvl=30
    net.ipv4.tcp_keepalive_probes=3
    net.ipv4.tcp_keepalive_time=240
    net.ipv4.tcp_max_syn_backlog=8192
    net.ipv4.tcp_sack=1
    net.ipv4.tcp_syn_retries=3
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1
    vm.swappiness=10
    vm.min_free_kbytes=65536
    net.ipv4.ip_local_port_range=1024 65535
    
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.all.log_martians = 1
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.conf.default.log_martians = 1
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.secure_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.icmp_ignore_bogus_error_responses = 1
    
    [root@localhost tools]# cat /proc/meminfo | grep -i HugePage
    AnonHugePages:  743424 kB
    HugePages_Total:  0
    HugePages_Free:  0
    HugePages_Rsvd:  0
    HugePages_Surp:  0
    Hugepagesize:  2048 kB
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    do you have redis server running as script will detect if redis server is running and if so disable transparent huge pages

    does it show up in meminfo output after running
    Code:
    sysctl -p
    i forgot to add that command to hptweaks.sh heh
     
  5. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    Do not have redis server installd at all.
    Does not show up in meminfo output after running
    Code:
    sysctl -p
    either.
    Note it does not reach
    Code:
    echo "vm.nr_hugepages=$NRHUGEPAGES_COUNT" >> /etc/sysctl.conf
    vm.nr_hugepages does not get added to sysctl.conf (refer to the cat in the post above);
     
  6. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    post to gist or pastebin output of this command
    Code:
    cd /usr/local/src/centminmod/tools
    bash -xv hptweaks.sh
     
  7. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
  8. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  9. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    That's weird, still no output. Double checked if the file got updated.
     
  10. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    another pastebin posted run for
    Code:
    cd /usr/local/src/centminmod/tools
    bash -xv hptweaks.sh
    
     
  11. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
  12. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    kernel setting is still set at [never] = disabled
    Code:
    ++ cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
    ++ grep -o '\[.*\]'
    + HP_CHECK='[never]'
    output of
    Code:
    cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
    is still set to [never] on your system = disabled

    tools/hptweak.sh doesn't enable transparent huge pages if disabled = [never], it only configures settings if transparent huge pages is detected as enabled = [always]
     
  13. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    I am aware of that actually, was testing if it'd output a message saying so. ;)
     
  14. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    oh yeah it should of said something you're right there !

    weird !

    however, doesn't matter that much as the intention is to call this hptweaks.sh from inc/zendopcache_tweaks.inc as part of PHP compilation and huge pages detection process so folks won't be looking for specific output of hptweaks.sh

    edit: integrated hptweaks.sh into inc/zendopcache_tweaks.inc so PHP recompile should trigger the detection and dynamic allocation for vm.nr_hugepages and max locked memory system limits but only for centos 7 with huge pages support for now (not centos 6)
     
    Last edited: Dec 18, 2015
  15. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    seems to work on live centos 7 virtualbox guest OS test
    Code:
    [root@centos7 centminmod]# tools/hptweaks.sh
    
    transparent huge pages not supported
    no tweaks needed
     
  16. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    Works on my CentOS 7 server aswell but not on my CentOS 6 one.
    Not sure if it is OS related or an issue on my end though.
     
  17. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Strange as the if else routine just checks for the existence of /sys/kernel/mm/redhat_transparent_hugepage/enabled and not the OS version centminmod/hptweaks.sh at 123.09beta01 · centminmod/centminmod · GitHub

    and that it's [always] set

    ah found the bug I should have this line positioned outside of the if routine
    Code:
    HP_CHECK=$(cat /sys/kernel/mm/redhat_transparent_hugepage/enabled | grep -o '\[.*\]')
     
  18. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    Found it. Seems like it goes in here:
    Code:
    # check if redis installed as redis server requires huge pages disabled
    if [[ ! "$(rpm -ql redis | grep not)" ]]; then
      if [[ -f /sys/kernel/mm/redhat_transparent_hugepage/enabled ]]; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
      fi
    fi
    
    And sets it to never eventhough I haven't redis server installed at all.
     
  19. eva2000

    eva2000 Administrator Staff Member

    54,519
    12,211
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,780
    Local Time:
    9:03 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Last edited: Dec 18, 2015
  20. Ahmad

    Ahmad Active Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    12:03 AM
    1.9.9
    10.1.10
    Was running standalone, I will test again now :)