Join the community today
Become a Member

[Solved] WP - Not hitting Redis when using Unix Socket

Discussion in 'Bug Reports' started by Nik, Dec 28, 2017.

  1. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    • CentOS Version: CentOS 7 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.13.8
    • PHP Version Installed: 7.2.00
    • MariaDB MySQL Version Installed: 10.1.30
    • When was last time updated Centmin Mod code base ? : Dec 26, 2017
    • Persistent Config: LETSENCRYPT_DETECT='y'
    I'm a newcomer to Centminmod, great resource.

    I have 2 issues at hand:

    1. Unix Socket for Redis:
    I was playing around with a fresh install (on a KVM based VM), updated nginx and php to latest. I deployed redis cache and used option 22 to create my first vhost with WP which went well.

    Now trying to tweak it and upon editing redisbackend (and commenting out/in the unix socket as per your guide) I see that the cache isn't working. Moving it back to TCP gets it going again (I've tried variations of unix:/ unix:// and unix:/// to no avail):

    Code:
      #server 127.0.0.1:6379 weight=1 max_fails=3 fail_timeout=30s;
      server unix://tmp/redis.sock;
    
    I'm using curl -s https://mysite.com | tail -3 to check timestamps.

    As per redis log I see it's waiting on connections at redis.sock.

    Code:
    13:M 27 Dec 14:31:12.593 # Server initialized
    913:M 27 Dec 14:31:12.593 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    913:M 27 Dec 14:31:12.593 * Ready to accept connections
    913:M 27 Dec 14:31:12.593 * The server is now ready to accept connections at /tmp/redis.sock
    913:M 27 Dec 15:04:50.569 * 1 changes in 900 seconds. Saving...
    913:M 27 Dec 15:04:50.571 * Background saving started by pid 5764
    
    

    2. Transparent Huge Pages (THP) support

    Code:
    13:M 27 Dec 14:31:12.593 # Server initialized
    913:M 27 Dec 14:31:12.593 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    
    I tried using hptweaks.sh and it worked but the config did not survive a reboot. Should I add this to rc.local?

    @eva2000 Thanks for your help.
     
  2. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    You followed unix socket instructions for redis from Redis - How to install Redis server on Centmin Mod LEMP stack ?
    Unfortunately, I now remember that nginx-helper wp plugin itself doesn't support unix sockets as it looks for TCP port so you need to use TCP for Redis Nginx level caching. If you have other web apps supporting Redis unix sockets, you can leverage and use them there.

    If anyone is a PHP developer or understands the code Allow use of redis sockets instead of tcp:port · Issue #121 · rtCamp/nginx-helper · GitHub, you could try forking/modifying nginx-helper wp plugin to support unix sockets GitHub - rtCamp/nginx-helper: Nginx Helper for WordPress caching, permalinks & efficient file handling in multisite

    run hptweaks.sh is debug mode and copy and paste output to pastebin.com or gist.github.com and post link here
    Code (Text):
    cd /usr/local/src/centminmod/tools
    bash -x hptweaks.sh
    
     
  3. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Thanks, yes I followed instructions to the letter for redis sockets. I get that the nginx cache purge plugin makes Unix sockets useless but still doesn't explain why it won't look into cache. Not that I'll be using it so just curious.

    I'll get you the hwtweak debug soon as I'm back on a laptop.

    Thanks.
     
  4. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Thanks, yes I followed instructions to the letter for redis sockets. I get that the nginx cache purge plugin makes Unix sockets useless but still doesn't explain why it won't look into cache. Not that I'll be using it so just curious.

    I'll get you the hwtweak debug soon as I'm back on a laptop.

    Thanks.
     
  5. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Here's the gist:
    hptweaks · GitHub

    And yes it's added to rc.local so it appears it's starting up before. If I restart redis after a reboot that warning goes away.
     
    Last edited: Dec 28, 2017
  6. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    yeah a redis restart without reboot should do the same thing
     
  7. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    So looks like redis is starting up before rc.local is processed. Ideas on where I should stuff this?
     
  8. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    output for command
    Code (Text):
    systemd-analyze blame --no-pager
    
     
  9. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Code (Text):
              5.604s mariadb.service
              5.242s kdump.service
              4.252s csf.service
              3.574s network.service
              2.983s php-fpm.service
              2.015s nginx.service
              1.419s postfix.service
              1.382s cloud-init.service
              1.291s cloud-config.service
              1.225s cloud-final.service
               761ms cloud-init-local.service
               619ms lfd.service
               344ms dev-vda1.device
               290ms tuned.service
               244ms memcached.service
               201ms rhel-import-state.service
               150ms auditd.service
               146ms systemd-vconsole-setup.service
               134ms nfs-config.service
               125ms proc-fs-nfsd.mount
                91ms iptables.service
                84ms systemd-random-seed.service
                81ms rhel-readonly.service
                70ms systemd-udev-trigger.service
                65ms polkit.service
                55ms pure-ftpd.service
                52ms sshd.service
                47ms chronyd.service
                43ms systemd-journald.service
                41ms dev-vdb.swap
                34ms ip6tables.service
                32ms systemd-tmpfiles-setup.service
                31ms sysstat.service
                30ms redis.service
                30ms dev-mqueue.mount
                29ms rsyslog.service
                27ms sys-kernel-debug.mount
                25ms systemd-user-sessions.service
                24ms rc-local.service
                23ms rhel-dmesg.service
                23ms kmod-static-nodes.service
                23ms systemd-udevd.service
                23ms systemd-sysctl.service
                22ms systemd-logind.service
                22ms systemd-tmpfiles-setup-dev.service
                22ms dev-hugepages.mount
                20ms gssproxy.service
                19ms systemd-remount-fs.service
                18ms rpc-statd-notify.service
                18ms systemd-journal-flush.service
                 9ms systemd-fsck-root.service
                 4ms systemd-update-utmp-runlevel.service
                 3ms systemd-update-utmp.service
                 2ms sys-kernel-config.mount
    
    


    Actually restarting redis this time after a reboot didn't clear the warning.
     
  10. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    contents of /etc/rc.local ?
    Code (Text):
    cat /etc/rc.local
    
     
  11. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    stranged tested it on my centos 7.4 server, despite /etc/rc.local showing 'never' for THP, the actual value is set to always on server reboot
    Code (Text):
    cat /etc/rc.local               
    #!/bin/bash
    # THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
    #
    # It is highly advisable to create own systemd services or udev rules
    # to run scripts during boot instead of using this file.
    #
    # In contrast to previous versions due to parallel execution during boot
    # this script will NOT be run after all other services.
    #
    # Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
    # that this script will be executed during boot.
    
    touch /var/lock/subsys/local
    ulimit -n 262144
    echo 65536 > /sys/module/nf_conntrack/parameters/hashsize
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    

    on server reboot should be set to never but set to always!
    Code (Text):
    cat /sys/kernel/mm/transparent_hugepage/enabled
    [always] madvise never
    

    but other rc.local setting is set
    Code (Text):
    cat /sys/module/nf_conntrack/parameters/hashsize
    65536
    

    checking to see if tuned is auto overriding the value
    Code (Text):
    tuned-adm active
    Current active profile: virtual-guest
    

    Code (Text):
    cat /usr/lib/tuned/virtual-guest/tuned.conf
    #
    # tuned configuration
    #
    
    [main]
    summary=Optimize for running inside a virtual guest
    include=throughput-performance
    
    [sysctl]
    # If a workload mostly uses anonymous memory and it hits this limit, the entire
    # working set is buffered for I/O, and any more write buffering would require
    # swapping, so it's time to throttle writes until I/O can catch up.  Workloads
    # that mostly use file mappings may be able to use even higher values.
    #
    # The generator of dirty data starts writeback at this percentage (system default
    # is 20%)
    vm.dirty_ratio = 30
    
    # Filesystem I/O is usually much more efficient than swapping, so try to keep
    # swapping low.  It's usually safe to go even lower than this on systems with
    # server-grade storage.
    vm.swappiness = 30
    

    virtual-guest includes throughput-performance tuned profile too
    Code (Text):
    cat /usr/lib/tuned/throughput-performance/tuned.conf
    #
    # tuned configuration
    #
    
    [main]
    summary=Broadly applicable tuning that provides excellent performance across a variety of common server workloads
    
    [cpu]
    governor=performance
    energy_perf_bias=performance
    min_perf_pct=100
    
    [disk]
    readahead=>4096
    
    [sysctl]
    # ktune sysctl settings for rhel6 servers, maximizing i/o throughput
    #
    # Minimal preemption granularity for CPU-bound tasks:
    # (default: 1 msec#  (1 + ilog(ncpus)), units: nanoseconds)
    kernel.sched_min_granularity_ns = 10000000
    
    # SCHED_OTHER wake-up granularity.
    # (default: 1 msec#  (1 + ilog(ncpus)), units: nanoseconds)
    #
    # This option delays the preemption effects of decoupled workloads
    # and reduces their over-scheduling. Synchronous workloads will still
    # have immediate wakeup/sleep latencies.
    kernel.sched_wakeup_granularity_ns = 15000000
    
    # If a workload mostly uses anonymous memory and it hits this limit, the entire
    # working set is buffered for I/O, and any more write buffering would require
    # swapping, so it's time to throttle writes until I/O can catch up.  Workloads
    # that mostly use file mappings may be able to use even higher values.
    #
    # The generator of dirty data starts writeback at this percentage (system default
    # is 20%)
    vm.dirty_ratio = 40
    
    # Start background writeback (via writeback threads) at this percentage (system
    # default is 10%)
    vm.dirty_background_ratio = 10
    
    # PID allocation wrap value.  When the kernel's next PID value
    # reaches this value, it wraps back to a minimum PID value.
    # PIDs of value pid_max or larger are not allocated.
    #
    # A suggested value for pid_max is 1024 * <# of cpu cores/threads in system>
    # e.g., a box with 32 cpus, the default of 32768 is reasonable, for 64 cpus,
    # 65536, for 4096 cpus, 4194304 (which is the upper limit possible).
    #kernel.pid_max = 65536
    
    # The swappiness parameter controls the tendency of the kernel to move
    # processes out of physical memory and onto the swap disk.
    # 0 tells the kernel to avoid swapping processes out of physical memory
    # for as long as possible
    # 100 tells the kernel to aggressively swap processes out of physical memory
    # and move them to swap cache
    vm.swappiness=10
    

    not specifically seeing a setting for transparent_hugepages or is it enabled by default and needs disabling specifically ?
     
  12. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    This is strange. Setting disappeared from rc.local all on its own.

    Code:
    #!/bin/bash
    # THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
    #
    # It is highly advisable to create own systemd services or udev rules
    # to run scripts during boot instead of using this file.
    #
    # In contrast to previous versions due to parallel execution during boot
    # this script will NOT be run after all other services.
    #
    # Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
    # that this script will be executed during boot.
    
    touch /var/lock/subsys/local
    ulimit -n 262144
    echo 65536 > /sys/module/nf_conntrack/parameters/hashsize
    
    
     
  13. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    strange indeed

    what output do you get for
    Code (Text):
    tuned-adm active
    
     
  14. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Current active profile: virtual-guest
     
  15. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    will have to investigate more as tried several methods and all on reboot set it back to 'always' instead of 'never'. Only haven't tried override via tuned as want to avoid that as not all systems have tuned enabled.
     
  16. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    @Nik I just updated centmin mod 123.09beta01's tools/hptweaks.sh routine to add some workaround code to do the following on CentOS 7 systems

    1. ensure redis service starts after rc.local
    2. ensure 'never' is set in /sys/kernel/mm/transparent_hugepage/enabled on system reboots - did this by adding a custom systemd service called disable-thp that runs on reboot to set it

    on most recent centmin mod 123.09beta01 you can update via cmupdate command (on old versions need to use centmin.sh menu option 23 submenu option 2).
    Code (Text):
    cmupdate
    cmdir
    tools/hptweaks.sh
    

    then test it out by rebooting server and checking redis logs to see the transparent huge pages warning is present
    Code (Text):
    reboot
    

    Code (Text):
    tail -10 /var/log/redis/redis.log
    


    redis service after rc.local
    Code (Text):
    systemd-analyze blame --no-pager
             38.514s network.service
              1.438s docker.service
               915ms mariadb.service
               912ms kdump.service
               445ms csf.service
               323ms postfix.service
               290ms lvm2-monitor.service
               287ms php56-php-fpm.service
               287ms php71-php-fpm.service
               284ms php72-php-fpm.service
               273ms php70-php-fpm.service
               272ms nginx.service
               212ms php-fpm.service
               157ms dev-md1.device
               134ms lfd.service
               109ms auditd.service
               107ms pmcd.service
               102ms tuned.service
                60ms systemd-logind.service
                59ms polkit.service
                58ms chronyd.service
                53ms rhel-dmesg.service
                53ms systemd-vconsole-setup.service
                49ms systemd-udev-trigger.service
                49ms iptables.service
                45ms sysstat.service
                43ms systemd-tmpfiles-setup-dev.service
                39ms rhel-readonly.service
                37ms systemd-user-sessions.service
                37ms mdmonitor.service
                25ms sshd.service
                24ms ip6tables.service
                22ms rc-local.service
                21ms redis.service
                21ms systemd-journal-flush.service
                19ms xinetd.service
                17ms systemd-sysctl.service
                15ms memcached.service
                14ms systemd-remount-fs.service
                14ms kmod-static-nodes.service
                13ms systemd-udevd.service
                12ms rhel-import-state.service
                12ms systemd-fsck@dev-md2.service
                10ms systemd-journald.service
                10ms systemd-fsck-root.service
                10ms rsyslog.service
                 9ms dev-mqueue.mount
                 9ms sys-kernel-debug.mount
                 9ms dev-hugepages.mount
                 8ms systemd-tmpfiles-setup.service
                 7ms systemd-update-utmp-runlevel.service
                 6ms systemd-random-seed.service
                 6ms systemd-update-utmp.service
                 4ms dev-sdb3.swap
                 3ms home.mount
                 3ms dev-sda3.swap
                 3ms systemd-readahead-replay.service
                 2ms systemd-readahead-collect.service
                 1ms tmp.mount
                 1ms systemd-readahead-done.service
               973us sys-kernel-config.mount
    
     
  17. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Brilliant. Should I take it out of rc.local and rerun hptweaks from fresh again?
     
  18. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    just leave it in /etc/rc.local as it may get picked up by redis now as the redis service runs after rc.local
     
    • Like Like x 1
  19. Nik

    Nik New Member

    15
    1
    3
    Dec 7, 2015
    Ratings:
    +3
    Local Time:
    10:29 AM
    Seems to work on reboot. Thank you for the quick help.

    Code (Text):
      3.445s network.service
              1.566s kdump.service
               958ms mariadb.service
               593ms csf.service
               530ms cloud-init-local.service
               502ms cloud-init.service
               417ms postfix.service
               334ms cloud-final.service
               311ms cloud-config.service
               264ms php-fpm.service
               261ms lfd.service
               243ms nginx.service
               161ms dev-vda1.device
               151ms systemd-vconsole-setup.service
               147ms tuned.service
               138ms auditd.service
               103ms proc-fs-nfsd.mount
                77ms iptables.service
                59ms rhel-readonly.service
                57ms systemd-udev-trigger.service
                54ms systemd-tmpfiles-setup-dev.service
                51ms chronyd.service
                45ms polkit.service
                45ms systemd-tmpfiles-setup.service
                45ms dev-vdb.swap
                40ms pure-ftpd.service
                38ms gssproxy.service
                37ms memcached.service
                34ms ip6tables.service
                33ms rhel-import-state.service
                33ms systemd-journald.service
                31ms systemd-logind.service
                30ms dev-mqueue.mount
                27ms sys-kernel-debug.mount
                26ms kmod-static-nodes.service
                23ms rc-local.service
                22ms rhel-dmesg.service
                21ms systemd-udevd.service
                21ms sshd.service
                19ms systemd-user-sessions.service
                18ms redis.service
                17ms systemd-remount-fs.service
                16ms sysstat.service
                12ms systemd-sysctl.service
                12ms rsyslog.service
                11ms systemd-journal-flush.service
                11ms nfs-config.service
                 8ms rpc-statd-notify.service
                 8ms systemd-random-seed.service
                 7ms systemd-fsck-root.service
                 5ms dev-hugepages.mount
                 3ms systemd-update-utmp.service
                 3ms systemd-update-utmp-runlevel.service
                 3ms sys-kernel-config.mount
    
    
     
  20. eva2000

    eva2000 Administrator Staff Member

    31,638
    7,029
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,634
    Local Time:
    1:29 AM
    Nginx 1.13.x
    MariaDB 5.5
    excellent :D