Want to subscribe to topics you're interested in?
Become a Member

Redis Crash

Discussion in 'Other Centmin Mod Installed software' started by Matt Williams, Jan 1, 2016.

  1. Matt Williams

    Matt Williams WordPress Fanatic

    537
    104
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +157
    Local Time:
    5:53 PM
    latest
    10
    Today, Redis decided to just die on one of my VM's with Centminmod beta09 - How do I fix or uninstall redis and re-install? The only way the WP sites stay online is if I type redis-server to accept connections but the second I exit - it goes down again. I had to disconnect the nginx helper and redis object cache from the sites so the sites would stay online.


    Code:
    [root@ngx ~]# service redis status
    Redirecting to /bin/systemctl status  redis.service
    redis.service - Redis persistent key-value database
       Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled)
      Drop-In: /etc/systemd/system/redis.service.d
               └─limit.conf
       Active: failed (Result: exit-code) since Thu 2015-12-31 18:11:49 UTC; 5s ago
      Process: 3671 ExecStop=/usr/bin/redis-shutdown (code=exited, status=1/FAILURE)
      Process: 3667 ExecStart=/usr/bin/redis-server /etc/redis.conf --daemonize no (code=exited, status=1/FAILURE)
    Main PID: 3667 (code=exited, status=1/FAILURE)
    
    Dec 31 18:11:49 ngx.domain.com systemd[1]: Started Redis persistent key-value database.
    Dec 31 18:11:49 ngx.domain.com systemd[1]: redis.service: main process exited, code=exited, status=1/FAILURE
    Dec 31 18:11:49 ngx.domain.com redis-shutdown[3671]: Could not connect to Redis at 127.0.0.1:6379: Connection refused
    Dec 31 18:11:49 ngx.domain.com systemd[1]: redis.service: control process exited, code=exited status=1
    Dec 31 18:11:49 ngx.domain.com systemd[1]: Unit redis.service entered failed state.
    
     
  2. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    check redis log for clues /var/log/redis/redis.log
     
  3. Matt Williams

    Matt Williams WordPress Fanatic

    537
    104
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +157
    Local Time:
    5:53 PM
    latest
    10
    I see:

     
  4. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    do what it suggests with redis-check-aof --fix and name of the appendonly file which by default is appendonly.aof at /var/lib/redis/appendonly.aof
    Code:
    grep appendonly /etc/redis.conf
    appendonly yes
    # The name of the append only file (default: "appendonly.aof")
    appendfilename "appendonly.aof"
     
  5. Matt Williams

    Matt Williams WordPress Fanatic

    537
    104
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +157
    Local Time:
    5:53 PM
    latest
    10
    so what exactly would I run? I ran - ./redis-check-aof --fix but it said "not found" what directory would I run that from?
     
  6. Xon

    Xon Active Member

    173
    61
    28
    Nov 16, 2015
    Ratings:
    +229
    Local Time:
    5:53 AM
    1.15.x
    MariaDB 10.3.x
    If you are using redis just for caching, try removing the contents of /var/lib/redis

    There may be a dump.rdb file or appendonly.aof in there.
     
  7. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah if it's just caching data, it can be lost and rebuilt anyway
    Code:
    ls -lah /var/lib/redis
    total 163M
    drwxr-xr-x.  2 redis redis   42 Jan  1 05:17 .
    drwxr-xr-x. 38 root  root  4.0K Jan  1 03:19 ..
    -rw-r--r--.  1 redis redis 163M Jan  1 05:02 appendonly.aof
    -rw-r--r--.  1 redis redis  32K Jan  1 05:17 dump.rdb
    Code:
    redis-check-aof /var/lib/redis/appendonly.aof
    AOF analyzed: size=170410940, ok_up_to=170410940, diff=0
    AOF is valid
    Code:
    redis-check-aof --fix /var/lib/redis/appendonly.aof
    AOF analyzed: size=170410940, ok_up_to=170410940, diff=0
    AOF is valid
     
  8. Matt Williams

    Matt Williams WordPress Fanatic

    537
    104
    43
    Nov 22, 2014
    Virginia, USA
    Ratings:
    +157
    Local Time:
    5:53 PM
    latest
    10
    That didn't work unfortunately. I think redis on this VM is done with lol!

    Anyway, Having another issue with Redis. in the redis.conf file, it has:

    But the issue is - they are still being cached
     
  9. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    should work that command what does output when run ?
    Code:
    redis-check-aof --fix /var/lib/redis/appendonly.aof
    as to caching did you configure and enable nginx-helper ?

    port for you should be 6379 for redis default port, i am using a redis load balanced cluster on port 19001

    upload_2016-1-1_19-32-55.png

    How are you confirming whether pages are cached ? you can check the nginx-cache key in redis cache in phpredisadmin. I don't see any of those urls in cache
     
  10. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest
    mine also crashed but all I have in the logs are

    Code:
    146:M 04 Feb 22:31:54.091 * 10 changes in 300 seconds. Saving...
    146:M 04 Feb 22:31:54.121 * Background saving started by pid 14620
    14620:C 04 Feb 22:32:02.661 * DB saved on disk
    14620:C 04 Feb 22:32:02.684 * RDB: 355 MB of memory used by copy-on-write
    146:M 04 Feb 22:32:02.761 * Background saving terminated with success
    just a bunch of those and no errors.
     
  11. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    how much memory is redis using ? you can grep on the fields of output in redis-cli info command
    Code:
    redis-cli info | egrep 'used_memory|mem_'
    used_memory:9255840
    used_memory_human:8.83M
    used_memory_rss:12763136
    used_memory_peak:12363208
    used_memory_peak_human:11.79M
    used_memory_lua:78848
    mem_fragmentation_ratio:1.38
    mem_allocator:jemalloc-3.6.0
     
  12. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest
    I think it didn't crash but I wonder why it stopped. Last time it was running was February 4.
     
  13. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest
    used_memory_human:449.62M

    I had to start redis to get this though.
     
  14. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    what version of redis ? 3.0.7 is latest

    so update too
    Code:
    yum update redis --enablerepo=remi --disableplugin=priorities
     
  15. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    how much memory does that server have ?
     
  16. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest
  17. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest
    1GB

    maxmemory = 450
     
  18. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Could be running up your memory limits on background saves Redis Persistence – Redis and FAQ – Redis did you set overcommit_memory=1 in TCP settings as outlined in 1st post at Redis - How to install Redis server on Centmin Mod LEMP stack | Centmin Mod Community ?

    from FAQ
     
  19. eva2000

    eva2000 Administrator Staff Member

    55,424
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,839
    Local Time:
    7:53 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  20. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:53 AM
    latest
    latest