Join the community today
Register Now

Bug in initd memcached script causing problems starting/restarting memcached

Discussion in 'Bug Reports' started by jeffwidman, May 18, 2015.

Tags:
  1. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    Hitting a weird bug with the initd script that manages memcached in .08beta2


    Whenever the file /etc/init.d/memcached is open in vim, I can stop the memcached service, but not restart it or start it. I played with it a bit and got the following

    Code:
    # Memcached running; admin stats page connects successfully.
    # Open /etc/init.d/memcached in vim. Without exiting vim, run the following:
    [root@cloud1 ~]# service memcached status
    Memcached server is running # stats page still connecting
    [root@cloud1 ~]# service memcached stop
    Memcached server stopped # stats page stops connecting
    [root@cloud1 ~]# service memcached status
    Memcached server is running # but it's not, I just stopped it, plus the stats page can't connect
    [root@cloud1 ~]# service memcached start
    Memcached server already running # no it's not
    [root@cloud1 ~]# service memcached status
    Memcached server is running # not it's not
    [root@cloud1 ~]# service memcached restart
    Memcached server stopped
    Memcached server already running # still not running
    # Close the init.d script in vim
    [root@cloud1 ~]# service memcached status
    Memcached server not running # now it correctly sees it's not running
    [root@cloud1 ~]# service memcached restart
    Memcached server already stopped # contrast with above, here it realizes it's not running
    Memcached server started # now it successfully boots
    
    I'm guessing opening the file in vim causes Linux to lock something somewhere that is preventing the status function in this script from realizing that memcached is actually stopped. So when I call `restart` or `start`, it checks the memcached status before trying to restart it, gets a report that memcached is already running (even though it's not), and exits without actually starting memcached.

    Any ideas how to fix the function that checks whether memcached is currently running?

    Normally wouldn't bother me, but this is causing my Ansible playbooks to report they've succeeded when in fact memcached is still stopped. I've got one Ansible task that edits the file, followed by another that restarts memcached. The second one is reporting success, but when I manually check memcached, it's stopped--again, I'm guessing the restart function stopped memcached, checked status, said "oh, it's already running" and exited without trying to start memcached.
     
    Last edited: May 18, 2015
  2. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  4. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    Yep, that fixed it. Thanks!
     
  5. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Thanks for bug report and confirmation of the fix :D
     
  6. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    Afraid I spoke too soon. That fixed the vim bug, but Ansible is still having problems.

    It's not worth worrying about, so I just swapped my Ansible playbook to use the command module instead of the service module. It works fine, it's just hacky though and implies there's some unexpected behavior in the memcached init.d script compared to how init.d scripts normally work. I even tried the service module with a sleep param per this thread, but no dice. I'm guessing writing init.d scripts is tricky business.
     
  7. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    if you can find out what processes are running via (ps ax | grep memcached) when ansible is running and doing it's thing, could probably narrow it down
     
  8. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  9. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    Nice!

    Just tried it manually before I try running Ansible plays against it, and it looks a little buggy still--doesn't actually stop memcache:
    Code:
    [root@cloud1 ~]# service memcached status
    Memcached server is running
    [root@cloud1 ~]# service memcached start
    Memcached server already running
    [root@cloud1 ~]# service memcached stop
    Memcached server stopped # This is wrong, my memcache stats page says it's still running
    cat: /var/run/memcached/memcached1.pid: No such file or directory # leftover debug output?
    [root@cloud1 ~]# service memcached start
    Memcached server already running # yep, it is
    
     
  10. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    revised code again at new /etc/init.d/memcached for centminmod.com with pid file (revision history at Revisions · memcached
     
  11. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    I just ran into an issue where the memcached gui page on "purge this server" wasn't clearing memcached. I manually ran "service memcached restart" while at the following path: /home/nginx/domains/domain.com/public/wp-content/cache/supercache/domain.com and got
    Code:
    shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    
    a quick `cd ~` followed by `service memcached restart` and it worked fine. But feels like something's off here in that init script if it can't be run in a subdirectory.
     
  12. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    weird never seen that one what do you get when you run
    Code:
    bash -x /etc/init.d/memcached restart
    when within that subdirectory
     
  13. jeffwidman

    jeffwidman Active Member

    152
    27
    28
    Dec 3, 2014
    Ratings:
    +51
    Local Time:
    1:04 AM
    Oops, stupid me. The problem is I was running tmux so jumping around different terminal windows and forgot that I'd deleted that directory in a different terminal window. So linux was complaining that I was trying to run the command from a non-existent directory. Sorry about that.

    BTW, tmux is awesome, don't know how I lived without this before!
     
  14. eva2000

    eva2000 Administrator Staff Member

    53,148
    12,110
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,645
    Local Time:
    6:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    ah ha ! :)