MariaDB MYSQL server down :-(

First standard response so everyone else can read - Centmin Mod is provide as is, so short of scripted related bugs or issues, so troubleshooting is left to the Centmin Mod user to deal with. So I do not provide any free support for such.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

and quick check of your log - you ran out of disk space during innodb table recovery - make sure you have enough disk free space = at least 5x times amount of free disk space as your largest innodb table size

Code:

150615 20:18:52 [ERROR] mysqld: Error writing file '/var/lib/mysql/server64.phcorner.net.pid' (Errcode: 28 "No space left on device")
150615 20:18:52 [ERROR] Can't start server: can't create PID file: No space left on device

 

depends on what you have on server

Code:

du -h --max-depth=1 /
du -h --max-depth=1 /home
du -h --max-depth=1 /usr
du -h --max-depth=1 /var

You can empty out /svr-setup and empty ccache cache (via ccache -c) as most files get re-downloaded when needed from centmin.sh runs and ccache gets rebuilt when used. Can also empty yum caches too

Code:

df -h
rm -rf /svr-setup/*
ccache -c
yum clean all
rm -rf /var/cache/yum/*
df -h

Code:

ccache -h
Usage:
    ccache [options]
    ccache compiler [compiler options]
    compiler [compiler options]          (via symbolic link)

Options:
    -c, --cleanup         delete old files and recalculate size counters
                          (normally not needed as this is done automatically)
    -C, --clear           clear the cache completely (except configuration)
    -F, --max-files=N     set maximum number of files in cache to N (use 0 for
                          no limit)
    -M, --max-size=SIZE   set maximum size of cache to SIZE (use 0 for no
                          limit); available suffixes: k, M, G, T (decimal) and
                          Ki, Mi, Gi, Ti (binary); default suffix: G
    -o, --set-config=K=V  set configuration key K to value V
    -p, --print-config    print current configuration options
    -s, --show-stats      show statistics summary
    -z, --zero-stats      zero statistics counters

    -h, --help            print this help text
    -V, --version         print version and copyright information

See also <http://ccache.samba.org>.

innodb tables use much more memory and disk space than myisam so you need to account for that

 

what about

Code:

df -h

 

non-optimal disk partitioning there ! / which /var resides on only has 50GB partition with 1.8GB available

/home has 626GB so you can move entire mysql data directory to /home/mysql instead

commands to run to move to /home/mysql - obviously mysqladmin command might not work as your mysql server is already down heh but put it in as reference for basic datadir moves

Code:

service nginx stop; echo "prevent front side errors showing"
mysqladmin flush-tables; sleep 60; echo "wait for 60 seconds"
service mysql stop
mkdir -p /home/mysql
cp -a /etc/my.cnf /etc/my.cnf-varlibmysql.bak
sed -i 's|datadir=\/var\/lib\/mysql|datadir=\/home\/mysql|' /etc/my.cnf
sed -i 's|slow_query_log_file=\/var\/lib\/mysql|slow_query_log_file=\/home\/mysql|' /etc/my.cnf

make sure datadir variable etc in /etc/my.cnf is changed first

Code:

grep '\/var\/lib\/mysql' /etc/my.cnf
grep '\/home\/mysql' /etc/my.cnf

i.e. defaults

Code:

grep '\/var\/lib\/mysql' /etc/my.cnf
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
slow_query_log_file=/var/lib/mysql/slowq.log
socket=/var/lib/mysql/mysql.sock

if changed proceed

Code:

\cp -Rpf /var/lib/mysql/* /home/mysql/
chown mysql:mysql /home/mysql
service mysql start; echo "do not reboot server or restart mysql as innodb might be doing recovery - check mysql error log for details"
service nginx start

rest would be up to you as I said I don't provide free troubleshooting and technical support for stuff like this

 

check mysql error log - if it's related to mysql.sock might need to edit /etc/my.cnf and change it back for the 2 instances of socket=/var/lib/mysql/mysql.sock

 

ah forgot

Code:

chown mysql:mysql /home/mysql

also comment out in /etc/my.cnf these 2 values for now

Code:

innodb_buffer_pool_dump_at_shutdown=1
innodb_buffer_pool_load_at_startup=1

 

again always check mysql error log

 

might apply

 

yeah i corrected my sed replacements above to

Code:

sed -i 's|datadir=\/var\/lib\/mysql|datadir=\/home\/mysql|' /etc/my.cnf
sed -i 's|slow_query_log_file=\/var\/lib\/mysql|slow_query_log_file=\/home\/mysql|' /etc/my.cnf