MariaDB MariaDB fail to start

today when i reboot my vps (centos 7 and nearly lasted centmin .09beta), i got error by start Mariadb

Code:

 
systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf, protecthome.conf
   Active: failed (Result: exit-code) since Thu 2016-11-10 10:24:04 UTC; 15s ago
  Process: 1475 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=226/NAMESPACE)
  Process: 11515 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=226/NAMESPACE)
.
.
.
Failed at step NAMESPACE spawning /bin/sh: No space left on device

here is my df -h command

---

Code:

# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg-root  408G   52G  336G  14% /
devtmpfs              32G     0   32G   0% /dev
tmpfs                 32G     0   32G   0% /dev/shm
tmpfs                 32G  9.0M   32G   1% /run
tmpfs                 32G     0   32G   0% /sys/fs/cgroup
/dev/sda1            243M  204M   27M  89% /boot
/dev/mapper/vg-tmp   976M  261M  649M  29% /tmp
tmpfs                6.3G     0  6.3G   0% /run/user/0

Btw, i fail to start centmin command too

 

Sorry, not vps. It's dedicate. I setup this server 31 days ago with .09beta lasted at this time. I forgot to update. When i reboot, this error happen.

Code:

# mysqladmin ver
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!

Code:

journalctl -u mariadb --no-pager
-- Logs begin at Thu 2016-11-10 10:20:47 UTC, end at Thu 2016-11-10 11:17:07 UTC. --
Nov 10 10:20:59 xxx systemd[1]: Starting MariaDB database server...
Nov 10 10:21:00 xxx systemd[1475]: Failed at step NAMESPACE spawning /bin/sh: No such file or directory
Nov 10 10:21:00 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
Nov 10 10:21:00 xxx systemd[1]: Failed to start MariaDB database server.
Nov 10 10:21:00 xxx systemd[1]: Unit mariadb.service entered failed state.
Nov 10 10:21:00 xxx systemd[1]: mariadb.service failed.
Nov 10 10:23:17 xxx systemd[1]: Starting MariaDB database server...
Nov 10 10:23:17 xxx systemd[11483]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
Nov 10 10:23:17 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
Nov 10 10:23:17 xxx systemd[1]: Failed to start MariaDB database server.
Nov 10 10:23:17 xxx systemd[1]: Unit mariadb.service entered failed state.
Nov 10 10:23:17 xxx systemd[1]: mariadb.service failed.
Nov 10 10:24:04 xxx systemd[1]: Starting MariaDB database server...
Nov 10 10:24:04 xxx systemd[11515]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
Nov 10 10:24:04 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
Nov 10 10:24:04 xxx systemd[1]: Failed to start MariaDB database server.
Nov 10 10:24:04 xxx systemd[1]: Unit mariadb.service entered failed state.
Nov 10 10:24:04 xxx systemd[1]: mariadb.service failed.
Nov 10 11:11:54 xxx systemd[1]: Starting MariaDB database server...
Nov 10 11:11:54 xxx systemd[12230]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
Nov 10 11:11:54 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
Nov 10 11:11:54 xxx systemd[1]: Failed to start MariaDB database server.
Nov 10 11:11:54 xxx systemd[1]: Unit mariadb.service entered failed state.
Nov 10 11:11:54 xxx systemd[1]: mariadb.service failed.

Yes, it's contain.

Code:

[Service]
ProtectHome=false

Code:

cat /usr/lib/systemd/system/mariadb.service | grep Priv
PrivateNetwork=false
# NoNewPrivileges=true
PrivateDevices=true
# LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.
PrivateTmp=false
You have new mail in /var/spool/mail/root

Code:

egrep -rn 'Priv|Protect' /usr/lib/systemd/system/
/usr/lib/systemd/system/cyrus-imapd.service:11:PrivateTmp=true
/usr/lib/systemd/system/mariadb.service:36:PrivateNetwork=false
/usr/lib/systemd/system/mariadb.service:49:ProtectSystem=full
/usr/lib/systemd/system/mariadb.service:52:# NoNewPrivileges=true
/usr/lib/systemd/system/mariadb.service:54:PrivateDevices=true
/usr/lib/systemd/system/mariadb.service:57:ProtectHome=true
/usr/lib/systemd/system/mariadb.service:118:# LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.
/usr/lib/systemd/system/mariadb.service:119:PrivateTmp=false
/usr/lib/systemd/system/brandbot.service:6:PrivateTmp=true
/usr/lib/systemd/system/systemd-timedated.service:18:PrivateTmp=yes
/usr/lib/systemd/system/systemd-timedated.service:19:ProtectSystem=yes
/usr/lib/systemd/system/systemd-timedated.service:20:ProtectHome=yes
/usr/lib/systemd/system/systemd-localed.service:18:PrivateTmp=yes
/usr/lib/systemd/system/systemd-localed.service:19:PrivateDevices=yes
/usr/lib/systemd/system/systemd-localed.service:20:PrivateNetwork=yes
/usr/lib/systemd/system/systemd-localed.service:21:ProtectSystem=yes
/usr/lib/systemd/system/systemd-localed.service:22:ProtectHome=yes
/usr/lib/systemd/system/systemd-machined.service:20:PrivateTmp=yes
/usr/lib/systemd/system/systemd-machined.service:21:PrivateDevices=yes
/usr/lib/systemd/system/systemd-machined.service:22:PrivateNetwork=yes
/usr/lib/systemd/system/systemd-machined.service:23:ProtectSystem=full
/usr/lib/systemd/system/systemd-machined.service:24:ProtectHome=yes
/usr/lib/systemd/system/systemd-hostnamed.service:18:PrivateTmp=yes
/usr/lib/systemd/system/systemd-hostnamed.service:19:PrivateDevices=yes
/usr/lib/systemd/system/systemd-hostnamed.service:20:PrivateNetwork=yes
/usr/lib/systemd/system/systemd-hostnamed.service:21:ProtectSystem=yes
/usr/lib/systemd/system/systemd-hostnamed.service:22:ProtectHome=yes
/usr/lib/systemd/system/named.service:22:PrivateTmp=true
/usr/lib/systemd/system/garb.service:21:PrivateTmp=false
/usr/lib/systemd/system/ntpd.service:9:PrivateTmp=true

 

After reboot, i fail to start centmin command too.

 

Nothing happen when type centmin

 

i get error with this command.

Code:

mysqladmin var | grep tmpdir | tr -s ' '
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!

 

Code:

[client]
socket=/var/lib/mysql/mysql.sock

[mysql]
max_allowed_packet = 128M

[mysqld]
ignore_db_dirs=cmsetiofiotest
local-infile=0
ignore_db_dirs=lost+found
character-set-server=utf8
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

#bind-address=127.0.0.1
# optimized my.cnf for MariaDB 5.5.x
# by eva2000
# vbtechsupport.com

tmpdir=/home/mysqltmp

innodb=ON
#skip-federated
##skip-pbxt
##skip-pbxt_statistics
#skip-archive
#skip-name-resolve
#old_passwords
back_log = 1024
max_connections = 1500
key_buffer_size = 1024M
myisam_sort_buffer_size = 1024M
myisam_max_sort_file_size = 10240M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 2M
table_definition_cache = 8192
table_open_cache = 8192
thread_cache_size = 384
wait_timeout = 1800
connect_timeout = 10
tmp_table_size = 1536M
max_heap_table_size = 1536M
max_allowed_packet = 128M
max_seeks_for_key = 1000
group_concat_max_len = 1024
max_length_for_sort_data = 1024
net_buffer_length = 16384
max_connect_errors = 100000
concurrent_insert = 2
read_rnd_buffer_size = 512K
bulk_insert_buffer_size = 8M
# query_cache boost for MariaDB >10.1.2+
# https://community.centminmod.com/posts/30811/
query_cache_limit = 1536K
query_cache_size = 256M
query_cache_type = 1
query_cache_min_res_unit = 2K
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
default-storage-engine = InnoDB

log_warnings=1
slow_query_log=0
long_query_time=1
slow_query_log_file=/var/lib/mysql/slowq.log
log-error=/var/log/mysqld.log

# innodb settings
innodb_large_prefix=1
innodb_purge_threads = 4
innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_open_files = 2000
innodb_data_file_path= ibdata1:10M:autoextend
innodb_buffer_pool_size = 12288M

## https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-system-variables/#innodb_buffer_pool_instances
innodb_buffer_pool_instances=12

innodb_log_files_in_group = 2
innodb_log_file_size = 1024M
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 2
#innodb_thread_concurrency = 68
innodb_lock_wait_timeout=50
innodb_flush_method = O_DIRECT
innodb_support_xa=1

# 200 * # DISKS
innodb_io_capacity = 600
innodb_io_capacity_max = 1200
innodb_read_io_threads = 16
innodb_write_io_threads = 16
innodb_flush_neighbors = 0

# mariadb settings
[mariadb]
#thread-handling = pool-of-threads
#thread-pool-size= 20
#mysql --port=3307 --protocol=tcp
#extra-port=3307
#extra-max-connections=1

userstat = 0
key_cache_segments = 1
aria_group_commit = none
aria_group_commit_interval = 0
aria_log_file_size = 1024M
aria_log_purge_type = immediate
aria_pagecache_buffer_size = 1024M
aria_sort_buffer_size = 256M

[mariadb-5.5]
innodb_file_format = Barracuda
innodb_file_per_table = 1

#ignore_db_dirs=
query_cache_strip_comments=0

innodb_read_ahead = linear
innodb_adaptive_flushing_method = estimate
innodb_flush_neighbor_pages = 1
innodb_stats_update_need_lock = 0
innodb_log_block_size = 512

log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk

[mysqld_safe]
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
#nice = -5
open-files-limit = 8192

[mysqldump]
quick
max_allowed_packet = 128M

[myisamchk]
tmpdir=/home/mysqltmp
key_buffer = 1536M
sort_buffer = 384M
read_buffer = 384M
write_buffer = 384M

[mysqlhotcopy]
interactive-timeout

[mariadb-10.0]
innodb_file_format = Barracuda
innodb_file_per_table = 1

# 2 variables needed to switch from XtraDB to InnoDB plugins
#plugin-load=ha_innodb
#ignore_builtin_innodb

## MariaDB 10 only save and restore buffer pool pages
## warm up InnoDB buffer pool on server restarts
#innodb_buffer_pool_dump_at_shutdown=1
#innodb_buffer_pool_load_at_startup=1
innodb_buffer_pool_populate=0
## Disabled settings
performance_schema=OFF
innodb_stats_on_metadata=OFF
innodb_sort_buffer_size=2M
innodb_online_alter_log_max_size=128M
query_cache_strip_comments=0
log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk

[mariadb-10.1]
innodb_file_format = Barracuda
innodb_file_per_table = 1

## wsrep specific
# wsrep_on=OFF
# wsrep_provider
# wsrep_cluster_address
# binlog_format=ROW
# default_storage_engine=InnoDB
# innodb_autoinc_lock_mode=2
# innodb_doublewrite=1
# query_cache_size=0

# 2 variables needed to switch from XtraDB to InnoDB plugins
#plugin-load=ha_innodb
#ignore_builtin_innodb

## MariaDB 10 only save and restore buffer pool pages
## warm up InnoDB buffer pool on server restarts
#innodb_buffer_pool_dump_at_shutdown=1
#innodb_buffer_pool_load_at_startup=1
innodb_buffer_pool_populate=0
## Disabled settings
performance_schema=OFF
innodb_stats_on_metadata=OFF
innodb_sort_buffer_size=2M
innodb_online_alter_log_max_size=128M
query_cache_strip_comments=0
log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk

# Defragmenting unused space on InnoDB tablespace
innodb_defragment=1
innodb_defragment_n_pages=7
innodb_defragment_stats_accuracy=0
innodb_defragment_fill_factor_n_recs=20
innodb_defragment_fill_factor=0.9
innodb_defragment_frequency=40

 

How to backup database without mysql running ? Can i copy all database files ?

 

Code:

yum -q list installed MariaDB-Server
Installed Packages
MariaDB-server.x86_64                                                                                              10.1.18-1.el7.centos  

Still cant start mysql service after add # to my.cnf

"Failed at step NAMESPACE spawning /bin/sh: No space left on device" Not sure how my server no space left ?

 

.centmin

Before show menu i see alot of error message

Code:

./centmin.sh

inc/cpcheck.inc: line 494: cannot create temp file for here-document: No space left on device
inc/cpcheck.inc: line 595: cannot create temp file for here-document: No space left on device
inc/cpcheck.inc: line 639: cannot create temp file for here-document: No space left on device
inc/jemalloc.inc: line 9: cannot create temp file for here-document: No space left on device
Cannot create temporary file in /tmp/: No space left on device
inc/jemalloc.inc: line 1: 13162 Aborted                 gcc "$JEMSTATS_CFILE" -o ex_stats_print -I${JEMALLOC_PATH}/include -L${JEMALLOC_PATH}/lib -Wl,-rpath,${JEMALLOC_PATH}/lib -ljemalloc
inc/cpcheck.inc: line 1291: cannot create temp file for here-document: No space left on device

 

Code:

df -hT
Filesystem          Type      Size  Used Avail Use% Mounted on
/dev/mapper/vg-root ext4      408G   40G  348G  11% /
devtmpfs            devtmpfs   32G     0   32G   0% /dev
tmpfs               tmpfs      32G     0   32G   0% /dev/shm
tmpfs               tmpfs      32G   17M   32G   1% /run
tmpfs               tmpfs      32G     0   32G   0% /sys/fs/cgroup
/dev/sda1           ext2      243M  204M   27M  89% /boot
/dev/mapper/vg-tmp  ext4      976M  261M  649M  29% /tmp
tmpfs               tmpfs     6.3G     0  6.3G   0% /run/user/0

Code:

du -sh /tmp/
258M    /tmp/