Want to subscribe to topics you're interested in?
Become a Member

MariaDB MariaDB fail to start

Discussion in 'MariaDB & General MySQL news & discussions' started by YuchiRO, Nov 10, 2016.

  1. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    today when i reboot my vps (centos 7 and nearly lasted centmin .09beta), i got error by start Mariadb

    Code:
     
    systemctl status mariadb.service
    ● mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/mariadb.service.d
               └─migrated-from-my.cnf-settings.conf, protecthome.conf
       Active: failed (Result: exit-code) since Thu 2016-11-10 10:24:04 UTC; 15s ago
      Process: 1475 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=226/NAMESPACE)
      Process: 11515 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=226/NAMESPACE)
    .
    .
    .
    Failed at step NAMESPACE spawning /bin/sh: No space left on device
    
    here is my df -h command

    Code:
    # df -h
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/mapper/vg-root  408G   52G  336G  14% /
    devtmpfs              32G     0   32G   0% /dev
    tmpfs                 32G     0   32G   0% /dev/shm
    tmpfs                 32G  9.0M   32G   1% /run
    tmpfs                 32G     0   32G   0% /sys/fs/cgroup
    /dev/sda1            243M  204M   27M  89% /boot
    /dev/mapper/vg-tmp   976M  261M  649M  29% /tmp
    tmpfs                6.3G     0  6.3G   0% /run/user/0
    
    Btw, i fail to start centmin command too :(
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    What version of mariadb ? vps type ? kvm, xen or openvz ?
    when was last time you updated 123.09beta01 beta ? when was last time you ran centmin.sh menu ?
    Code (Text):
    mysqladmin ver

    from faq item 19 FAQ - CentminMod.com LEMP Nginx web stack for CentOS what's output for
    Code (Text):
    journalctl -u mariadb --no-pager

    but could be related to this bug which was fixed 4+ months ago MariaDB 10.1.16 Unable to restart fix | Centmin Mod Community

    double check if /etc/systemd/system/mariadb.service.d/protecthome.conf file exist and it's contents

    i.e.
    Code (Text):
    cat /etc/systemd/system/mariadb.service.d/protecthome.conf
    [Service]
    ProtectHome=false


    or this systemd bug Systemd fails to start a service with ProtectHome=yes and symlinks. · Issue #4502 · systemd/systemd · GitHub

    or this mariadb bug [MDEV-10925] systemctl makes all versions of mariadn fail to start - JIRA

    output for this command
    Code (Text):
    cat /usr/lib/systemd/system/mariadb.service | grep Priv

    and
    Code (Text):
    egrep -rn 'Priv|Protect|Restrict' /usr/lib/systemd/system/ 
     
  3. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    Sorry, not vps. It's dedicate. I setup this server 31 days ago with .09beta lasted at this time. I forgot to update. When i reboot, this error happen.

    Code:
    # mysqladmin ver
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    
    Code:
    journalctl -u mariadb --no-pager
    -- Logs begin at Thu 2016-11-10 10:20:47 UTC, end at Thu 2016-11-10 11:17:07 UTC. --
    Nov 10 10:20:59 xxx systemd[1]: Starting MariaDB database server...
    Nov 10 10:21:00 xxx systemd[1475]: Failed at step NAMESPACE spawning /bin/sh: No such file or directory
    Nov 10 10:21:00 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
    Nov 10 10:21:00 xxx systemd[1]: Failed to start MariaDB database server.
    Nov 10 10:21:00 xxx systemd[1]: Unit mariadb.service entered failed state.
    Nov 10 10:21:00 xxx systemd[1]: mariadb.service failed.
    Nov 10 10:23:17 xxx systemd[1]: Starting MariaDB database server...
    Nov 10 10:23:17 xxx systemd[11483]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
    Nov 10 10:23:17 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
    Nov 10 10:23:17 xxx systemd[1]: Failed to start MariaDB database server.
    Nov 10 10:23:17 xxx systemd[1]: Unit mariadb.service entered failed state.
    Nov 10 10:23:17 xxx systemd[1]: mariadb.service failed.
    Nov 10 10:24:04 xxx systemd[1]: Starting MariaDB database server...
    Nov 10 10:24:04 xxx systemd[11515]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
    Nov 10 10:24:04 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
    Nov 10 10:24:04 xxx systemd[1]: Failed to start MariaDB database server.
    Nov 10 10:24:04 xxx systemd[1]: Unit mariadb.service entered failed state.
    Nov 10 10:24:04 xxx systemd[1]: mariadb.service failed.
    Nov 10 11:11:54 xxx systemd[1]: Starting MariaDB database server...
    Nov 10 11:11:54 xxx systemd[12230]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
    Nov 10 11:11:54 xxx systemd[1]: mariadb.service: control process exited, code=exited status=226
    Nov 10 11:11:54 xxx systemd[1]: Failed to start MariaDB database server.
    Nov 10 11:11:54 xxx systemd[1]: Unit mariadb.service entered failed state.
    Nov 10 11:11:54 xxx systemd[1]: mariadb.service failed.
    Yes, it's contain.
    Code:
    [Service]
    ProtectHome=false
    Code:
    cat /usr/lib/systemd/system/mariadb.service | grep Priv
    PrivateNetwork=false
    # NoNewPrivileges=true
    PrivateDevices=true
    # LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.
    PrivateTmp=false
    You have new mail in /var/spool/mail/root
    
    Code:
    egrep -rn 'Priv|Protect' /usr/lib/systemd/system/
    /usr/lib/systemd/system/cyrus-imapd.service:11:PrivateTmp=true
    /usr/lib/systemd/system/mariadb.service:36:PrivateNetwork=false
    /usr/lib/systemd/system/mariadb.service:49:ProtectSystem=full
    /usr/lib/systemd/system/mariadb.service:52:# NoNewPrivileges=true
    /usr/lib/systemd/system/mariadb.service:54:PrivateDevices=true
    /usr/lib/systemd/system/mariadb.service:57:ProtectHome=true
    /usr/lib/systemd/system/mariadb.service:118:# LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.
    /usr/lib/systemd/system/mariadb.service:119:PrivateTmp=false
    /usr/lib/systemd/system/brandbot.service:6:PrivateTmp=true
    /usr/lib/systemd/system/systemd-timedated.service:18:PrivateTmp=yes
    /usr/lib/systemd/system/systemd-timedated.service:19:ProtectSystem=yes
    /usr/lib/systemd/system/systemd-timedated.service:20:ProtectHome=yes
    /usr/lib/systemd/system/systemd-localed.service:18:PrivateTmp=yes
    /usr/lib/systemd/system/systemd-localed.service:19:PrivateDevices=yes
    /usr/lib/systemd/system/systemd-localed.service:20:PrivateNetwork=yes
    /usr/lib/systemd/system/systemd-localed.service:21:ProtectSystem=yes
    /usr/lib/systemd/system/systemd-localed.service:22:ProtectHome=yes
    /usr/lib/systemd/system/systemd-machined.service:20:PrivateTmp=yes
    /usr/lib/systemd/system/systemd-machined.service:21:PrivateDevices=yes
    /usr/lib/systemd/system/systemd-machined.service:22:PrivateNetwork=yes
    /usr/lib/systemd/system/systemd-machined.service:23:ProtectSystem=full
    /usr/lib/systemd/system/systemd-machined.service:24:ProtectHome=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:18:PrivateTmp=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:19:PrivateDevices=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:20:PrivateNetwork=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:21:ProtectSystem=yes
    /usr/lib/systemd/system/systemd-hostnamed.service:22:ProtectHome=yes
    /usr/lib/systemd/system/named.service:22:PrivateTmp=true
    /usr/lib/systemd/system/garb.service:21:PrivateTmp=false
    /usr/lib/systemd/system/ntpd.service:9:PrivateTmp=true
    
     
  4. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    After reboot, i fail to start centmin command too.
     
  5. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    what errors and exact command used ?
     
  6. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    Nothing happen when type centmin
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    from df output
    Code (Text):
    /dev/mapper/vg-tmp   976M  261M  649M  29% /tmp
    

    post output for
    Code (Text):
    mysqladmin var | grep tmpdir | tr -s ' '
    

    Example
    Code (Text):
    mysqladmin var | grep tmpdir | tr -s ' '
    | innodb_tmpdir | |
    | slave_load_tmpdir | /home/mysqltmp |
    | tmpdir | /home/mysqltmp |
     
  8. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    i get error with this command.

    Code:
    mysqladmin var | grep tmpdir | tr -s ' '
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    Ooops post contents for /etc/my.cnf instead
    Code (Text):
    cat /etc/my.cnf
     
  10. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    Code:
    [client]
    socket=/var/lib/mysql/mysql.sock
    
    [mysql]
    max_allowed_packet = 128M
    
    [mysqld]
    ignore_db_dirs=cmsetiofiotest
    local-infile=0
    ignore_db_dirs=lost+found
    character-set-server=utf8
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    
    #bind-address=127.0.0.1
    # optimized my.cnf for MariaDB 5.5.x
    # by eva2000
    # vbtechsupport.com
    
    tmpdir=/home/mysqltmp
    
    innodb=ON
    #skip-federated
    ##skip-pbxt
    ##skip-pbxt_statistics
    #skip-archive
    #skip-name-resolve
    #old_passwords
    back_log = 1024
    max_connections = 1500
    key_buffer_size = 1024M
    myisam_sort_buffer_size = 1024M
    myisam_max_sort_file_size = 10240M
    join_buffer_size = 2M
    read_buffer_size = 2M
    sort_buffer_size = 2M
    table_definition_cache = 8192
    table_open_cache = 8192
    thread_cache_size = 384
    wait_timeout = 1800
    connect_timeout = 10
    tmp_table_size = 1536M
    max_heap_table_size = 1536M
    max_allowed_packet = 128M
    max_seeks_for_key = 1000
    group_concat_max_len = 1024
    max_length_for_sort_data = 1024
    net_buffer_length = 16384
    max_connect_errors = 100000
    concurrent_insert = 2
    read_rnd_buffer_size = 512K
    bulk_insert_buffer_size = 8M
    # query_cache boost for MariaDB >10.1.2+
    # https://community.centminmod.com/posts/30811/
    query_cache_limit = 1536K
    query_cache_size = 256M
    query_cache_type = 1
    query_cache_min_res_unit = 2K
    query_prealloc_size = 262144
    query_alloc_block_size = 65536
    transaction_alloc_block_size = 8192
    transaction_prealloc_size = 4096
    default-storage-engine = InnoDB
    
    log_warnings=1
    slow_query_log=0
    long_query_time=1
    slow_query_log_file=/var/lib/mysql/slowq.log
    log-error=/var/log/mysqld.log
    
    # innodb settings
    innodb_large_prefix=1
    innodb_purge_threads = 4
    innodb_file_format = Barracuda
    innodb_file_per_table = 1
    innodb_open_files = 2000
    innodb_data_file_path= ibdata1:10M:autoextend
    innodb_buffer_pool_size = 12288M
    
    ## https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-system-variables/#innodb_buffer_pool_instances
    innodb_buffer_pool_instances=12
    
    innodb_log_files_in_group = 2
    innodb_log_file_size = 1024M
    innodb_log_buffer_size = 32M
    innodb_flush_log_at_trx_commit = 2
    #innodb_thread_concurrency = 68
    innodb_lock_wait_timeout=50
    innodb_flush_method = O_DIRECT
    innodb_support_xa=1
    
    # 200 * # DISKS
    innodb_io_capacity = 600
    innodb_io_capacity_max = 1200
    innodb_read_io_threads = 16
    innodb_write_io_threads = 16
    innodb_flush_neighbors = 0
    
    # mariadb settings
    [mariadb]
    #thread-handling = pool-of-threads
    #thread-pool-size= 20
    #mysql --port=3307 --protocol=tcp
    #extra-port=3307
    #extra-max-connections=1
    
    userstat = 0
    key_cache_segments = 1
    aria_group_commit = none
    aria_group_commit_interval = 0
    aria_log_file_size = 1024M
    aria_log_purge_type = immediate
    aria_pagecache_buffer_size = 1024M
    aria_sort_buffer_size = 256M
    
    [mariadb-5.5]
    innodb_file_format = Barracuda
    innodb_file_per_table = 1
    
    #ignore_db_dirs=
    query_cache_strip_comments=0
    
    innodb_read_ahead = linear
    innodb_adaptive_flushing_method = estimate
    innodb_flush_neighbor_pages = 1
    innodb_stats_update_need_lock = 0
    innodb_log_block_size = 512
    
    log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
    
    [mysqld_safe]
    socket=/var/lib/mysql/mysql.sock
    log-error=/var/log/mysqld.log
    #nice = -5
    open-files-limit = 8192
    
    [mysqldump]
    quick
    max_allowed_packet = 128M
    
    [myisamchk]
    tmpdir=/home/mysqltmp
    key_buffer = 1536M
    sort_buffer = 384M
    read_buffer = 384M
    write_buffer = 384M
    
    [mysqlhotcopy]
    interactive-timeout
    
    [mariadb-10.0]
    innodb_file_format = Barracuda
    innodb_file_per_table = 1
    
    # 2 variables needed to switch from XtraDB to InnoDB plugins
    #plugin-load=ha_innodb
    #ignore_builtin_innodb
    
    ## MariaDB 10 only save and restore buffer pool pages
    ## warm up InnoDB buffer pool on server restarts
    #innodb_buffer_pool_dump_at_shutdown=1
    #innodb_buffer_pool_load_at_startup=1
    innodb_buffer_pool_populate=0
    ## Disabled settings
    performance_schema=OFF
    innodb_stats_on_metadata=OFF
    innodb_sort_buffer_size=2M
    innodb_online_alter_log_max_size=128M
    query_cache_strip_comments=0
    log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
    
    [mariadb-10.1]
    innodb_file_format = Barracuda
    innodb_file_per_table = 1
    
    ## wsrep specific
    # wsrep_on=OFF
    # wsrep_provider
    # wsrep_cluster_address
    # binlog_format=ROW
    # default_storage_engine=InnoDB
    # innodb_autoinc_lock_mode=2
    # innodb_doublewrite=1
    # query_cache_size=0
    
    # 2 variables needed to switch from XtraDB to InnoDB plugins
    #plugin-load=ha_innodb
    #ignore_builtin_innodb
    
    ## MariaDB 10 only save and restore buffer pool pages
    ## warm up InnoDB buffer pool on server restarts
    #innodb_buffer_pool_dump_at_shutdown=1
    #innodb_buffer_pool_load_at_startup=1
    innodb_buffer_pool_populate=0
    ## Disabled settings
    performance_schema=OFF
    innodb_stats_on_metadata=OFF
    innodb_sort_buffer_size=2M
    innodb_online_alter_log_max_size=128M
    query_cache_strip_comments=0
    log_slow_filter =admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
    
    # Defragmenting unused space on InnoDB tablespace
    innodb_defragment=1
    innodb_defragment_n_pages=7
    innodb_defragment_stats_accuracy=0
    innodb_defragment_fill_factor_n_recs=20
    innodb_defragment_fill_factor=0.9
    innodb_defragment_frequency=40
    
     
  11. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    How to backup database without mysql running ? Can i copy all database files ?
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    only backup entire /var/lib/mysql with mysql server down - you can not pick and choose files in that directory as you will corrupt data.

    backup ALL databases on server not just one including mysql system databases - entire /var/lib/mysql and subdirectories and move it to safe place first. But probably want to hire a professional to do this if you don't know how as you can permanently corrupt data!

    what version of mariadb do you have installed
    Code (Text):
    yum -q list installed MariaDB-Server
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    actually try this comment out in /etc/my.cnf
    Code (Text):
    #tmpdir=/home/mysqltmp

    then try restart mysql
    Code (Text):
    mysqlrestart
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    are you running/logged in as full root user or sudo user ?
     
  15. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    also what happens for commands
    Code (Text):
    cd /usr/local/src/centminmod
    ./centmin.sh
     
  16. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    Code:
    yum -q list installed MariaDB-Server
    Installed Packages
    MariaDB-server.x86_64                                                                                              10.1.18-1.el7.centos  
    Still cant start mysql service after add # to my.cnf

    "Failed at step NAMESPACE spawning /bin/sh: No space left on device" Not sure how my server no space left ?
     
  17. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    .centmin

    Before show menu i see alot of error message

    Code:
    ./centmin.sh
    
    inc/cpcheck.inc: line 494: cannot create temp file for here-document: No space left on device
    inc/cpcheck.inc: line 595: cannot create temp file for here-document: No space left on device
    inc/cpcheck.inc: line 639: cannot create temp file for here-document: No space left on device
    inc/jemalloc.inc: line 9: cannot create temp file for here-document: No space left on device
    Cannot create temporary file in /tmp/: No space left on device
    inc/jemalloc.inc: line 1: 13162 Aborted                 gcc "$JEMSTATS_CFILE" -o ex_stats_print -I${JEMALLOC_PATH}/include -L${JEMALLOC_PATH}/lib -Wl,-rpath,${JEMALLOC_PATH}/lib -ljemalloc
    inc/cpcheck.inc: line 1291: cannot create temp file for here-document: No space left on device
    
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    Try yum update first

    post output for these commands
    Code (Text):
    yum clean all
    yum list updates
    yum -y update
     
  19. eva2000

    eva2000 Administrator Staff Member

    30,963
    6,918
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,420
    Local Time:
    1:00 AM
    Nginx 1.13.x
    MariaDB 5.5
    ah further clues your /tmp is full

    output for
    Code (Text):
    df -hTi
    du -sh /tmp/
     
  20. YuchiRO

    YuchiRO Member

    93
    6
    8
    Jan 12, 2015
    Ratings:
    +8
    Local Time:
    10:00 PM
    5.5.4
    Code:
    df -hT
    Filesystem          Type      Size  Used Avail Use% Mounted on
    /dev/mapper/vg-root ext4      408G   40G  348G  11% /
    devtmpfs            devtmpfs   32G     0   32G   0% /dev
    tmpfs               tmpfs      32G     0   32G   0% /dev/shm
    tmpfs               tmpfs      32G   17M   32G   1% /run
    tmpfs               tmpfs      32G     0   32G   0% /sys/fs/cgroup
    /dev/sda1           ext2      243M  204M   27M  89% /boot
    /dev/mapper/vg-tmp  ext4      976M  261M  649M  29% /tmp
    tmpfs               tmpfs     6.3G     0  6.3G   0% /run/user/0
    

    Code:
    du -sh /tmp/
    258M    /tmp/