Want to subscribe to topics you're interested in?
Become a Member

MariaDB Unable To Start during installation

Discussion in 'Install & Upgrades or Pre-Install Questions' started by BamaStangGuy, Sep 14, 2017.

  1. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    7:49 PM
    Just did a completely fresh install and received errors during the installation that mariadb could not start.

    Once installation is done MariaDB would still not start

    Code:
    [root@li40-144 ~]# systemctl -l status mariadb.service
    ● mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/mariadb.service.d
               └─migrated-from-my.cnf-settings.conf, openfileslimit.conf, protecthome.conf
       Active: failed (Result: exit-code) since Thu 2017-09-14 06:44:06 UTC; 1min 28s ago
      Process: 5626 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 5389 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      Process: 5387 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
     Main PID: 5626 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    Sep 14 06:44:05 hostname systemd[1]: Starting MariaDB database server...
    Sep 14 06:44:06 hostname mysqld[5626]: 2017-09-14  6:44:06 140142445205760 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 5626 ...
    Sep 14 06:44:06 hostname systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
    Sep 14 06:44:06 hostname systemd[1]: Failed to start MariaDB database server.
    Sep 14 06:44:06 hostname systemd[1]: Unit mariadb.service entered failed state.
    Sep 14 06:44:06 hostname systemd[1]: mariadb.service failed.
     
    Last edited by a moderator: Sep 14, 2017
  2. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    7:49 PM
    Code:
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: The InnoDB memory heap is disabled
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: Compressed tables use zlib 1.2.7
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: Using Linux native AIO
    2017-09-14  6:42:02 140519981660416 [Note] InnoDB: Using SSE crc32 instructions
    2017-09-14  6:42:02 140519981660416 [ERROR] mysqld: Can't create/write to file '/home/mysqltmp/ibZV7lGI' (Errcode: 30 "Read-only file system")
    2017-09-14 06:42:02 7fcd5b973900  InnoDB: Error: unable to create temporary file; errno: 30
    2017-09-14  6:42:02 140519981660416 [ERROR] Plugin 'InnoDB' init function returned error.
    2017-09-14  6:42:02 140519981660416 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    2017-09-14  6:42:02 140519981660416 [Note] Plugin 'FEEDBACK' is disabled.
    2017-09-14  6:42:02 140519981660416 [ERROR] Unknown/unsupported storage engine: InnoDB
    2017-09-14  6:42:02 140519981660416 [ERROR] Aborting
    
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: The InnoDB memory heap is disabled
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: Compressed tables use zlib 1.2.7
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: Using Linux native AIO
    2017-09-14  6:44:06 140142445205760 [Note] InnoDB: Using SSE crc32 instructions
    2017-09-14  6:44:06 140142445205760 [ERROR] mysqld: Can't create/write to file '/home/mysqltmp/ibKwtXsk' (Errcode: 30 "Read-only file system")
    2017-09-14 06:44:06 7f7574aa1900  InnoDB: Error: unable to create temporary file; errno: 30
    2017-09-14  6:44:06 140142445205760 [ERROR] Plugin 'InnoDB' init function returned error.
    2017-09-14  6:44:06 140142445205760 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    2017-09-14  6:44:06 140142445205760 [Note] Plugin 'FEEDBACK' is disabled.
    2017-09-14  6:44:06 140142445205760 [ERROR] Unknown/unsupported storage engine: InnoDB
    2017-09-14  6:44:06 140142445205760 [ERROR] Aborting
    More info:
     
  3. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
    What does your /etc/my.cnf look like ? You using MariaDB Galera Cluster ?

    Your systemctl status looks different from a working system with a few active processes i.e.
    Code (Text):
       Active: failed (Result: exit-code) since Thu 2017-09-14 06:44:06 UTC; 1min 28s ago
      Process: 5626 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 5389 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      Process: 5387 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    


    on a working system
    Code (Text):
    systemctl status mariadb | sed -e "s|$(hostname)|hostname|"
    * mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/mariadb.service.d
               `-migrated-from-my.cnf-settings.conf, openfileslimit.conf, protecthome.conf
       Active: active (running) since Tue 2017-09-12 11:10:23 UTC; 1 day 23h ago
     Main PID: 15000 (mysqld)
       Status: "Taking your SQL requests now..."
       CGroup: /system.slice/mariadb.service
               `-15000 /usr/sbin/mysqld
    
    Sep 12 11:10:23 hostname systemd[1]: Starting MariaDB database server...
    Sep 12 11:10:23 hostname mysqld[15000]: 2017-09-12 11:10:23 140616390711552 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 15000 ...
    Sep 12 11:10:23 hostname systemd[1]: Started MariaDB database server.
    

    what's output for
    Code (Text):
    ls -lah /home | grep mysqltmp    
    

    example
    Code (Text):
    ls -lah /home | grep mysqltmp    
    drwxrwxrwt  2 mysql mysql 4.0K Sep 12 11:10 mysqltmp
    

    and output for command
    Code (Text):
    journalctl -u mariadb --no-pager
    

    contents of /usr/lib/systemd/system/mariadb.service ?
    Code (Text):
    cat /usr/lib/systemd/system/mariadb.service
    

    contents of /etc/systemd/system/mariadb.service.d/protecthome.conf
    Code (Text):
    cat /etc/systemd/system/mariadb.service.d/protecthome.conf
    

    example on working system
    Code (Text):
    cat /etc/systemd/system/mariadb.service.d/protecthome.conf
    [Service]
    ProtectHome=false
    
     
  4. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    7:49 PM
    All I did was create a brand new Linode instance with Centos 7 and then copy the installer line from
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh

    It simply fails during installation (installer tries to connect to database and can not over and over again.) and then fails to start after installation is done.

    I can repeat this on different sized Linodes though the $40 one is the one I need it working on.

    I will start up a new one shortly and run some commands before I start the install.
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
    wonder if it's due to CentOS 7.4 release changes or something centmin mod isn't picking up with CentOS 7.4

    after yum update to CentOS 7.4
    Code (Text):
    cat /etc/redhat-release
    CentOS Linux release 7.4.1708 (Core)
    


    looks like it is
    Code (Text):
    service mariadb restart
    Redirecting to /bin/systemctl restart mariadb.service
    Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
    


    Code (Text):
    journalctl -u mariadb --no-pager | sed -e "s|$(hostname)|hostname|"
    -- Logs begin at Tue 2017-09-12 10:50:18 UTC, end at Thu 2017-09-14 11:35:52 UTC. --
    Sep 12 11:00:48 hostname systemd[1]: Starting MariaDB database server...
    Sep 12 11:00:49 hostname mysqld[19208]: 2017-09-12 11:00:49 140266854897920 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 19208 ...
    Sep 12 11:00:49 hostname systemd[1]: Started MariaDB database server.
    Sep 12 11:01:04 hostname systemd[1]: Stopping MariaDB database server...
    Sep 12 11:01:06 hostname systemd[1]: Starting MariaDB database server...
    Sep 12 11:01:07 hostname mysqld[20074]: 2017-09-12 11:01:07 139630060120320 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 20074 ...
    Sep 12 11:01:07 hostname systemd[1]: Started MariaDB database server.
    Sep 12 11:01:07 hostname systemd[1]: Stopping MariaDB database server...
    Sep 12 11:01:10 hostname systemd[1]: Stopped MariaDB database server.
    Sep 12 11:01:11 hostname systemd[1]: Starting MariaDB database server...
    Sep 12 11:01:12 hostname mysqld[20342]: 2017-09-12 11:01:12 139828499077376 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 20342 ...
    Sep 12 11:01:13 hostname systemd[1]: Started MariaDB database server.
    Sep 12 11:01:22 hostname systemd[1]: Stopping MariaDB database server...
    Sep 12 11:01:24 hostname systemd[1]: Stopped MariaDB database server.
    Sep 12 11:10:23 hostname systemd[1]: Starting MariaDB database server...
    Sep 12 11:10:23 hostname mysqld[15000]: 2017-09-12 11:10:23 140616390711552 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 15000 ...
    Sep 12 11:10:23 hostname systemd[1]: Started MariaDB database server.
    Sep 14 11:28:10 hostname systemd[1]: Stopping MariaDB database server...
    Sep 14 11:28:12 hostname systemd[1]: Starting MariaDB database server...
    Sep 14 11:28:13 hostname mysqld[31011]: 2017-09-14 11:28:13 140206596241664 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 31011 ...
    Sep 14 11:28:13 hostname systemd[1]: Started MariaDB database server.
    Sep 14 11:35:49 hostname systemd[1]: Stopping MariaDB database server...
    Sep 14 11:35:51 hostname systemd[1]: Starting MariaDB database server...
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB) starting as process 12235 ...
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: The InnoDB memory heap is disabled
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: Compressed tables use zlib 1.2.7
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: Using Linux native AIO
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] InnoDB: Using SSE crc32 instructions
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [ERROR] mysqld: Can't create/write to file '/home/mysqltmp/ibfl1r4q' (Errcode: 30 "Read-only file system")
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 7fdd661e4900  InnoDB: Error: unable to create temporary file; errno: 30
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [ERROR] Plugin 'InnoDB' init function returned error.
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [Note] Plugin 'FEEDBACK' is disabled.
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [ERROR] Unknown/unsupported storage engine: InnoDB
    Sep 14 11:35:52 hostname mysqld[12235]: 2017-09-14 11:35:52 140588877760768 [ERROR] Aborting
    Sep 14 11:35:52 hostname systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
    Sep 14 11:35:52 hostname systemd[1]: Failed to start MariaDB database server.
    Sep 14 11:35:52 hostname systemd[1]: Unit mariadb.service entered failed state.
    Sep 14 11:35:52 hostname systemd[1]: mariadb.service failed.
    


    For now quick fix is comment out tmpdir line with hash # in front in /etc/my.cnf and restart MariaDB MySQL server
    Code (Text):
    #tmpdir=/home/mysqltmp
    


    Updated fix MariaDB MySQL Not Restarting After CentOS 7.4 Update Fix
     
    Last edited: Sep 14, 2017
  6. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
  7. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    7:49 PM
    Is there anyway we can do this before installing?
     
  8. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
    Working on solution now could disable custom tmpdir but that defeats the purpose of it's inclusion as some servers have low /tmp disk free space and won't be able to handle mysql database maintenance routines properly i.e. repairing corrupt database tables etc if /tmp doesn't have the disk free space to accommodate up to 2-3x times the largest mysql database table size of your databases. It's problem you won't experience normally, until you actually need it most i.e. corrupt mysql tables needing repair. That's why i placed MySQL's tmpdir = /home/mysqltmp and setup /etc/systemd/system/mariadb.service.d/protecthome.conf to set ProtectHome=false to disable that systemd security restriction, but seems this is being ignored after CentOS 7.4 update
     
  9. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
  10. bassie

    bassie Active Member

    525
    109
    43
    Apr 29, 2016
    Ratings:
    +333
    Local Time:
    2:49 AM
    @BamaStangGuy Please remove li40-144.members.linod....... from your logs as it resolves to your server. (Bruteforce, SPAM etc prevention)
     
    • Informative Informative x 1
  11. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    7:49 PM
    It isn't a working Linode anymore (at least not mine as I destroyed it)
     
    • Like Like x 1
  12. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
  13. eva2000

    eva2000 Administrator Staff Member

    29,719
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    10:49 AM
    Nginx 1.13.x
    MariaDB 5.5
    Updated both Centmin Mod 123.08stable and 123.09beta01 builds with permanent workaround fix for MariaDB MySQL server not restarting after CentOS 7.4 updates. The issue is CentOS 7.4 only issue and MariaDB 10.1 I believe as systemd protections didn't get added until MariaDB 10.1. So CentOS 6 and MariaDB 10.0.x users (123.08stable) would probably not experience the issue. Details here MariaDB MySQL Not Restarting After CentOS 7.4 Update Fix

    To update your Centmin Mod builds follow instructions at centminmod.com/upgrade.html and respective version threads below:
    After updating Centmin Mod, run centmin.sh one more time for fix to auto populate to /etc/systemd/system/mariadb.service.d/protecthome.conf which should now contain
    Code (Text):
    [Service]
    ProtectSystem=false
    ProtectHome=false
    

    When you re-run centmin.sh on updated code, you will get a one time message like below:
    Code (Text):
    ./centmin.sh 
    
    Update MariaDB 10 setting for ProtectSystem=false addition
    
     
    Last edited: Sep 14, 2017
    • Like Like x 2