Welcome to Centmin Mod Community
Become a Member

MariaDB MariaDB don't start

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by pamamolf, Nov 7, 2016.

  1. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hello

    MariaDB refuse to start and i can see:

    Code:
    Starting mysql (via systemctl):  Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
                                                               [FAILED]
    [root@server ~]# systemctl status mariadb.service
    ● mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/mariadb.service.d
               └─migrated-from-my.cnf-settings.conf, protecthome.conf
       Active: failed (Result: exit-code) since Sun 2016-11-06 20:21:55 UTC; 6s ago
      Process: 29190 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 28948 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      Process: 28945 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
     Main PID: 29190 (code=exited, status=1/FAILURE)
       Status: "Applying batch of log records for InnoDB: Progress 96"
       CGroup: /system.slice/mariadb.service
    
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 78 79 80 log/log0recv.cc:1342(recv_parse_or_apply_log_rec_body)[0x7fc1bb820c6f]
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 81 82 83 log/log0recv.cc:1764(recv_recover_page_func(unsigned long, buf_block_t*))[0x7fc1bb8231bf]
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 84 85 86 buf/buf0buf.cc:4841(buf_page_io_complete(buf_page_t*))[0x7fc1bb92c81c]
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 87 88 fil/fil0fil.cc:6169(fil_aio_wait(unsigned long))[0x7fc1bb982c18]
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 89 90 91 srv/srv0start.cc:543(io_handler_thread)[0x7fc1bb8b4650]
    Nov 06 20:21:55 server.strategium.ru mysqld[29190]: 92 93 94 /lib64/libpthread.so.0(+0x7dc5)[0x7fc1bac66dc5]
    Nov 06 20:21:55 server.strategium.ru systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
    Nov 06 20:21:55 server.strategium.ru systemd[1]: Failed to start MariaDB database server.
    Nov 06 20:21:55 server.strategium.ru systemd[1]: Unit mariadb.service entered failed state.
    Nov 06 20:21:55 server.strategium.ru systemd[1]: mariadb.service failed.
    and this:


    It seems that is dead...
     
  2. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Do you recommend me to re import from backup the database or to reinstall (don't know how) the MariaDB ?
     
  3. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    12:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    FAQ item 19 for checking mariadb mysql logs fully
     
  4. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    It reports the same issues:

    Code:
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0cabfc700 InnoDB: Error: page 3394 log sequence number 163849267361
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: p       c 8d?$  w;
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: End of page dump
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0cb3fd700 InnoDB: uncompressed page, stored checksum in field1 408885540, calculated checksums for field1: crc32 3843253918, innodb 408885540, none 3735928559, stored checksum in field2 171467839, calculated checksums for field2: crc32 3843253918, innodb 171467839, none 3735928559, page LSN 38 619369335, low 4 bytes of LSN at page end 619369335, page number (if stored to page already) 3, space id (if created with >= MySQL-4.1.1 and stored already) 611
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: page type 17855 meaning INDEX
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Page may be an index page where index id is 1369
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0cb3fd700  InnoDB: Assertion failure in thread 140603459360512 in file page0page.cc line 159
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: We intentionally generate a memory trap.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: If you get repeated assertion failures or crashes, even
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: immediately after the mysqld startup, there may be
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: corruption in the InnoDB tablespace. Please refer to
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: about forcing recovery.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 161106 20:35:46 [ERROR] mysqld got signal 6 ;
    Nov 06 20:35:46 server.domain.com mysqld[33515]: This could be because you hit a bug. It is also possible that this binary
    Nov 06 20:35:46 server.domain.com mysqld[33515]: or one of the libraries it was linked against is corrupt, improperly built,
    Nov 06 20:35:46 server.domain.com mysqld[33515]: or misconfigured. This error can also be caused by malfunctioning hardware.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
    Nov 06 20:35:46 server.domain.com mysqld[33515]: We will try our best to scrape up some info that will hopefully help
    Nov 06 20:35:46 server.domain.com mysqld[33515]: diagnose the problem, but since we have already crashed,
    Nov 06 20:35:46 server.domain.com mysqld[33515]: something is definitely wrong and this may fail.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: Server version: 10.1.18-MariaDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: key_buffer_size=2147483648
    Nov 06 20:35:46 server.domain.com mysqld[33515]: read_buffer_size=2097152
    Nov 06 20:35:46 server.domain.com mysqld[33515]: max_used_connections=0
    Nov 06 20:35:46 server.domain.com mysqld[33515]: max_threads=1502
    Nov 06 20:35:46 server.domain.com mysqld[33515]: thread_count=0
    Nov 06 20:35:46 server.domain.com mysqld[33515]: It is possible that mysqld could use up to
    Nov 06 20:35:46 server.domain.com mysqld[33515]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8280173 K  bytes of memory
    Nov 06 20:35:46 server.domain.com mysqld[33515]: Hope that's ok; if not, decrease some variables in the equation.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: Thread pointer: 0x0x0
    Nov 06 20:35:46 server.domain.com mysqld[33515]: Attempting backtrace. You can use the following information to find out
    Nov 06 20:35:46 server.domain.com mysqld[33515]: where mysqld died. If you see no messages after this, something went
    Nov 06 20:35:46 server.domain.com mysqld[33515]: terribly wrong...
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 58 stack_bottom = 0x0 thread_stack 0x48400
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0c9bfa700 InnoDB: Error: page 7670 log sequence number 163849936585
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 59 60 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe4fd6f7a6e]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 61 2016-11-06 20:35:46 7fe0c9bfa700 InnoDB: Error: page 7671 log sequence number 163849936720
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0cabfc700 InnoDB: Error: page 9321 log sequence number 163849919899
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0c9bfa700 InnoDB: Error: page 7672 log sequence number 163849935623
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 2016-11-06 20:35:46 7fe0cabfc700 InnoDB: Error: page 9322 log sequence number 163849067961
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 62 63 /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7fe4fd21d8d5]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 64 65 /lib64/libpthread.so.0(+0xf100)[0x7fe4fc838100]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 66 67 68 /lib64/libc.so.6(gsignal+0x37)[0x7fe4fab915f7]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 69 70 71 mysys/stacktrace.c:268(my_print_stacktrace)[0x7fe4fab92ce8]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 72 73 page/page0page.cc:141(page_dir_find_owner_slot(unsigned char const*))[0x7fe4fd40fdb0]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 74 75 76 page/page0cur.cc:2074(page_cur_delete_rec(page_cur_t*, dict_index_t const*, unsigned long const*, mtr_t*))[0x7fe4fd40ec45]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 77 78 79 page/page0cur.cc:1998(page_cur_parse_delete_rec(unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*))[0x7fe4fd40f4bb]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 80 81 log/log0recv.cc:1342(recv_parse_or_apply_log_rec_body)[0x7fe4fd3eac6f]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 82 83 84 log/log0recv.cc:1764(recv_recover_page_func(unsigned long, buf_block_t*))[0x7fe4fd3ed1bf]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 85 86 87 buf/buf0buf.cc:4841(buf_page_io_complete(buf_page_t*))[0x7fe4fd4f681c]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 88 89 fil/fil0fil.cc:6169(fil_aio_wait(unsigned long))[0x7fe4fd54cc18]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 90 91 92 srv/srv0start.cc:543(io_handler_thread)[0x7fe4fd47e650]
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 93 94 95 /lib64/libpthread.so.0(+0x7dc5)[0x7fe4fc830dc5]
    Nov 06 20:35:47 server.domain.com systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
    Nov 06 20:35:47 server.domain.com systemd[1]: Failed to start MariaDB database server.
    Nov 06 20:35:47 server.domain.com systemd[1]: Unit mariadb.service entered failed state.
    Nov 06 20:35:47 server.domain.com systemd[1]: mariadb.service failed.
    Nov 06 20:35:49 server.domain.com systemd[1]: Stopped MariaDB database server.
    and:

    Code:
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 207274716872616d71666c61726172746568336c746b6a706b70370000008e1a000000008100e4756b343838366a31676e706d6c73707663386875616869676a36000000741a000000008901a2763664743064377267317130756f2016-11-06 20:35:46 7fe0c9bfa700 InnoDB: Error: page 472 log sequence number 163849512588
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 317274393868656930306b33000000891a000000002016-11-06 20:35:46 7fe0c9bfa700 InnoDB: Error: page 473 log sequence number 163849513179
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 9102867264346a6d7366713062716362716f6b346e3839747568323431000000be1a000000009904742016-11-06 20:35:46 7fe0cabfc700 InnoDB: Error: page 334 log sequence number 163849502332
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: 6135376b6d30716532356a32723736736433766d337475397237000000801a00000000a10286686665376263356f34306b697430617068687570657635357437000000a11a00000000a9ffda717168722016-11-06 20:35:46 7fe0cabfc700 InnoDB: Error: page 351 log sequence number 163849506145
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: is in the future! Current system log sequence number 163848714746.
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: Your database may be corrupt or you may have copied the InnoDB
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: tablespace but not the InnoDB log files. See
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
    Nov 06 20:35:46 server.domain.com mysqld[33515]: InnoDB: for more information.
    I think that there is no way to get this fixed as the server hard a Raid card failure and it seems i end up with corrupt data :(

    So i have to follow one path and just reimport database or re install MariaDB (don't know how) or both?
     
  5. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    12:57 PM
    Nginx 1.13.x
    MariaDB 5.5
  6. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Don't know anyone to hire with my low budget but it shouldn't be so hard i think as i have a working backup....

    Need some info on how to remove all database data from /var/lib/mysql and if then doesn't start to re install it.... ?
     
  7. Sunka

    Sunka Active Member

    932
    243
    43
    Oct 31, 2015
    Rijeka, Croatia
    Ratings:
    +394
    Local Time:
    3:57 AM
    Nginx 1.13.3
    MariaDB 10.1.24
  8. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I just delete database folder and re insert and it works :)

    Thanks !!!!
     
    • Optimistic Optimistic x 1
  9. eva2000

    eva2000 Administrator Staff Member

    31,022
    6,925
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,431
    Local Time:
    12:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 2
  10. pamamolf

    pamamolf Well-Known Member

    2,836
    254
    83
    May 31, 2014
    Ratings:
    +450
    Local Time:
    4:57 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    As everything works i will not touch anything.....but good to know :)

    Hope that i will never need that info again !