问题描述
数据库所在主机磁盘发生损坏。在修复后重新启动数据库服务,在启动时产生如下错误:
2019-12-16 05:16:43 7f21f7360700 InnoDB: Assertion failure in thread 139783858161408 in file btr0pcur.cc line 432 InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == buf_block_get_page_no(btr_pcur_get_block(cursor)) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 21:16:43 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=1 max_threads=151 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68109 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2879060 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f21f735fe18 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8e82e5] /usr/sbin/mysqld(handle_fatal_signal+0x494)[0x669714] /lib64/libpthread.so.0(+0xf7e0)[0x7f2221b2d7e0] /lib64/libc.so.6(gsignal+0x35)[0x7f22205cd495] /lib64/libc.so.6(abort+0x175)[0x7f22205cec75] /usr/sbin/mysqld[0xa94d83] /usr/sbin/mysqld[0xa37514] /usr/sbin/mysqld[0xa3a54b] /usr/sbin/mysqld[0x99882c] /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x65)[0x5a8615] /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1e)[0x81c8fe] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x179)[0x6c6989] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x3da)[0x6c59ea] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x250)[0x70afc0] /usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x19f)[0x70b85f] /usr/sbin/mysqld[0x6e50bd] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x27ee)[0x6e8b3e] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x370)[0x6eb540] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x112e)[0x6ec77e] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xcf)[0x6b815f] /usr/sbin/mysqld(handle_one_connection+0x47)[0x6b8287] /usr/sbin/mysqld(pfs_spawn_thread+0x12d)[0x98409d] /lib64/libpthread.so.0(+0x7aa1)[0x7f2221b25aa1] /lib64/libc.so.6(clone+0x6d)[0x7f2220683bcd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f21c8006c90): is an invalid pointer Connection ID (thread ID): 2 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2019-12-16 05:16:44 28933 [Note] Plugin 'FEDERATED' is disabled. 2019-12-16 05:16:44 28933 [Note] InnoDB: Using atomics to ref count buffer pool pages 2019-12-16 05:16:44 28933 [Note] InnoDB: The InnoDB memory heap is disabled 2019-12-16 05:16:44 28933 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-12-16 05:16:44 28933 [Note] InnoDB: Memory barrier is not used 2019-12-16 05:16:44 28933 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-12-16 05:16:44 28933 [Note] InnoDB: Using Linux native AIO 2019-12-16 05:16:44 28933 [Note] InnoDB: Not using CPU crc32 instructions 2019-12-16 05:16:44 28933 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-12-16 05:16:44 28933 [Note] InnoDB: Completed initialization of buffer pool 2019-12-16 05:16:44 28933 [Note] InnoDB: Highest supported file format is Barracuda. 2019-12-16 05:16:44 28933 [Note] InnoDB: The log sequence numbers 905825208157 and 905825208157 in ibdata files do not match the log sequence number 905825347990 in the ib_logfiles! 2019-12-16 05:16:44 28933 [Note] InnoDB: Database was not shutdown normally! 2019-12-16 05:16:44 28933 [Note] InnoDB: Starting crash recovery. 2019-12-16 05:16:44 28933 [Note] InnoDB: Reading tablespace information from the .ibd files... 2019-12-16 05:16:44 28933 [Note] InnoDB: Restoring possible half-written data pages 2019-12-16 05:16:44 28933 [Note] InnoDB: from the doublewrite buffer... 2019-12-16 05:16:44 28933 [Note] InnoDB: 128 rollback segment(s) are active. 2019-12-16 05:16:44 28933 [Note] InnoDB: Waiting for purge to start 2019-12-16 05:16:44 28933 [Note] InnoDB: 5.6.45 started; log sequence number 905825347990 2019-12-16 05:16:44 28933 [Note] Server hostname (bind-address): '*'; port: 3306 2019-12-16 05:16:44 28933 [Note] IPv6 is available. 2019-12-16 05:16:44 28933 [Note] - '::' resolves to '::'; 2019-12-16 05:16:44 28933 [Note] Server socket created on IP: '::'. 2019-12-16 05:16:44 28933 [Note] Event Scheduler: Loaded 0 events 2019-12-16 05:16:44 28933 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.6.45' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
根据错误提示,可能是 InnoDB 表发生损坏,并且数据库在不断重启。
问题原因
未知,以目前知识水平只能推测是 InnoDB 表损坏。
解决办法
既然没有找到原因,自然也谈不上解决办法。但是问题最后还是解决了,下面是误打误撞的过程:
由于数据库在不断重启,按照「14.21.2 Forcing InnoDB Recovery」指南,设置innodb_force_recovery=1后,数据库可以启动,不再自动重启。
对表进行CHECK TABLE ...操作,有三张表会导致数据库重启。由此推测这三张表存在问题,可能发生损坏。
尝试导出数据,但是不成功。即使设置innodb_force_recovery=6后,依旧无法导出数据。使用SELECT * FROM DB_NAME语句,会导致数据库重启。
接下来是神奇的地方:(1)在SELECT * ... LIMIT时,查询部分数据可以成功,但是达到某一条数据后失败。(1)在SELECT * ... ORDER BY DESC LIMIT时,查询部分数据可以成功,同样到达某条数据时失败。(3)但是这两个语句能够查询出完成数据集(可以说不能“越过”某条数据,在某条数据之后将会发生错误)。(4)如果SELECT WHERE ID = "<id>"语句,可以将所有数据查询出来。
接下来还有惊喜地方:(1)设置innodb_force_recovery=0后,数据库能够启动,而且不会一直重启。(1)我尝试使用OPTIMIZE TABLE tbl;命令,可以修复表。
附加说明
而实际上不是并不是磁盘损坏,而是文件系统损坏。
相关链接
Assertion failure in thread 139768130299648 in file btr0pcur.c line 430
参考文献
14.21.2 Forcing InnoDB Recovery
How can I fix database corruption on an InnoDB table?