Upgrading from Rocky to Stein fails on MariaDB
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
Fix Released
|
Critical
|
Mark Goddard | ||
Rocky |
Fix Committed
|
Medium
|
Mark Goddard | ||
Stein |
Fix Released
|
Critical
|
Mark Goddard |
Bug Description
MariaDB was upgraded from 10.1 to 10.3 in the Stein release. It seems upgrades do not work currently, as seen in kayobe upgrade testing and anecdotally in IRC.
There are no useful error logs either in `docker logs mariadb` or /var/log/
Using the following command I was able to get some logs on startup:
docker run --rm -it --name mariadb_test -e KOLLA_CONFIG_
For some reason the -x causes logs to be dumped to stdout/err.
Here's a dump of the output: http://
The pertinent logs seem to be these:
nohup /usr/libexec/mysqld --basedir=/usr --datadir=
++ grep 'WSREP: Recovered position:' /tmp/wsrep_
+ local rp=
+ '[' -z '' ']'
++ grep WSREP /tmp/wsrep_
++ grep 'skipping position recovery'
+ local skipped=
+ '[' -z '' ']'
++ cat /tmp/wsrep_
+ log_error 'WSREP: Failed to recover position: '\''nohup: appending output to '\''/var/
2019-03-15 17:58:59 0 [Note] /usr/libexec/mysqld (mysqld 10.3.10-
2019-03-15 17:58:59 0 [Note] InnoDB: Using Linux native AIO
2019-03-15 17:58:59 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-03-15 17:58:59 0 [Note] InnoDB: Uses event mutexes
2019-03-15 17:58:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-03-15 17:58:59 0 [Note] InnoDB: Number of pools: 1
2019-03-15 17:58:59 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-03-15 17:58:59 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
2019-03-15 17:59:00 0 [Note] InnoDB: Completed initialization of buffer pool
2019-03-15 17:59:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-03-15 17:59:00 0 [ERROR] InnoDB: Upgrade after a crash is not supported. This redo log was created before MariaDB 10.2.2.
2019-03-15 17:59:00 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2019-03-15 17:59:00 0 [Note] InnoDB: Starting shutdown...
2019-03-15 17:59:00 0 [ERROR] Plugin '\''InnoDB'\'' init function returned error.
2019-03-15 17:59:00 0 [ERROR] Plugin '\''InnoDB'\'' registration as a STORAGE ENGINE failed.
2019-03-15 17:59:00 0 [Note] Plugin '\''FEEDBACK'\'' is disabled.
2019-03-15 17:59:00 0 [ERROR] Unknown/unsupported storage engine: innodb
2019-03-15 17:59:00 0 [ERROR] Aborting'\'''
Changed in kolla-ansible: | |
importance: | Undecided → High |
assignee: | nobody → Mark Goddard (mgoddard) |
Changed in kolla-ansible: | |
importance: | High → Critical |
Following https:/ /serverfault. com/questions/ 379714/ unknown- unsupported- storage- engine- innodb- mysql-ubuntu, I removed /var/lib/ mysql/ib_ logfile0 and /var/lib/ mysql/ib_ logfile1, which allowed the service to start. There are various bad looking logs, e.g.
[Warning] InnoDB: Table mysql/innodb_ index_stats has length mismatch in the column name table_name. Please run mysql_upgrade
This is clearly not the right course of action, but does show what is blocking the startup.
We also see these messages, suggesting we need to run mysql_upgrade:
[Warning] InnoDB: Table mysql/innodb_ table_stats has length mismatch in the column name table_name. Please run mysql_upgrade