Liberty to Liberty Upgrade: MariaDB upgrade stuck on post dpkg configure task

[Bjoern]

While upgrading from OSA 12.0.15 to 12.2.8 I noticed that mariadb got stuck during upgrade at the Galera Server start task :

infra01:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 33236 3832 ? Ss 13:02 0:00 /sbin/init
root 267 0.0 0.0 19480 180 ? S 13:02 0:00 upstart-udev-bridge --daemon
message+ 270 0.0 0.0 39116 320 ? Ss 13:02 0:00 dbus-daemon --system --fork
root 299 0.0 0.0 49272 3220 ? Ss 13:02 0:00 /lib/systemd/systemd-udevd --daemon
syslog 319 0.0 0.0 405568 2756 ? Ssl 13:02 0:00 rsyslogd
root 346 0.0 0.0 15412 1708 ? S 13:02 0:00 upstart-file-bridge --daemon
root 349 0.0 0.0 15264 212 ? S 13:02 0:00 upstart-socket-bridge --daemon
root 385 0.0 0.0 10224 4208 ? Ss 13:02 0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
root 491 0.0 0.0 12788 1936 lxc/tty4 Ss+ 13:02 0:00 /sbin/getty -8 38400 tty4
root 494 0.0 0.0 12788 1944 lxc/tty2 Ss+ 13:02 0:00 /sbin/getty -8 38400 tty2
root 495 0.0 0.0 12788 1960 lxc/tty3 Ss+ 13:02 0:00 /sbin/getty -8 38400 tty3
root 509 0.0 0.0 23656 2324 ? Ss 13:02 0:00 cron
root 515 0.0 0.0 61380 5424 ? Ss 13:02 0:00 /usr/sbin/sshd -D
root 557 0.0 0.0 18236 3140 ? S 13:02 0:00 /bin/bash /usr/bin/mysqld_safe
mysql 980 48.9 0.8 9022584 3463372 ? Sl 13:03 14:30 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep_provider=/usr/lib/galera/libgalera_smm.so --log-error=/var/log/mysql_logs/galera_server_error.log --open-files-l
root 1133 0.0 0.0 12788 1908 lxc/console Ss+ 13:03 0:00 /sbin/getty -8 38400 console
root 1137 0.0 0.0 12788 1880 lxc/tty1 Ss+ 13:03 0:00 /sbin/getty -8 38400 tty1
root 2461 0.0 0.0 93340 6400 ? Ss 13:26 0:00 sshd: root@notty
root 3796 0.0 0.0 4448 688 ? Ss 13:27 0:00 /bin/sh -c LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python
root 3797 0.0 0.0 99696 53536 ? S 13:27 0:00 /usr/bin/python
root 3801 3.2 0.0 73620 38596 ? S 13:27 0:10 /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install mariadb-galera-server-10.0 galera-3
root 3808 0.0 0.0 18304 3432 ? S 13:27 0:00 /bin/bash
root 4324 0.0 0.0 13476 3472 ? S 13:27 0:00 /usr/bin/dpkg --force-confdef --force-confold --status-fd 43 --configure galera-3:amd64 mariadb-galera-server-10.0:amd64
root 4325 0.0 0.0 58320 13536 ? S 13:27 0:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/mariadb-galera-server-10.0.postinst configure 10.0.27+maria-1~trusty
root 4331 0.0 0.0 9680 2792 ? S 13:27 0:00 /bin/bash -e /var/lib/dpkg/info/mariadb-galera-server-10.0.postinst configure 10.0.27+maria-1~trusty
mysql 4474 0.4 0.1 5024208 580608 ? Sl 13:31 0:00 /usr/sbin/mysqld --bootstrap --user=mysql --disable-log-bin --skip-grant-tables --default-storage-engine=myisam
root 4495 0.0 0.0 15568 2108 ? R+ 13:32 0:00 ps uax

To overcome this issue I had to manually stop the database, the playbooks continued at this point.

Additionally I noticed that mysql_install_db is running as part of the dpkg post install on an existing DB, as here shown for infra02

root 3508 0.0 0.0 4448 768 ? Ss 13:34 0:00 /bin/sh -c LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python
root 3509 0.6 0.0 99700 53392 ? S 13:34 0:00 /usr/bin/python
root 3513 3.4 0.0 73620 38428 ? S 13:34 0:01 /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install mariadb-galera-server-10.0 galera-3
root 4026 0.0 0.0 13476 3452 ? S 13:34 0:00 /usr/bin/dpkg --force-confdef --force-confold --status-fd 43 --configure galera-3:amd64 mariadb-galera-server-10.0:amd64
root 4027 0.3 0.0 58320 13428 ? S 13:34 0:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/mariadb-galera-server-10.0.postinst configure 10.0.27+maria-1~trusty
root 4033 0.0 0.0 9672 2752 ? S 13:34 0:00 /bin/bash -e /var/lib/dpkg/info/mariadb-galera-server-10.0.postinst configure 10.0.27+maria-1~trusty
root 4072 0.0 0.0 9640 2704 ? S 13:34 0:00 /bin/bash /usr/bin/mysql_install_db --rpm --user=mysql --disable-log-bin
root 4073 0.0 0.0 4344 1420 ? S 13:34 0:00 logger -p daemon.err -t mysqld_safe -i
root 4157 0.0 0.0 9640 1800 ? S 13:35 0:00 /bin/bash /usr/bin/mysql_install_db --rpm --user=mysql --disable-log-bin
mysql 4159 14.5 0.1 5237520 600636 ? Sl 13:35 0:00 /usr/sbin/mysqld --lc-messages=en_US --bootstrap --basedir=/usr --datadir=/var/lib/mysql --log-warnings=0 --loose-skip-ndbcluster --disable-log-bin --user=mysql --max_allowed_packet=8M --net_buffer_length=16K
root 4186 0.0 0.0 15568 2120 pts/0 R+ 13:35 0:00 ps aux

which concerns me and that maybe the reason why the upgrade got stuck in the first place.
Since Liberty is now EOL, I'm not sure how we handle this issue which may be reappearing on future versions. That's why I opened this issue

[Bjoern (bjoern-t)]

As you may see, the galera version updated from 10.0.27 to 10.0.28

[Jean-Philippe Evrard (jean-philippe-evrard)]

See bug triage conversation here:
http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-11-29.log.html#t2016-11-29T16:05:07

TL:DR; Bjoern's gonna contact ubuntu support.

[Jean-Philippe Evrard (jean-philippe-evrard)]

Do you have an update?

[Bjoern (bjoern-t)]

No update yet, I'll assign it to myself

[Jean-Philippe Evrard (jean-philippe-evrard)]

Bjoern, Liberty is now end of life, so I'll mark this as won't fix.

I'll keep you assigned to it, in case you need to track it.

Thanks for the work!