juju agent upgrade causes mysqld to stop (part of same systemd cgroup)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Percona Cluster Charm |
Fix Released
|
Critical
|
James Page | ||
percona-cluster (Juju Charms Collection) |
Invalid
|
Critical
|
Unassigned | ||
percona-xtradb-cluster-5.6 (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Juju agent upgrade causes whole openstack down which is critical.
How to reproduce:
$ juju bootstrap --config agent-version=2.0.2
$ juju deploy ./bundle.yaml
$ juju run --unit mysql/0 'pgrep -af mysqld'
14743 /bin/sh /usr/bin/
15242 /usr/sbin/mysqld --basedir=/usr --datadir=
(controller model)
$ juju upgrade-juju -m controller --agent-version 2.0.3
(openstack model)
$ juju upgrade-juju --agent-version 2.0.3
$ juju run --unit mysql/0 'pgrep -af mysqld'
-> empty (no mysqld is running)
$ juju status mysql
Model Controller Cloud/Region Version
default localhost-localhost localhost/localhost 2.0.3
App Version Status Scale Charm Store Rev OS Notes
mysql 5.6.21-25.8 error 1 percona-cluster jujucharms 241 ubuntu
Unit Workload Agent Machine Public address Ports Message
mysql/0* error idle 2 10.0.8.104 hook failed: "config-changed"
Machine State DNS Inst id Series AZ2 started 10.0.8.104 juju-50b253-2 xenial
...
Changed in charm-percona-cluster: | |
importance: | Undecided → Critical |
status: | New → Confirmed |
Changed in percona-cluster (Juju Charms Collection): | |
status: | Confirmed → Invalid |
Changed in charm-percona-cluster: | |
status: | Confirmed → Triaged |
milestone: | none → 17.05 |
Changed in charm-percona-cluster: | |
status: | Fix Committed → Fix Released |
Somehow mysqld "Normal shutdown" kicked twice around agent upgrading.
00:37 is localtime while 15:37 is UTC.
[/var/log/ mysql/error. log]
...
2017-02-12 15:37:47 15242 [Note] /usr/sbin/mysqld: Normal shutdown
...
2017-02-12 15:37:52 15242 [Note] /usr/sbin/mysqld: Shutdown complete
...
2017-02-12 15:38:52 20201 [Note] /usr/sbin/mysqld: Normal shutdown
...
2017-02-12 15:38:56 20201 [Note] /usr/sbin/mysqld: Shutdown complete
[journalctl -u mysql] d/mysql[ 19644]: MySQL PID not found, pid_file detected/guessed: /var/run/ mysqld/ mysqld. pid d/mysql[ 19648]: MySQL PID not found, pid_file detected/guessed: /var/run/ mysqld/ mysqld. pid
Feb 12 15:37:47 juju-50b253-2 systemd[1]: Stopping LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Feb 12 15:37:47 juju-50b253-2 mysql[19583]: * Stopping MySQL (Percona XtraDB Cluster) mysqld
Feb 12 15:37:52 juju-50b253-2 /etc/init.
Feb 12 15:37:52 juju-50b253-2 /etc/init.
Feb 12 15:37:52 juju-50b253-2 mysql[19583]: ...done.
Feb 12 15:37:52 juju-50b253-2 systemd[1]: Stopped LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
Feb 12 15:38:03 juju-50b253-2 systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Feb 12 15:38:04 juju-50b253-2 mysql[20252]: * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Feb 12 15:38:04 juju-50b253-2 mysql[20252]: ...done.
Feb 12 15:38:04 juju-50b253-2 systemd[1]: Started LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
[unit-mysql-0.log] upgrader desired tool version: 2.0.2 0.juju- log Leader unit - bootstrap required=True 0.juju- log Writing file /etc/mysql/ percona- xtradb- cluster. conf.d/ mysqld. cnf root:root 444 0.config- changed Unknown operation bootstrap-pxc. 0.config- changed * Bootstrapping Percona XtraDB Cluster database server mysqld leadership mysql/0 will renew mysql leadership at 2017-02-12 15:38:30.142709216 +0000 UTC 0.config- changed ...done. 0.juju- log Bootstrap PXC Succeeded
unit-mysql-0: 00:37:31 INFO juju.worker.
...
unit-mysql-0: 00:37:44 DEBUG unit.mysql/
unit-mysql-0: 00:37:47 INFO unit.mysql/
unit-mysql-0: 00:37:52 INFO unit.mysql/
unit-mysql-0: 00:37:52 INFO unit.mysql/
unit-mysql-0: 00:38:00 INFO juju.worker.
unit-mysql-0: 00:38:03 INFO unit.mysql/
unit-mysql-0: 00:38:04 DEBUG unit.mysql/
...
unit-mysql-0: 00:38:51 INFO juju.worker. upgrader upgrade requested from 2.0.2 to 2.0.3