Charm upgrade from rev. 248-253 to rev. 259 with min-cluster-size: 3 fails with Bootstrap PXC failed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Percona Cluster Charm |
Fix Released
|
Critical
|
David Ames |
Bug Description
# Issue
Upgrading an existing percona-cluster charm with min-cluster-size: 3 from rev. 250 to rev. 259 results in one of the units failing with the error message "Bootstrap PXC failed".
I did a bit more digging and I managed to reproduce it at least on rev. 248-253. Rev. 254 and up seemed to work fine.
I saw there was some work done in that area of code last year in lp#1668833, but not sure if related.
# Reproduction
## bundle.yaml
applications:
mysql:
charm: cs:percona-
num_units: 3
options:
min-
## Steps
* `juju deploy bundle.yaml`
* Wait for cluster to settle
* `juju upgrade-charm mysql`
# Logs
Excerpt from /var/log/
2018-03-13 14:48:46 DEBUG upgrade-charm active
2018-03-13 14:48:46 INFO juju-log Unit is ready
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'install'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:48:50 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:48:50 INFO juju-log MySQL already installed, skipping
2018-03-13 14:48:54 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-
2018-03-13 14:48:55 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:48:58 DEBUG juju-log Writing file /etc/mysql/
2018-03-13 14:49:05 DEBUG config-changed Unknown operation bootstrap-pxc.
2018-03-13 14:49:15 DEBUG config-changed Job for run-r7f3ba74eb1
2018-03-13 14:49:15 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '--service-
2018-03-13 14:49:15 DEBUG config-changed Traceback (most recent call last):
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed main()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed hooks.execute(
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed self._hooks[
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed return f(*args, **kwargs)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed bootstrap=not leader_
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed bootstrap_pxc()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/
2018-03-13 14:49:15 DEBUG config-changed raise Exception(
2018-03-13 14:49:15 DEBUG config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '--service-
2018-03-13 14:49:15 ERROR juju.worker.
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'install'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:49:22 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:49:22 INFO juju-log MySQL already installed, skipping
2018-03-13 14:49:24 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-
2018-03-13 14:49:25 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:49:26 DEBUG config-changed Unknown operation bootstrap-pxc.
# Versions
juju 2.3.4
percona-cluster rev. 250, 259
summary: |
- Upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails with - Bootstrap PXC failed + Charm upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails + with Bootstrap PXC failed |
summary: |
- Charm upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails - with Bootstrap PXC failed + Charm upgrade from rev. 248-253 to rev. 259 with min-cluster-size: 3 + fails with Bootstrap PXC failed |
description: | updated |
description: | updated |
Changed in charm-percona-cluster: | |
status: | New → Confirmed |
importance: | Undecided → Critical |
assignee: | nobody → Liam Young (gnuoy) |
milestone: | none → 18.02 |
Changed in charm-percona-cluster: | |
milestone: | 18.02 → 18.05 |
Changed in charm-percona-cluster: | |
assignee: | Liam Young (gnuoy) → David Ames (thedac) |
Changed in charm-percona-cluster: | |
status: | Fix Committed → Fix Released |
milestone: | 18.05 → 18.02 |
We think this is is_leader_ bootstrapped getting in the way because root-password is not in leader settings.
upgrade-charm should stabilize expected leader settings values.