| 2018-03-13 14:54:43 |
Sandor Zeestraten |
bug |
|
|
added bug |
| 2018-03-13 15:12:18 |
Sandor Zeestraten |
summary |
Upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails with Bootstrap PXC failed |
Charm upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails with Bootstrap PXC failed |
|
| 2018-03-14 09:57:04 |
Sandor Zeestraten |
summary |
Charm upgrade from rev. 250 to rev. 259 with min-cluster-size: 3 fails with Bootstrap PXC failed |
Charm upgrade from rev. 248-253 to rev. 259 with min-cluster-size: 3 fails with Bootstrap PXC failed |
|
| 2018-03-14 09:58:21 |
Sandor Zeestraten |
description |
# Issue
Upgrading an existing percona-cluster cluster with min-cluster-size: 3 from rev. 250 to rev. 259 results in one of the units failing with the error message "Bootstrap PXC failed".
I saw there was some work done in that area of code last year in lp#1668833, but not sure if related.
# Reproduction
## bundle.yaml
applications:
mysql:
charm: cs:percona-cluster-250
num_units: 3
options:
min-cluster-size: 3
## Steps
* `juju deploy bundle.yaml`
* Wait for cluster to settle
* `juju upgrade-charm mysql`
# Logs
Excerpt from /var/log/juju/unit-percona-cluster-0.log from an reproduction on LXD:
2018-03-13 14:48:46 DEBUG upgrade-charm active
2018-03-13 14:48:46 INFO juju-log Unit is ready
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'install'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:48:50 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:48:50 INFO juju-log MySQL already installed, skipping
2018-03-13 14:48:54 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:48:55 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:48:58 DEBUG juju-log Writing file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf root:root 444
2018-03-13 14:49:05 DEBUG config-changed Unknown operation bootstrap-pxc.
2018-03-13 14:49:15 DEBUG config-changed Job for run-r7f3ba74eb1d44473b3ea94030535d578.service failed because the control process exited with error code. See "systemctl status run-r7f3ba74eb1d44473b3ea94030535d578.service" and "journalctl -xe" for details.
2018-03-13 14:49:15 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 DEBUG config-changed Traceback (most recent call last):
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 837, in <module>
2018-03-13 14:49:15 DEBUG config-changed main()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 827, in main
2018-03-13 14:49:15 DEBUG config-changed hooks.execute(sys.argv)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-13 14:49:15 DEBUG config-changed self._hooks[hook_name]()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-13 14:49:15 DEBUG config-changed return f(*args, **kwargs)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 374, in config_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap=not leader_bootstrapped)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 237, in render_config_restart_on_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap_pxc()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/percona_utils.py", line 475, in bootstrap_pxc
2018-03-13 14:49:15 DEBUG config-changed raise Exception(error_msg)
2018-03-13 14:49:15 DEBUG config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 ERROR juju.worker.uniter.operation runhook.go:113 hook "config-changed" failed: exit status 1
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'install'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:49:22 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:49:22 INFO juju-log MySQL already installed, skipping
2018-03-13 14:49:24 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:49:25 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:49:26 DEBUG config-changed Unknown operation bootstrap-pxc.
# Versions
juju 2.3.4
percona-cluster rev. 250, 259 |
# Issue
Upgrading an existing percona-cluster cluster with min-cluster-size: 3 from rev. 250 to rev. 259 results in one of the units failing with the error message "Bootstrap PXC failed".
I did a bit more digging and I managed to reproduce it at least on rev. 248-253. Rev. 254 and up seemed to work fine.
I saw there was some work done in that area of code last year in lp#1668833, but not sure if related.
# Reproduction
## bundle.yaml
applications:
mysql:
charm: cs:percona-cluster-250
num_units: 3
options:
min-cluster-size: 3
## Steps
* `juju deploy bundle.yaml`
* Wait for cluster to settle
* `juju upgrade-charm mysql`
# Logs
Excerpt from /var/log/juju/unit-percona-cluster-0.log from an reproduction on LXD:
2018-03-13 14:48:46 DEBUG upgrade-charm active
2018-03-13 14:48:46 INFO juju-log Unit is ready
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'install'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:48:50 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:48:50 INFO juju-log MySQL already installed, skipping
2018-03-13 14:48:54 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:48:55 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:48:58 DEBUG juju-log Writing file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf root:root 444
2018-03-13 14:49:05 DEBUG config-changed Unknown operation bootstrap-pxc.
2018-03-13 14:49:15 DEBUG config-changed Job for run-r7f3ba74eb1d44473b3ea94030535d578.service failed because the control process exited with error code. See "systemctl status run-r7f3ba74eb1d44473b3ea94030535d578.service" and "journalctl -xe" for details.
2018-03-13 14:49:15 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 DEBUG config-changed Traceback (most recent call last):
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 837, in <module>
2018-03-13 14:49:15 DEBUG config-changed main()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 827, in main
2018-03-13 14:49:15 DEBUG config-changed hooks.execute(sys.argv)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-13 14:49:15 DEBUG config-changed self._hooks[hook_name]()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-13 14:49:15 DEBUG config-changed return f(*args, **kwargs)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 374, in config_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap=not leader_bootstrapped)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 237, in render_config_restart_on_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap_pxc()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/percona_utils.py", line 475, in bootstrap_pxc
2018-03-13 14:49:15 DEBUG config-changed raise Exception(error_msg)
2018-03-13 14:49:15 DEBUG config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 ERROR juju.worker.uniter.operation runhook.go:113 hook "config-changed" failed: exit status 1
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'install'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:49:22 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:49:22 INFO juju-log MySQL already installed, skipping
2018-03-13 14:49:24 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:49:25 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:49:26 DEBUG config-changed Unknown operation bootstrap-pxc.
# Versions
juju 2.3.4
percona-cluster rev. 250, 259 |
|
| 2018-03-14 11:37:37 |
Sandor Zeestraten |
description |
# Issue
Upgrading an existing percona-cluster cluster with min-cluster-size: 3 from rev. 250 to rev. 259 results in one of the units failing with the error message "Bootstrap PXC failed".
I did a bit more digging and I managed to reproduce it at least on rev. 248-253. Rev. 254 and up seemed to work fine.
I saw there was some work done in that area of code last year in lp#1668833, but not sure if related.
# Reproduction
## bundle.yaml
applications:
mysql:
charm: cs:percona-cluster-250
num_units: 3
options:
min-cluster-size: 3
## Steps
* `juju deploy bundle.yaml`
* Wait for cluster to settle
* `juju upgrade-charm mysql`
# Logs
Excerpt from /var/log/juju/unit-percona-cluster-0.log from an reproduction on LXD:
2018-03-13 14:48:46 DEBUG upgrade-charm active
2018-03-13 14:48:46 INFO juju-log Unit is ready
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'install'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:48:50 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:48:50 INFO juju-log MySQL already installed, skipping
2018-03-13 14:48:54 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:48:55 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:48:58 DEBUG juju-log Writing file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf root:root 444
2018-03-13 14:49:05 DEBUG config-changed Unknown operation bootstrap-pxc.
2018-03-13 14:49:15 DEBUG config-changed Job for run-r7f3ba74eb1d44473b3ea94030535d578.service failed because the control process exited with error code. See "systemctl status run-r7f3ba74eb1d44473b3ea94030535d578.service" and "journalctl -xe" for details.
2018-03-13 14:49:15 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 DEBUG config-changed Traceback (most recent call last):
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 837, in <module>
2018-03-13 14:49:15 DEBUG config-changed main()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 827, in main
2018-03-13 14:49:15 DEBUG config-changed hooks.execute(sys.argv)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-13 14:49:15 DEBUG config-changed self._hooks[hook_name]()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-13 14:49:15 DEBUG config-changed return f(*args, **kwargs)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 374, in config_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap=not leader_bootstrapped)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 237, in render_config_restart_on_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap_pxc()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/percona_utils.py", line 475, in bootstrap_pxc
2018-03-13 14:49:15 DEBUG config-changed raise Exception(error_msg)
2018-03-13 14:49:15 DEBUG config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 ERROR juju.worker.uniter.operation runhook.go:113 hook "config-changed" failed: exit status 1
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'install'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:49:22 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:49:22 INFO juju-log MySQL already installed, skipping
2018-03-13 14:49:24 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:49:25 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:49:26 DEBUG config-changed Unknown operation bootstrap-pxc.
# Versions
juju 2.3.4
percona-cluster rev. 250, 259 |
# Issue
Upgrading an existing percona-cluster charm with min-cluster-size: 3 from rev. 250 to rev. 259 results in one of the units failing with the error message "Bootstrap PXC failed".
I did a bit more digging and I managed to reproduce it at least on rev. 248-253. Rev. 254 and up seemed to work fine.
I saw there was some work done in that area of code last year in lp#1668833, but not sure if related.
# Reproduction
## bundle.yaml
applications:
mysql:
charm: cs:percona-cluster-250
num_units: 3
options:
min-cluster-size: 3
## Steps
* `juju deploy bundle.yaml`
* Wait for cluster to settle
* `juju upgrade-charm mysql`
# Logs
Excerpt from /var/log/juju/unit-percona-cluster-0.log from an reproduction on LXD:
2018-03-13 14:48:46 DEBUG upgrade-charm active
2018-03-13 14:48:46 INFO juju-log Unit is ready
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'install'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:48:49 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:48:50 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:48:50 INFO juju-log MySQL already installed, skipping
2018-03-13 14:48:54 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:48:55 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:48:58 DEBUG juju-log Writing file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf root:root 444
2018-03-13 14:49:05 DEBUG config-changed Unknown operation bootstrap-pxc.
2018-03-13 14:49:15 DEBUG config-changed Job for run-r7f3ba74eb1d44473b3ea94030535d578.service failed because the control process exited with error code. See "systemctl status run-r7f3ba74eb1d44473b3ea94030535d578.service" and "journalctl -xe" for details.
2018-03-13 14:49:15 ERROR juju-log Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 DEBUG config-changed Traceback (most recent call last):
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 837, in <module>
2018-03-13 14:49:15 DEBUG config-changed main()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 827, in main
2018-03-13 14:49:15 DEBUG config-changed hooks.execute(sys.argv)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-13 14:49:15 DEBUG config-changed self._hooks[hook_name]()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-13 14:49:15 DEBUG config-changed return f(*args, **kwargs)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 374, in config_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap=not leader_bootstrapped)
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/config-changed", line 237, in render_config_restart_on_changed
2018-03-13 14:49:15 DEBUG config-changed bootstrap_pxc()
2018-03-13 14:49:15 DEBUG config-changed File "/var/lib/juju/agents/unit-percona-cluster-0/charm/hooks/percona_utils.py", line 475, in bootstrap_pxc
2018-03-13 14:49:15 DEBUG config-changed raise Exception(error_msg)
2018-03-13 14:49:15 DEBUG config-changed Exception: Bootstrap PXC failed: Command '['systemd-run', '--service-type=forking', 'service', 'mysql', 'bootstrap-pxc']' returned non-zero exit status 1
2018-03-13 14:49:15 ERROR juju.worker.uniter.operation runhook.go:113 hook "config-changed" failed: exit status 1
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'install'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'upgrade'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'config_changed'
2018-03-13 14:49:21 DEBUG juju-log Hardening function 'update_status'
2018-03-13 14:49:22 DEBUG juju-log No hardening applied to 'config_changed'
2018-03-13 14:49:22 INFO juju-log MySQL already installed, skipping
2018-03-13 14:49:24 DEBUG juju-log Leader is NOT bootstrapped root-password: 34bc1742-26cd-11e8-b877-aa4304b6c454
2018-03-13 14:49:25 DEBUG juju-log Leader unit - bootstrap required=True
2018-03-13 14:49:26 DEBUG config-changed Unknown operation bootstrap-pxc.
# Versions
juju 2.3.4
percona-cluster rev. 250, 259 |
|
| 2018-03-14 13:39:45 |
Liam Young |
charm-percona-cluster: status |
New |
Confirmed |
|
| 2018-03-14 13:39:47 |
Liam Young |
charm-percona-cluster: importance |
Undecided |
Critical |
|
| 2018-03-14 13:39:49 |
Liam Young |
charm-percona-cluster: assignee |
|
Liam Young (gnuoy) |
|
| 2018-03-14 13:39:51 |
Liam Young |
charm-percona-cluster: milestone |
|
18.02 |
|
| 2018-03-14 16:38:00 |
Ryan Beisner |
charm-percona-cluster: milestone |
18.02 |
18.05 |
|
| 2018-03-14 21:46:42 |
David Ames |
charm-percona-cluster: assignee |
Liam Young (gnuoy) |
David Ames (thedac) |
|
| 2018-03-14 21:47:01 |
OpenStack Infra |
charm-percona-cluster: status |
Confirmed |
In Progress |
|
| 2018-03-15 09:15:19 |
OpenStack Infra |
charm-percona-cluster: status |
In Progress |
Fix Committed |
|
| 2018-03-16 13:52:00 |
Liam Young |
charm-percona-cluster: status |
Fix Committed |
Fix Released |
|
| 2018-03-16 13:52:13 |
Liam Young |
charm-percona-cluster: milestone |
18.05 |
18.02 |
|
