sstpassword often set to wrong value in cluster and ha relations
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Percona Cluster Charm |
Fix Released
|
Wishlist
|
James Page | ||
percona-cluster (Juju Charms Collection) |
Invalid
|
Wishlist
|
Unassigned |
Bug Description
If the sst-password charm config option is not set then the charm generates a password. Unfortunatly in a clustered environment all units race to do this independantly of one another which can result in the wrong password being shared.
For example in this three node deployment, the password is consistent accross the units:
$ juju run --unit percona/0 "relation-get -r cluster:10 mysql-sstuser.
P2cqSpzb7Jk6fhw
P2cqSpzb7Jk6fhw
$ juju run --unit percona/1 "relation-get -r cluster:10 mysql-sstuser.
P2cqSpzb7Jk6fhw
P2cqSpzb7Jk6fhw
$ juju run --unit percona/2 "relation-get -r cluster:10 mysql-sstuser.
P2cqSpzb7Jk6fhw
P2cqSpzb7Jk6fhw
but it's wrong:
$ juju run --service percona "mysql -h localhost -u sstuser --password=
- MachineId: "10"
ReturnCode: 1
Stderr: |
ERROR 1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/0
- MachineId: "11"
ReturnCode: 1
Stderr: |
ERROR 1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/1
- MachineId: "12"
ReturnCode: 1
Stderr: |
ERROR 1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/2
I added some logging to the charm to see what happened to the password:
$ juju run --service percona "grep -E 'sstuser' /var/log/
- MachineId: "10"
Stdout: |
2015-05-12 15:18:40 DEBUG unit.percona/
2015-05-12 15:18:40 DEBUG unit.percona/
2015-05-12 15:18:40 INFO unit.percona/
2015-05-12 15:18:40 DEBUG unit.percona/
2015-05-12 15:20:00 DEBUG unit.percona/
2015-05-12 15:20:00 DEBUG unit.percona/
2015-05-12 15:20:00 DEBUG unit.percona/
2015-05-12 15:20:02 DEBUG unit.percona/
2015-05-12 15:20:02 DEBUG unit.percona/
UnitId: percona/0
- MachineId: "11"
Stdout: |
2015-05-12 15:18:39 DEBUG unit.percona/
2015-05-12 15:18:39 DEBUG unit.percona/
2015-05-12 15:18:39 INFO unit.percona/
2015-05-12 15:18:39 DEBUG unit.percona/
2015-05-12 15:19:59 DEBUG unit.percona/
2015-05-12 15:19:59 DEBUG unit.percona/
2015-05-12 15:19:59 DEBUG unit.percona/
2015-05-12 15:20:06 DEBUG unit.percona/
UnitId: percona/1
- MachineId: "12"
Stdout: |
2015-05-12 15:18:44 DEBUG unit.percona/
2015-05-12 15:18:44 DEBUG unit.percona/
2015-05-12 15:18:44 INFO unit.percona/
2015-05-12 15:18:44 DEBUG unit.percona/
2015-05-12 15:20:06 DEBUG unit.percona/
2015-05-12 15:20:06 DEBUG unit.percona/
2015-05-12 15:20:06 DEBUG unit.percona/
2015-05-12 15:20:08 DEBUG unit.percona/
UnitId: percona/2
Each unit generated it's own password and percona/0 actually had the correct password before the peer stored value was overwritten by percona/1:
juju run --service percona "mysql -h localhost -u sstuser --password=
- MachineId: "10"
Stdout: |
now()
2015-05-12 15:45:53
UnitId: percona/0
- MachineId: "11"
Stdout: |
now()
2015-05-12 15:45:53
UnitId: percona/1
- MachineId: "12"
Stdout: |
now()
2015-05-12 15:45:53
UnitId: percona/2
Unsuprisingly the password is incorrect in the ha relation as well:
$ juju run --unit mysql-hacluster/0 "relation-get -r ha:11 - percona/0 | grep -Eoh 'password.*'"
password=
$ juju run --unit mysql-hacluster/1 "relation-get -r ha:11 - percona/1 | grep -Eoh 'password.*'"
password=
$ juju run --unit mysql-hacluster/2 "relation-get -r ha:11 - percona/2 | grep -Eoh 'password.*'"
password=
Related branches
- Edward Hope-Morley: Needs Resubmitting
- Mario Splivalo (community): Approve
- OpenStack Charmers: Pending requested
-
Diff: 501 lines (+210/-40)11 files modifiedcharm-helpers-hooks.yaml (+1/-1)
charmhelpers/contrib/database/mysql.py (+21/-6)
charmhelpers/contrib/network/ip.py (+5/-3)
charmhelpers/core/hookenv.py (+32/-0)
charmhelpers/core/hugepage.py (+8/-1)
charmhelpers/core/strutils.py (+30/-0)
hooks/percona_hooks.py (+21/-3)
hooks/percona_utils.py (+12/-9)
tests/charmhelpers/contrib/amulet/utils.py (+47/-16)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+1/-1)
tests/charmhelpers/core/hookenv.py (+32/-0)
description: | updated |
Changed in percona-cluster (Juju Charms Collection): | |
importance: | Undecided → Critical |
assignee: | nobody → Liam Young (gnuoy) |
importance: | Critical → High |
status: | New → Confirmed |
Changed in percona-cluster (Juju Charms Collection): | |
milestone: | none → 15.07 |
tags: | added: openstack |
Changed in percona-cluster (Juju Charms Collection): | |
status: | Confirmed → In Progress |
Changed in percona-cluster (Juju Charms Collection): | |
milestone: | 15.07 → 15.10 |
tags: | added: oil |
Changed in percona-cluster (Juju Charms Collection): | |
status: | Confirmed → Triaged |
Changed in charm-percona-cluster: | |
assignee: | nobody → Liam Young (gnuoy) |
importance: | Undecided → Wishlist |
status: | New → Triaged |
Changed in percona-cluster (Juju Charms Collection): | |
status: | Triaged → Invalid |
Changed in charm-percona-cluster: | |
milestone: | none → 17.08 |
Changed in charm-percona-cluster: | |
status: | Fix Committed → Fix Released |
Another side effect is that adding a new unit updates the peer relation with the incorrest password:
$ juju run --unit percona/0 "cluster_ id=\$(relation- ids cluster); relation-get -r \$cluster_id mysql-sstuser. passwd percona/1; relation-get -r \$cluster_id mysql-sstuser.p hZkbwYY3rTqKGF3 YN hZkbwYY3rTqKGF3 YN id=\$(relation- ids cluster); relation-get -r \$cluster_id mysql-sstuser. passwd percona/1; relation-get -r \$cluster_id mysql-sstuser. passwd percona/2;"
asswd percona/2;"
swZBF8PcwfnjgNr
swZBF8PcwfnjgNr
$ juju add-unit percona
$ juju run --unit percona/0 "cluster_
yS2h4ywcrY6TPR3 xHgdk2m7RYSzpbd 2x xHgdk2m7RYSzpbd 2x 0,percona/ 1,percona/ 2 "mysql -h localhost -u sstuser --password= yS2h4ywcrY6TPR3 xHgdk2m7RYSzpbd 2x -e 'select now() from dual;'" - MachineId: "15" @'localhost' (using password: YES) @'localhost' (using password: YES)\n" @'localhost' (using password: YES)\n"
yS2h4ywcrY6TPR3
$ juju run --unit percona/
ReturnCode: 1
Stderr: |
ERROR 1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/0
- MachineId: "16"
ReturnCode: 1
Stderr: "Warning: Permanently added '10.5.32.89' (ECDSA) to the list of known hosts.\r\nERROR
1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/1
- MachineId: "17"
ReturnCode: 1
Stderr: "Warning: Permanently added '10.5.32.90' (ECDSA) to the list of known hosts.\r\nERROR
1045 (28000): Access denied for user 'sstuser'
Stdout: ""
UnitId: percona/2