passwords not migrated from peer relation to leader settings during upgrade from ~15.04 to 17.11

[Paul Collins]

During an upgrade from percona-cluster ~15.04 (r52, revision-id: <email address hidden>) to 17.11,the upgrade-charm hook failed as follows:

2018-02-08 03:01:44 INFO upgrade-charm _mysql_exceptions.OperationalError: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")

I realized that all of the passwords for the various related services were in the peer relation:

stagingstack-is@wekufe:~$ alias jru='juju run --unit'
stagingstack-is@wekufe:~$ jru mysql/0 'relation-get -r $(relation-ids cluster) - mysql/0'
bootstrap-uuid: c5a4bbfa-0c79-11e8-9b00-bfea4352af21
mysql-cinder.passwd: 89S9qrK8nxGPrTYbsSW2J5P5zBZ5M2RS
mysql-glance.passwd: Fsw6NXfdnXjdMypGjSYhFRJPg9mdRBYp
mysql-keystone.passwd: t4H6YNn49hC45bnJ4NYrZNsmK3gP4fBb
mysql-neutron.passwd: LNFM4Xq48zBb8Mm5XShHGktHpTc2CVGm
mysql-nova.passwd: 74Vf7bVd6JLfhrrc97kYHp476ZN2tZHk
mysql-sstuser.passwd: notarealsstpassword
mysql.passwd: notarealrootpassword
private-address: 10.25.126.203
stagingstack-is@wekufe:~$ _

whereas the charm code was now looking at the leader settings, which had fewer and different passwords:

stagingstack-is@wekufe:~$ jru mysql/0 leader-get
bootstrap-uuid: c5a4bbfa-0c79-11e8-9b00-bfea4352af21
mysql.passwd: LtWcYLGWNT6tXpJyVb5WwHjzPMkjGZZt
sst-password: notarealsstpassword
stagingstack-is@wekufe:~$ _

Running the following before upgrading the charm avoids the problem:

jru mysql/0 'leader-set $(relation-get -r $(relation-ids cluster) - mysql/0 | grep passwd | sed -e "s/: /=/")'

[James Page (james-page)]

Commit bda5d1508190286d92951f1b9643d1b08a57ecae in charmhelpers switch the mysql helper to directly use leader_get/leader_set, rather than using the original peer storage helper which deals with the migration transparently.

That happened in March 2017, so I think the main issue here is the long jump between charm revisions for this specific upgrade. Automated testing will only ever upgrade from current stable to proposed stable so we don't cover this specific use-case.

[James Page (james-page)]

I think we need to introduce a charm specific migration step to resolve this specific issue.

[Alex Kavanagh (ajkavanagh)]

Adding it to the charm-guide for a note for upgraders. It looks like the issue is that the migration code in charm-helpers was to get things from disk to the peer relation, and now it tries to get things from disk to the leader-settings. What it doesn't do is migrate the passwords from the peer-relation to the leader-settings.