Scale out: Adding new node fails

[Andrew McLeod]

focal ussuri s390x

Install hook intermittently failing (e.g. on one of 3 units) with the following error:

subprocess.CalledProcessError: Command '['/snap/bin/mysql-shell.mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmpl8_2dg_k.py']' returned non-zero exit status 1

reason: the tmp file does not exist in the directory (the directory is there and contains nothing)

https://pastebin.canonical.com/p/YmV77wWWpD/

[David Ames (thedac)]

Triage:

Per the Traceback

One of the non-leaders is executing config_changed during the install hook. config_changed calls wait_until_cluster_available but the cluster is not ready yet.

More precise flags are required on the config_changed reactive handler.

[David Ames (thedac)]

Correction, it is the leader that is running config_changed too early.

[David Ames (thedac)]

This is a scale out failure when adding a node to the cluster

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728585

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728585
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-innodb-cluster/commit/?id=5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Submitter: Zuul
Branch: master

commit 5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Author: David Ames <email address hidden>
Date: Fri May 15 15:17:59 2020 -0700

    Scale-in and Scale-out of mysql-innodb-cluster

    Handle adding and removing units to the mysql-innodb-cluster.

    Change-Id: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Closes-Bug: #1877546
    Closes-Bug: #1874479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728587
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=7cbf9a88b001d73d290bb123f5bba010f51319b3
Submitter: Zuul
Branch: master

commit 7cbf9a88b001d73d290bb123f5bba010f51319b3
Author: David Ames <email address hidden>
Date: Wed May 13 11:37:31 2020 -0700

    Scale in and scale out

    Handle adding and removing units of the mysql-innodb-cluster gracefully.

    Review and land:
    https://github.com/openstack-charmers/zaza/pull/353
    https://github.com/openstack-charmers/zaza-openstack-tests/pull/275

    Closes-Bug: #1874479
    Closes-Bug: #1877546
    Closes-Bug: #1877616
    Depends-On: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Change-Id: I39210c760e7204f84365d5dcb4eeedbc0889041f