subprocess.CalledProcessError: Command '['leader-set', 'cluster-instance-configured-10-5-0-29=True']' returned non-zero exit status 1

[Felipe Reyes]

I have an environment in this state:

$ juju status mysql-innodb-cluster
Model Controller Cloud/Region Version SLA Timestamp
openidc serverstack serverstack/serverstack 2.9.29 unsupported 09:40:51-04:00

App Version Status Scale Charm Channel Rev Exposed Message
mysql-innodb-cluster 8.0.30 waiting 3 mysql-innodb-cluster 8.0/stable 26 no Instance not yet configured for clustering

Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0 active idle 0 10.5.0.100 Unit is ready: Mode: R/W, Cluster is NOT tolerant to any failures. 2 members are not active.
mysql-innodb-cluster/3* waiting idle 20 10.5.0.29 Instance not yet configured for clustering
mysql-innodb-cluster/4 waiting idle 21 10.5.3.54 Instance not yet configured for clustering

Machine State Address Inst id Series AZ Message
0 started 10.5.0.100 133fe97e-8523-4d66-b915-bd4b06cb7dca focal nova ACTIVE
20 started 10.5.0.29 08fccbdb-c13f-4258-9a6d-1375b3757b71 focal nova ACTIVE
21 started 10.5.3.54 477c28e2-7975-4344-91f4-754c1ca51417 focal nova ACTIVE

When running add-instance the action fails to run leader-set

$ juju run-action mysql-innodb-cluster/0 add-instance address=10.5.0.29 --wait
unit-mysql-innodb-cluster-0:
  UnitId: mysql-innodb-cluster/0
  id: "514"
  message: exit status 1
  results:
    ReturnCode: 1
    Stderr: |
      ERROR cannot write leadership settings: cannot write settings: not the leader
      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 265, in add_instance
          output = instance.configure_and_add_instance(address)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1091, in configure_and_add_instance
          self.configure_instance(address)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 614, in configure_instance
          leadership.leader_set({
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.8/site-packages/charms/reactive/decorators.py", line 219, in _wrapped
          return func(*args, **kwargs)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charms/leadership.py", line 62, in leader_set
          hookenv.leader_set(settings)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 1180, in inner_translate_exc2
          return f(*args, **kwargs)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 1241, in leader_set
          subprocess.check_call(cmd)
        File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['leader-set', 'cluster-instance-configured-10-5-0-29=True']' returned non-zero exit status 1.

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 379, in main
          action(args)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 272, in add_instance
          "output": e.stderr.decode("UTF-8"),
      AttributeError: 'NoneType' object has no attribute 'decode'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 391, in <module>
          sys.exit(main(sys.argv))
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 382, in main
          "output": e.output.decode("UTF-8"),
      AttributeError: 'AttributeError' object has no attribute 'output'
  status: failed
  timing:
    completed: 2022-08-10 13:39:34 +0000 UTC
    enqueued: 2022-08-10 13:39:01 +0000 UTC
    started: 2022-08-10 13:39:03 +0000 UTC

The documentation of the action states that add-instance must be run on a functioning member of the cluster, but says nothing about the leadership.

Action Description
add-instance Configure and add an instance to the cluster. *Note* This action must be
                                     run on an instance that is a functioning member of the cluster.

The action should handle correctly the fact that's not the leader and delegate that operation to the leader, probably using a peer relation.

[Vladimir Grevtsev (vlgrevtsev)]

Can confirm that this is happening once one is trying to add a cluster member from the alive unit, which isn't the leader:

$ juju status mysql-innodb-cluster
Model Controller Cloud/Region Version SLA Timestamp
openstack foundations-maas maas_cloud 2.9.42 unsupported 07:39:54Z

App Version Status Scale Charm Channel Rev Exposed Message
logrotated active 3 logrotated stable 10 no Unit is ready.
mysql-innodb-cluster 8.0.32 blocked 3 mysql-innodb-cluster 8.0/stable 43 no Cluster is inaccessible from this instance. Please check logs for details.

Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0 blocked idle 5/lxd/2 172.27.85.230 Cluster is inaccessible from this instance. Please check logs for details.
  logrotated/26 active idle 172.27.85.230 Unit is ready.
mysql-innodb-cluster/1 blocked idle 6/lxd/1 172.27.85.249 Cluster is inaccessible from this instance. Please check logs for details.
  logrotated/25 active idle 172.27.85.249 Unit is ready.
mysql-innodb-cluster/2* blocked executing 7/lxd/2 172.27.85.237 Cluster is inaccessible from this instance. Please check logs for details.
  logrotated/23 active executing 172.27.85.237 Unit is ready.

Machine State Address Inst id Series AZ Message
5 started 172.27.85.222 node06 focal default Deployed
5/lxd/2 started 172.27.85.230 juju-2f3dae-5-lxd-2 focal default Container started
6 started 172.27.85.223 node07 focal default Deployed
6/lxd/1 started 172.27.85.249 juju-2f3dae-6-lxd-1 focal default Container started
7 started 172.27.85.225 node08 focal default Deployed
7/lxd/2 started 172.27.85.237 juju-2f3dae-7-lxd-2 focal default Container started

130 ubuntu@orangebox84:~$ juju run-action mysql-innodb-cluster/0 add-instance address=172.27.84.111 --wait
unit-mysql-innodb-cluster-0:
  UnitId: mysql-innodb-cluster/0
  id: "90"
  message: exit status 1
  results:
    ReturnCode: 1
    Stderr: |
      ERROR cannot write leadership settings: cannot write settings: not the leader
      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/actions/add-instance", line 265, in add_instance
          output = instance.configure_and_add_instance(address)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1091, in configure_and_add_instance
          self.configure_instance(address)
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 614, in configure_instance
          leadership.leader_set({
        File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.8/site-packages/charms/reactive/decorators.py", line 219, in _wrapped
      ...