subprocess.CalledProcessError: Command '['leader-set', 'cluster-instance-configured-10-5-0-29=True']' returned non-zero exit status 1
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
I have an environment in this state:
$ juju status mysql-innodb-
Model Controller Cloud/Region Version SLA Timestamp
openidc serverstack serverstack/
App Version Status Scale Charm Channel Rev Exposed Message
mysql-innodb-
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-
mysql-innodb-
mysql-innodb-
Machine State Address Inst id Series AZ Message
0 started 10.5.0.100 133fe97e-
20 started 10.5.0.29 08fccbdb-
21 started 10.5.3.54 477c28e2-
When running add-instance the action fails to run leader-set
$ juju run-action mysql-innodb-
unit-mysql-
UnitId: mysql-innodb-
id: "514"
message: exit status 1
results:
ReturnCode: 1
Stderr: |
ERROR cannot write leadership settings: cannot write settings: not the leader
Traceback (most recent call last):
File "/var/lib/
output = instance.
File "/var/lib/
File "/var/lib/
File "/var/lib/
return func(*args, **kwargs)
File "/var/lib/
File "/var/lib/
return f(*args, **kwargs)
File "/var/lib/
File "/usr/lib/
raise CalledProcessEr
subproces
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/
File "/var/lib/
"output": e.stderr.
Attribute
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/
File "/var/lib/
"output": e.output.
Attribute
status: failed
timing:
completed: 2022-08-10 13:39:34 +0000 UTC
enqueued: 2022-08-10 13:39:01 +0000 UTC
started: 2022-08-10 13:39:03 +0000 UTC
The documentation of the action states that add-instance must be run on a functioning member of the cluster, but says nothing about the leadership.
Action Description
add-instance Configure and add an instance to the cluster. *Note* This action must be
The action should handle correctly the fact that's not the leader and delegate that operation to the leader, probably using a peer relation.
Can confirm that this is happening once one is trying to add a cluster member from the alive unit, which isn't the leader:
$ juju status mysql-innodb- cluster
Model Controller Cloud/Region Version SLA Timestamp
openstack foundations-maas maas_cloud 2.9.42 unsupported 07:39:54Z
App Version Status Scale Charm Channel Rev Exposed Message cluster 8.0.32 blocked 3 mysql-innodb- cluster 8.0/stable 43 no Cluster is inaccessible from this instance. Please check logs for details.
logrotated active 3 logrotated stable 10 no Unit is ready.
mysql-innodb-
Unit Workload Agent Machine Public address Ports Message cluster/ 0 blocked idle 5/lxd/2 172.27.85.230 Cluster is inaccessible from this instance. Please check logs for details. cluster/ 1 blocked idle 6/lxd/1 172.27.85.249 Cluster is inaccessible from this instance. Please check logs for details. cluster/ 2* blocked executing 7/lxd/2 172.27.85.237 Cluster is inaccessible from this instance. Please check logs for details.
mysql-innodb-
logrotated/26 active idle 172.27.85.230 Unit is ready.
mysql-innodb-
logrotated/25 active idle 172.27.85.249 Unit is ready.
mysql-innodb-
logrotated/23 active executing 172.27.85.237 Unit is ready.
Machine State Address Inst id Series AZ Message
5 started 172.27.85.222 node06 focal default Deployed
5/lxd/2 started 172.27.85.230 juju-2f3dae-5-lxd-2 focal default Container started
6 started 172.27.85.223 node07 focal default Deployed
6/lxd/1 started 172.27.85.249 juju-2f3dae-6-lxd-1 focal default Container started
7 started 172.27.85.225 node08 focal default Deployed
7/lxd/2 started 172.27.85.237 juju-2f3dae-7-lxd-2 focal default Container started
130 ubuntu@ orangebox84: ~$ juju run-action mysql-innodb- cluster/ 0 add-instance address= 172.27. 84.111 --wait innodb- cluster- 0: cluster/ 0 juju/agents/ unit-mysql- innodb- cluster- 0/charm/ actions/ add-instance" , line 265, in add_instance configure_ and_add_ instance( address) juju/agents/ unit-mysql- innodb- cluster- 0/charm/ lib/charm/ openstack/ mysql_innodb_ cluster. py", line 1091, in configure_ and_add_ instance
self. configure_ instance( address) juju/agents/ unit-mysql- innodb- cluster- 0/charm/ lib/charm/ openstack/ mysql_innodb_ cluster. py", line 614, in configure_instance
leadership. leader_ set({ juju/agents/ unit-mysql- innodb- cluster- 0/.venv/ lib/python3. 8/site- packages/ charms/ reactive/ decorators. py", line 219, in _wrapped
unit-mysql-
UnitId: mysql-innodb-
id: "90"
message: exit status 1
results:
ReturnCode: 1
Stderr: |
ERROR cannot write leadership settings: cannot write settings: not the leader
Traceback (most recent call last):
File "/var/lib/
output = instance.
File "/var/lib/
File "/var/lib/
File "/var/lib/
...