hook failed: "leader-elected"

[Wouter Bakker]

hook failed: "leader-elected"

---
/var/log/gnocchi/gnocchi-upgrade.log

2021-09-26 19:22:11,106 [295842] INFO gnocchi.service: Gnocchi version 4.3.4
2021-09-26 19:22:11,331 [295842] INFO gnocchi.cli.manage: Upgrading indexer SQLAlchemyIndexer: mysql+pymysql://***:***@10.11.11.125:3306/gnocchi?binary_prefix=true
2021-09-26 19:22:11,337 [295842] WARNING py.warnings: /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
  result = self._query(query)

2021-09-26 19:22:11,347 [295842] WARNING oslo_db.sqlalchemy.exc_filters: DBAPIError exception wrapped.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.InternalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')
2021-09-26 19:22:11,349 [295842] CRITICAL root: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.InternalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/gnocchi-upgrade", line 11, in <module>
    load_entry_point('gnocchi==4.3.4', 'console_scripts', 'gnocchi-upgrade')()
  File "/usr/lib/python3/dist-packages/gnocchi/cli/manage.py", line 69, in upgrade
    index.upgrade()
  File "/usr/lib/python3/dist-packages/gnocchi/indexer/sqlalchemy.py", line 334, in upgrade
    Base.metadata.create_all(connection)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/schema.py", line 4315, in create_all
    bind._run_visitor(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1618, in _run_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/visitors.py", line 138, in traverse_single
    return meth(obj, **kw)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/ddl.py", line 777, in visit_metadata
    self.traverse_single(
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/visitors.py", line 138, in traverse_single
    return meth(obj, **kw)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/ddl.py", line 821, in visit_table
    self.connection.execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1039, in _execute_ddl
    ret = self._execute_context(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1474, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
oslo_db.exception.DBError: (pymysql.err.InternalError) (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')
[SQL:
CREATE TABLE archive_policy (
 name VARCHAR(255) NOT NULL,
 back_window INTEGER NOT NULL,
 definition TEXT NOT NULL,
 aggregation_methods TEXT NOT NULL,
 PRIMARY KEY (name)
)CHARSET=utf8 ENGINE=InnoDB

]
(Background on this error at: http://sqlalche.me/e/2j85)

---
juju status

Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller maas/default 2.8.10 unsupported 19:21:44Z

App Version Status Scale Charm Store Channel Rev OS Message
ceilometer 14.0.0 blocked 3 ceilometer charmstore 284 ubuntu Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
ceilometer-hacluster active 3 hacluster charmstore 78 ubuntu Unit is ready and clustered
ceph-mon 15.2.12 active 3 ceph-mon charmstore 55 ubuntu Unit is ready and clustered
ceph-osd 15.2.12 active 3 ceph-osd charmstore 308 ubuntu Unit is ready (1 OSD)
ceph-radosgw 15.2.12 active 3 ceph-radosgw charmstore 296 ubuntu Unit is ready
cinder 16.2.1 active 3 cinder charmstore 310 ubuntu Unit is ready
cinder-ceph 16.2.1 active 3 cinder-ceph charmstore 262 ubuntu Unit is ready
cinder-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
cinder-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
dashboard-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
easyrsa 3.0.1 active 1 easyrsa charmstore 342 ubuntu Certificate Authority connected.
etcd 3.1.10 active 3 etcd charmstore 569 ubuntu Healthy with 3 known peers
glance 20.0.1 active 3 glance charmstore 305 ubuntu Unit is ready
glance-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
glance-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
glance-simplestreams-sync active 1 glance-simplestreams-sync charmstore 42 ubuntu Unit is ready (Sync completed at 09/26/21 19:17:07)
gnocchi 4.3.4 error 3 gnocchi charmstore 48 ubuntu hook failed: "leader-elected"
gnocchi-hacluster active 3 hacluster charmstore 78 ubuntu Unit is ready and clustered
gnocchi-mysql-router 8.0.26 active 3 mysql-router charmstore 11 ubuntu Unit is ready
keystone 17.0.0 active 3 keystone charmstore 323 ubuntu Application Ready
keystone-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
keystone-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
memcached active 1 memcached charmstore 32 ubuntu Unit is ready
mongodb unknown 0 mongodb charmstore 60 ubuntu
mysql-innodb-cluster 8.0.26 active 3 mysql-innodb-cluster charmstore 11 ubuntu Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
neutron-api 16.3.0 active 3 neutron-api charmstore 294 ubuntu Unit is ready
neutron-api-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
neutron-api-plugin-ovn 16.3.0 active 3 neutron-api-plugin-ovn charmstore 6 ubuntu Unit is ready
neutron-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
nova-cloud-controller 21.1.2 active 3 nova-cloud-controller charmstore 355 ubuntu Unit is ready
nova-cloud-controller-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
nova-compute 21.2.0 active 3 nova-compute charmstore 327 ubuntu Unit is ready
nova-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
ntp 3.5 active 3 ntp charmstore 45 ubuntu chrony: Ready
openstack-dashboard 18.3.2 active 3 openstack-dashboard charmstore 313 ubuntu Unit is ready
openstack-dashboard-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
openstack-trove unknown 0 openstack-trove charmstore 4 ubuntu
ovn-central 20.03.1 active 3 ovn-central charmstore 7 ubuntu Unit is ready
ovn-chassis 20.03.2 active 3 ovn-chassis charmstore 14 ubuntu Unit is ready
placement 3.0.0 active 3 placement charmstore 19 ubuntu Unit is ready
placement-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
placement-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready
rabbitmq-server 3.8.2 active 3 rabbitmq-server charmstore 110 ubuntu Unit is ready and clustered
radosgw-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
vault 1.5.4 active 3 vault charmstore 46 ubuntu Unit is ready (active: false, mlock: disabled)
vault-hacluster active 3 hacluster charmstore 76 ubuntu Unit is ready and clustered
vault-mysql-router 8.0.26 active 3 mysql-router charmstore 8 ubuntu Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceilometer/4 blocked idle 9/lxd/54 10.11.11.186 Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/3 active idle 10.11.11.186 Unit is ready and clustered
ceilometer/5* blocked idle 10/lxd/33 10.11.11.187 Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/1* active idle 10.11.11.187 Unit is ready and clustered
ceilometer/6 blocked idle 13/lxd/29 10.11.11.188 Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/2 active idle 10.11.11.188 Unit is ready and clustered
ceph-mon/8* active idle 10/lxd/4 10.11.11.45 Unit is ready and clustered
ceph-mon/9 active idle 9/lxd/35 10.11.11.122 Unit is ready and clustered
ceph-mon/10 active idle 13/lxd/4 10.11.11.128 Unit is ready and clustered
ceph-osd/3* active idle 9 10.11.11.55 Unit is ready (1 OSD)
ceph-osd/4 active idle 10 10.11.11.40 Unit is ready (1 OSD)
ceph-osd/5 active idle 13 10.11.11.123 Unit is ready (1 OSD)
ceph-radosgw/3 active idle 9/lxd/4 10.11.11.1 443/tcp Unit is ready
  radosgw-hacluster/5 active idle 10.11.11.1 Unit is ready and clustered
ceph-radosgw/5 active idle 10/lxd/5 10.11.11.46 443/tcp Unit is ready
  radosgw-hacluster/9 active idle 10.11.11.46 Unit is ready and clustered
ceph-radosgw/6* active idle 13/lxd/5 10.11.11.129 443/tcp Unit is ready
  radosgw-hacluster/10* active idle 10.11.11.129 Unit is ready and clustered
cinder/4* active idle 9/lxd/5 10.11.11.2 8776/tcp Unit is ready
  cinder-ceph/12 active idle 10.11.11.2 Unit is ready
  cinder-hacluster/5 active idle 10.11.11.2 Unit is ready and clustered
  cinder-mysql-router/5* active idle 10.11.11.2 Unit is ready
cinder/5 active idle 10/lxd/6 10.11.11.47 8776/tcp Unit is ready
  cinder-ceph/17* active idle 10.11.11.47 Unit is ready
  cinder-hacluster/9* active idle 10.11.11.47 Unit is ready and clustered
  cinder-mysql-router/7 active idle 10.11.11.47 Unit is ready
cinder/6 active idle 13/lxd/8 10.11.11.132 8776/tcp Unit is ready
  cinder-ceph/18 active idle 10.11.11.132 Unit is ready
  cinder-hacluster/12 active idle 10.11.11.132 Unit is ready and clustered
  cinder-mysql-router/8 active idle 10.11.11.132 Unit is ready
easyrsa/3* active idle 9/lxd/32 10.11.11.39 Certificate Authority connected.
etcd/12* active idle 13/lxd/16 10.11.11.142 2379/tcp Healthy with 3 known peers
etcd/13 active idle 9/lxd/37 10.11.11.143 2379/tcp Healthy with 3 known peers
etcd/14 active idle 10/lxd/19 10.11.11.144 2379/tcp Healthy with 3 known peers
glance-simplestreams-sync/3* active idle 13/lxd/24 10.11.11.170 Unit is ready (Sync completed at 09/26/21 19:17:07)
glance/11 active idle 13/lxd/9 10.11.11.133 9292/tcp Unit is ready
  glance-hacluster/25* active idle 10.11.11.133 Unit is ready and clustered
  glance-mysql-router/13* active idle 10.11.11.133 Unit is ready
glance/12 active idle 9/lxd/47 10.11.11.167 9292/tcp Unit is ready
  glance-hacluster/26 active idle 10.11.11.167 Unit is ready and clustered
  glance-mysql-router/14 active idle 10.11.11.167 Unit is ready
glance/13* active idle 10/lxd/26 10.11.11.168 9292/tcp Unit is ready
  glance-hacluster/27 active idle 10.11.11.168 Unit is ready and clustered
  glance-mysql-router/15 active idle 10.11.11.168 Unit is ready
gnocchi/8* error idle 9/lxd/52 10.11.11.182 8041/tcp hook failed: "leader-elected"
  gnocchi-hacluster/6* active idle 10.11.11.182 Unit is ready and clustered
  gnocchi-mysql-router/7* active idle 10.11.11.182 Unit is ready
gnocchi/9 active idle 10/lxd/31 10.11.11.180 8041/tcp Unit is ready
  gnocchi-hacluster/7 active idle 10.11.11.180 Unit is ready and clustered
  gnocchi-mysql-router/9 active idle 10.11.11.180 Unit is ready
gnocchi/10 active idle 13/lxd/27 10.11.11.181 8041/tcp Unit is ready
  gnocchi-hacluster/5 active idle 10.11.11.181 Unit is ready and clustered
  gnocchi-mysql-router/6 active idle 10.11.11.181 Unit is ready
keystone/16 active idle 9/lxd/34 10.11.11.118 5000/tcp Unit is ready
  keystone-hacluster/50 active idle 10.11.11.118 Unit is ready and clustered
  keystone-mysql-router/37 active idle 10.11.11.118 Unit is ready
keystone/17* active idle 10/lxd/15 10.11.11.119 5000/tcp Unit is ready
  keystone-hacluster/52 active idle 10.11.11.119 Unit is ready and clustered
  keystone-mysql-router/39 active idle 10.11.11.119 Unit is ready
keystone/18 active idle 13/lxd/3 10.11.11.127 5000/tcp Unit is ready
  keystone-hacluster/54* active idle 10.11.11.127 Unit is ready and clustered
  keystone-mysql-router/40* active idle 10.11.11.127 Unit is ready
memcached/4* active idle 9/lxd/31 10.11.11.38 11211/tcp Unit is ready
mysql-innodb-cluster/9 active idle 9/lxd/26 10.11.11.18 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/13 active idle 10/lxd/17 10.11.11.121 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/14* active idle 13/lxd/1 10.11.11.125 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
neutron-api/5 active idle 9/lxd/19 10.11.11.9 9696/tcp Unit is ready
  neutron-api-hacluster/4 active idle 10.11.11.9 Unit is ready and clustered
  neutron-api-plugin-ovn/4 active idle 10.11.11.9 Unit is ready
  neutron-mysql-router/5* active idle 10.11.11.9 Unit is ready
neutron-api/6 active idle 10/lxd/7 10.11.11.48 9696/tcp Unit is ready
  neutron-api-hacluster/5* active idle 10.11.11.48 Unit is ready and clustered
  neutron-api-plugin-ovn/6* active idle 10.11.11.48 Unit is ready
  neutron-mysql-router/6 active idle 10.11.11.48 Unit is ready
neutron-api/7* active idle 13/lxd/7 10.11.11.131 9696/tcp Unit is ready
  neutron-api-hacluster/6 active idle 10.11.11.131 Unit is ready and clustered
  neutron-api-plugin-ovn/7 active idle 10.11.11.131 Unit is ready
  neutron-mysql-router/8 active idle 10.11.11.131 Unit is ready
nova-cloud-controller/5 active idle 9/lxd/20 10.11.11.10 8774/tcp,8775/tcp Unit is ready
  nova-cloud-controller-hacluster/9 active idle 10.11.11.10 Unit is ready and clustered
  nova-mysql-router/6* active idle 10.11.11.10 Unit is ready
nova-cloud-controller/8 active idle 10/lxd/10 10.11.11.110 8774/tcp,8775/tcp Unit is ready
  nova-cloud-controller-hacluster/13* active idle 10.11.11.110 Unit is ready and clustered
  nova-mysql-router/9 active idle 10.11.11.110 Unit is ready
nova-cloud-controller/9* active idle 13/lxd/11 10.11.11.135 8774/tcp,8775/tcp Unit is ready
  nova-cloud-controller-hacluster/14 active idle 10.11.11.135 Unit is ready and clustered
  nova-mysql-router/10 active idle 10.11.11.135 Unit is ready
nova-compute/5 active idle 9 10.11.11.55 Unit is ready
  ntp/24* active idle 10.11.11.55 123/udp chrony: Ready
  ovn-chassis/30 active idle 10.11.11.55 Unit is ready
nova-compute/6* active idle 10 10.11.11.40 Unit is ready
  ntp/46 active idle 10.11.11.40 123/udp chrony: Ready
  ovn-chassis/29 active idle 10.11.11.40 Unit is ready
nova-compute/7 active idle 13 10.11.11.123 Unit is ready
  ntp/54 active idle 10.11.11.123 123/udp chrony: Ready
  ovn-chassis/31* active idle 10.11.11.123 Unit is ready
openstack-dashboard/4 active idle 9/lxd/21 10.11.11.11 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/4 active idle 10.11.11.11 Unit is ready
  openstack-dashboard-hacluster/3 active idle 10.11.11.11 Unit is ready and clustered
openstack-dashboard/5* active idle 10/lxd/11 10.11.11.111 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/6 active idle 10.11.11.111 Unit is ready
  openstack-dashboard-hacluster/4 active idle 10.11.11.111 Unit is ready and clustered
openstack-dashboard/6 active idle 13/lxd/10 10.11.11.134 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/7* active idle 10.11.11.134 Unit is ready
  openstack-dashboard-hacluster/5* active idle 10.11.11.134 Unit is ready and clustered
ovn-central/28 active idle 9/lxd/46 10.11.11.165 6641/tcp,6642/tcp Unit is ready
ovn-central/29 active idle 10/lxd/25 10.11.11.166 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db northd: active)
ovn-central/30* active idle 13/lxd/23 10.11.11.164 6641/tcp,6642/tcp Unit is ready (leader: ovnsb_db)
placement/6 active idle 9/lxd/24 10.11.11.16 8778/tcp Unit is ready
  placement-hacluster/7* active idle 10.11.11.16 Unit is ready and clustered
  placement-mysql-router/7 active idle 10.11.11.16 Unit is ready
placement/7 active idle 10/lxd/13 10.11.11.113 8778/tcp Unit is ready
  placement-hacluster/9 active idle 10.11.11.113 Unit is ready and clustered
  placement-mysql-router/8 active idle 10.11.11.113 Unit is ready
placement/8* active idle 13/lxd/14 10.11.11.138 8778/tcp Unit is ready
  placement-hacluster/10 active idle 10.11.11.138 Unit is ready and clustered
  placement-mysql-router/9* active idle 10.11.11.138 Unit is ready
rabbitmq-server/3* active idle 9/lxd/1 10.11.11.65 5672/tcp Unit is ready and clustered
rabbitmq-server/4 active idle 10/lxd/12 10.11.11.112 5672/tcp Unit is ready and clustered
rabbitmq-server/5 active idle 13/lxd/6 10.11.11.130 5672/tcp Unit is ready and clustered
vault/11* active idle 9/lxd/38 10.11.11.145 8200/tcp Unit is ready (active: false, mlock: disabled)
  vault-hacluster/9* active idle 10.11.11.145 Unit is ready and clustered
  vault-mysql-router/9 active idle 10.11.11.145 Unit is ready
vault/12 active idle 10/lxd/20 10.11.11.146 8200/tcp Unit is ready (active: true, mlock: disabled)
  vault-hacluster/8 active idle 10.11.11.146 Unit is ready and clustered
  vault-mysql-router/8 active idle 10.11.11.146 Unit is ready
vault/13 active idle 13/lxd/17 10.11.11.147 8200/tcp Unit is ready (active: false, mlock: disabled)
  vault-hacluster/10 active idle 10.11.11.147 Unit is ready and clustered
  vault-mysql-router/10* active idle 10.11.11.147 Unit is ready

Machine State DNS Inst id Series AZ Message
9 started 10.11.11.55 orcus focal NL-Dronten Deployed
9/lxd/1 started 10.11.11.65 juju-16c2c5-9-lxd-1 focal NL-Dronten Container started
9/lxd/4 started 10.11.11.1 juju-16c2c5-9-lxd-4 focal NL-Dronten Container started
9/lxd/5 started 10.11.11.2 juju-16c2c5-9-lxd-5 focal NL-Dronten Container started
9/lxd/19 started 10.11.11.9 juju-16c2c5-9-lxd-19 focal NL-Dronten Container started
9/lxd/20 started 10.11.11.10 juju-16c2c5-9-lxd-20 focal NL-Dronten Container started
9/lxd/21 started 10.11.11.11 juju-16c2c5-9-lxd-21 focal NL-Dronten Container started
9/lxd/24 started 10.11.11.16 juju-16c2c5-9-lxd-24 focal NL-Dronten Container started
9/lxd/26 started 10.11.11.18 juju-16c2c5-9-lxd-26 focal NL-Dronten Container started
9/lxd/31 started 10.11.11.38 juju-16c2c5-9-lxd-31 bionic NL-Dronten Container started
9/lxd/32 started 10.11.11.39 juju-16c2c5-9-lxd-32 bionic NL-Dronten Container started
9/lxd/34 started 10.11.11.118 juju-16c2c5-9-lxd-34 focal NL-Dronten Container started
9/lxd/35 started 10.11.11.122 juju-16c2c5-9-lxd-35 focal NL-Dronten Container started
9/lxd/37 started 10.11.11.143 juju-16c2c5-9-lxd-37 bionic NL-Dronten Container started
9/lxd/38 started 10.11.11.145 juju-16c2c5-9-lxd-38 focal NL-Dronten Container started
9/lxd/46 started 10.11.11.165 juju-16c2c5-9-lxd-46 focal NL-Dronten Container started
9/lxd/47 started 10.11.11.167 juju-16c2c5-9-lxd-47 focal NL-Dronten Container started
9/lxd/52 started 10.11.11.182 juju-16c2c5-9-lxd-52 focal NL-Dronten Container started
9/lxd/54 started 10.11.11.186 juju-16c2c5-9-lxd-54 focal NL-Dronten Container started
10 started 10.11.11.40 titan focal default Deployed
10/lxd/4 started 10.11.11.45 juju-16c2c5-10-lxd-4 focal default Container started
10/lxd/5 started 10.11.11.46 juju-16c2c5-10-lxd-5 focal default Container started
10/lxd/6 started 10.11.11.47 juju-16c2c5-10-lxd-6 focal default Container started
10/lxd/7 started 10.11.11.48 juju-16c2c5-10-lxd-7 focal default Container started
10/lxd/10 started 10.11.11.110 juju-16c2c5-10-lxd-10 focal default Container started
10/lxd/11 started 10.11.11.111 juju-16c2c5-10-lxd-11 focal default Container started
10/lxd/12 started 10.11.11.112 juju-16c2c5-10-lxd-12 focal default Container started
10/lxd/13 started 10.11.11.113 juju-16c2c5-10-lxd-13 focal default Container started
10/lxd/15 started 10.11.11.119 juju-16c2c5-10-lxd-15 focal default Container started
10/lxd/17 started 10.11.11.121 juju-16c2c5-10-lxd-17 focal default Container started
10/lxd/19 started 10.11.11.144 juju-16c2c5-10-lxd-19 bionic default Container started
10/lxd/20 started 10.11.11.146 juju-16c2c5-10-lxd-20 focal default Container started
10/lxd/25 started 10.11.11.166 juju-16c2c5-10-lxd-25 focal default Container started
10/lxd/26 started 10.11.11.168 juju-16c2c5-10-lxd-26 focal default Container started
10/lxd/31 started 10.11.11.180 juju-16c2c5-10-lxd-31 focal default Container started
10/lxd/33 started 10.11.11.187 juju-16c2c5-10-lxd-33 focal default Container started
13 started 10.11.11.123 makemake focal default Deployed
13/lxd/1 started 10.11.11.125 juju-16c2c5-13-lxd-1 focal default Container started
13/lxd/3 started 10.11.11.127 juju-16c2c5-13-lxd-3 focal default Container started
13/lxd/4 started 10.11.11.128 juju-16c2c5-13-lxd-4 focal default Container started
13/lxd/5 started 10.11.11.129 juju-16c2c5-13-lxd-5 focal default Container started
13/lxd/6 started 10.11.11.130 juju-16c2c5-13-lxd-6 focal default Container started
13/lxd/7 started 10.11.11.131 juju-16c2c5-13-lxd-7 focal default Container started
13/lxd/8 started 10.11.11.132 juju-16c2c5-13-lxd-8 focal default Container started
13/lxd/9 started 10.11.11.133 juju-16c2c5-13-lxd-9 focal default Container started
13/lxd/10 started 10.11.11.134 juju-16c2c5-13-lxd-10 focal default Container started
13/lxd/11 started 10.11.11.135 juju-16c2c5-13-lxd-11 focal default Container started
13/lxd/14 started 10.11.11.138 juju-16c2c5-13-lxd-14 focal default Container started
13/lxd/16 started 10.11.11.142 juju-16c2c5-13-lxd-16 bionic default Container started
13/lxd/17 started 10.11.11.147 juju-16c2c5-13-lxd-17 focal default Container started
13/lxd/23 started 10.11.11.164 juju-16c2c5-13-lxd-23 focal default Container started
13/lxd/24 started 10.11.11.170 juju-16c2c5-13-lxd-24 bionic default Container started
13/lxd/27 started 10.11.11.181 juju-16c2c5-13-lxd-27 focal default Container started
13/lxd/29 started 10.11.11.188 juju-16c2c5-13-lxd-29 focal default Container started