Juju won't add nova-compute to the correct AZ, even with customize-failure-domain: true

Bug #1888827 reported by Mirek
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Triaged
High
Unassigned

Bug Description

As in the title I have 3 zone setup in MAAS, Juju is deploying all units correctly, CEPH has correct CRASH map spread across the zones, but nova-compute hosts always end up in NOVA default AZ.
I even tried with pre-created AZ in OpenStack so names do match JUJU but still, the new host always ends up in NOVA AZ. I've confirmed Juju has JUJU_AVAILABILITY_ZONE set and that the customize-failure-domain is set to True but it doesn't change anything.
juju run --unit nova-compute/0 'env | grep JUJU_AVAILABILITY_ZONE'.

Model Controller Cloud/Region Version SLA Timestamp
openstack-ha maas-controller lab-maas/default 2.8.0 unsupported 10:10:10Z

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 15.2.3 active 3 ceph-mon jujucharms 48 ubuntu
ceph-osd 15.2.3 active 3 ceph-osd jujucharms 303 ubuntu
ceph-radosgw 15.2.3 active 3 ceph-radosgw jujucharms 289 ubuntu
cinder 16.1.0 active 3 cinder jujucharms 303 ubuntu
cinder-ceph 16.1.0 active 3 cinder-ceph jujucharms 256 ubuntu
cinder-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
dashboard-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
glance 20.0.0 active 3 glance jujucharms 297 ubuntu
glance-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
hacluster-ceph-radosgw active 3 hacluster jujucharms 68 ubuntu
hacluster-cinder active 3 hacluster jujucharms 68 ubuntu
hacluster-glance active 3 hacluster jujucharms 68 ubuntu
hacluster-horizon active 3 hacluster jujucharms 68 ubuntu
hacluster-keystone active 3 hacluster jujucharms 68 ubuntu
hacluster-neutron active 3 hacluster jujucharms 68 ubuntu
hacluster-nova active 3 hacluster jujucharms 68 ubuntu
hacluster-placement active 3 hacluster jujucharms 68 ubuntu
keystone 17.0.0 active 3 keystone jujucharms 316 ubuntu
keystone-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
memcached active 3 memcached jujucharms 29 ubuntu
mysql-innodb-cluster 8.0.20 active 3 mysql-innodb-cluster jujucharms 0 ubuntu
neutron-api 16.0.0 active 3 neutron-api jujucharms 286 ubuntu
neutron-api-plugin-ovn 16.0.0 active 3 neutron-api-plugin-ovn jujucharms 0 ubuntu
neutron-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
nova-cloud-controller 21.0.0 active 3 nova-cloud-controller jujucharms 345 ubuntu
nova-compute 21.0.0 active 2 nova-compute jujucharms 318 ubuntu
nova-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
openstack-dashboard 18.3.2 active 3 openstack-dashboard jujucharms 304 ubuntu
ovn-central 20.03.0 active 3 ovn-central jujucharms 0 ubuntu
ovn-chassis 20.03.0 active 2 ovn-chassis jujucharms 2 ubuntu
placement 3.0.0 active 3 placement jujucharms 11 ubuntu
placement-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
rabbitmq-server 3.8.2 active 3 rabbitmq-server jujucharms 103 ubuntu
vault 1.1.1 active 1 vault jujucharms 39 ubuntu
vault-mysql-router 8.0.20 active 1 mysql-router jujucharms 2 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0/lxd/0 10.10.70.115 Unit is ready and clustered
ceph-mon/1 active idle 1/lxd/0 10.10.70.113 Unit is ready and clustered
ceph-mon/2* active idle 2/lxd/0 10.10.70.112 Unit is ready and clustered
ceph-osd/0 active idle 0 10.10.70.101 Unit is ready (1 OSD)
ceph-osd/1 active idle 1 10.10.70.102 Unit is ready (1 OSD)
ceph-osd/2* active idle 2 10.10.70.103 Unit is ready (1 OSD)
ceph-radosgw/0 active idle 3/lxd/0 10.10.20.129 80/tcp Unit is ready
  hacluster-ceph-radosgw/1 active idle 10.10.20.129 Unit is ready and clustered
ceph-radosgw/1 active idle 4/lxd/0 10.10.20.131 80/tcp Unit is ready
  hacluster-ceph-radosgw/2 active idle 10.10.20.131 Unit is ready and clustered
ceph-radosgw/2* active idle 5/lxd/0 10.10.20.112 80/tcp Unit is ready
  hacluster-ceph-radosgw/0* active idle 10.10.20.112 Unit is ready and clustered
cinder/0 active idle 3/lxd/1 10.10.20.125 8776/tcp Unit is ready
  cinder-ceph/1 active idle 10.10.20.125 Unit is ready
  cinder-mysql-router/1 active idle 10.10.20.125 Unit is ready
  hacluster-cinder/1 active idle 10.10.20.125 Unit is ready and clustered
cinder/1 active idle 4/lxd/1 10.10.20.128 8776/tcp Unit is ready
  cinder-ceph/2 active idle 10.10.20.128 Unit is ready
  cinder-mysql-router/2 active idle 10.10.20.128 Unit is ready
  hacluster-cinder/2 active idle 10.10.20.128 Unit is ready and clustered
cinder/2* active idle 5/lxd/1 10.10.20.107 8776/tcp Unit is ready
  cinder-ceph/0* active idle 10.10.20.107 Unit is ready
  cinder-mysql-router/0* active idle 10.10.20.107 Unit is ready
  hacluster-cinder/0* active idle 10.10.20.107 Unit is ready and clustered
glance/0 active idle 3/lxd/2 10.10.20.127 9292/tcp Unit is ready
  glance-mysql-router/1 active idle 10.10.20.127 Unit is ready
  hacluster-glance/1 active idle 10.10.20.127 Unit is ready and clustered
glance/1 active idle 4/lxd/2 10.10.20.138 9292/tcp Unit is ready
  glance-mysql-router/2 active idle 10.10.20.138 Unit is ready
  hacluster-glance/2 active idle 10.10.20.138 Unit is ready and clustered
glance/2* active idle 5/lxd/2 10.10.20.110 9292/tcp Unit is ready
  glance-mysql-router/0* active idle 10.10.20.110 Unit is ready
  hacluster-glance/0* active idle 10.10.20.110 Unit is ready and clustered
keystone/0 active idle 3/lxd/3 10.10.20.136 5000/tcp Unit is ready
  hacluster-keystone/1 active idle 10.10.20.136 Unit is ready and clustered
  keystone-mysql-router/1 active idle 10.10.20.136 Unit is ready
keystone/1 active idle 4/lxd/3 10.10.20.124 5000/tcp Unit is ready
  hacluster-keystone/2 active idle 10.10.20.124 Unit is ready and clustered
  keystone-mysql-router/2 active idle 10.10.20.124 Unit is ready
keystone/2* active idle 5/lxd/3 10.10.20.116 5000/tcp Unit is ready
  hacluster-keystone/0* active idle 10.10.20.116 Unit is ready and clustered
  keystone-mysql-router/0* active idle 10.10.20.116 Unit is ready
memcached/0* active idle 3/lxd/4 10.10.20.117 11211/tcp Unit is ready and clustered
memcached/1 active idle 4/lxd/4 10.10.20.133 11211/tcp Unit is ready and clustered
memcached/2 active idle 5/lxd/4 10.10.20.114 11211/tcp Unit is ready and clustered
mysql-innodb-cluster/0 active idle 3/lxd/5 10.10.20.137 Unit is ready: Mode: R/O
mysql-innodb-cluster/1 active idle 4/lxd/5 10.10.20.121 Unit is ready: Mode: R/O
mysql-innodb-cluster/2* active idle 5/lxd/5 10.10.20.111 Unit is ready: Mode: R/W
neutron-api/3 active idle 3/lxd/15 10.10.20.120 9696/tcp Unit is ready
  hacluster-neutron/5 active idle 10.10.20.120 Unit is ready and clustered
  neutron-api-plugin-ovn/5 active idle 10.10.20.120 Unit is ready
  neutron-mysql-router/5 active idle 10.10.20.120 Unit is ready
neutron-api/4 active idle 4/lxd/14 10.10.20.113 9696/tcp Unit is ready
  hacluster-neutron/4 active idle 10.10.20.113 Unit is ready and clustered
  neutron-api-plugin-ovn/4 active idle 10.10.20.113 Unit is ready
  neutron-mysql-router/4 active idle 10.10.20.113 Unit is ready
neutron-api/5* active idle 5/lxd/14 10.10.20.132 9696/tcp Unit is ready
  hacluster-neutron/3* active idle 10.10.20.132 Unit is ready and clustered
  neutron-api-plugin-ovn/3* active idle 10.10.20.132 Unit is ready
  neutron-mysql-router/3* active idle 10.10.20.132 Unit is ready
nova-cloud-controller/0 active idle 3/lxd/7 10.10.20.123 8774/tcp,8775/tcp Unit is ready
  hacluster-nova/1 active idle 10.10.20.123 Unit is ready and clustered
  nova-mysql-router/1 active idle 10.10.20.123 Unit is ready
nova-cloud-controller/1 active idle 4/lxd/7 10.10.20.118 8774/tcp,8775/tcp Unit is ready
  hacluster-nova/2 active idle 10.10.20.118 Unit is ready and clustered
  nova-mysql-router/2 active idle 10.10.20.118 Unit is ready
nova-cloud-controller/2* active idle 5/lxd/7 10.10.20.108 8774/tcp,8775/tcp Unit is ready
  hacluster-nova/0* active idle 10.10.20.108 Unit is ready and clustered
  nova-mysql-router/0* active idle 10.10.20.108 Unit is ready
nova-compute/0* active idle 6 10.10.20.103 Unit is ready
  ovn-chassis/3* active idle 10.10.20.103 Unit is ready
nova-compute/2 active idle 8 10.10.20.101 Unit is ready
  ovn-chassis/5 active idle 10.10.20.101 Unit is ready
openstack-dashboard/3 active idle 3/lxd/14 10.10.20.134 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/3* active idle 10.10.20.134 Unit is ready
  hacluster-horizon/3* active idle 10.10.20.134 Unit is ready and clustered
openstack-dashboard/4 active idle 4/lxd/13 10.10.20.135 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/5 active idle 10.10.20.135 Unit is ready
  hacluster-horizon/5 active idle 10.10.20.135 Unit is ready and clustered
openstack-dashboard/5* active idle 5/lxd/13 10.10.20.115 80/tcp,443/tcp Unit is ready
  dashboard-mysql-router/4 active idle 10.10.20.115 Unit is ready
  hacluster-horizon/4 active idle 10.10.20.115 Unit is ready and clustered
ovn-central/3 active idle 3/lxd/13 10.10.20.141 6641/tcp,6642/tcp Unit is ready (northd: active)
ovn-central/4 active idle 4/lxd/12 10.10.20.142 6641/tcp,6642/tcp Unit is ready
ovn-central/5* active idle 5/lxd/12 10.10.20.140 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db)
placement/0 active idle 3/lxd/10 10.10.20.139 8778/tcp Unit is ready
  hacluster-placement/1 active idle 10.10.20.139 Unit is ready and clustered
  placement-mysql-router/1 active idle 10.10.20.139 Unit is ready
placement/1 active idle 4/lxd/10 10.10.20.126 8778/tcp Unit is ready
  hacluster-placement/2 active idle 10.10.20.126 Unit is ready and clustered
  placement-mysql-router/2 active idle 10.10.20.126 Unit is ready
placement/2* active idle 5/lxd/10 10.10.20.109 8778/tcp Unit is ready
  hacluster-placement/0* active idle 10.10.20.109 Unit is ready and clustered
  placement-mysql-router/0* active idle 10.10.20.109 Unit is ready
rabbitmq-server/0* active idle 3/lxd/11 10.10.20.122 5672/tcp,15672/tcp Unit is ready and clustered
rabbitmq-server/1 active idle 4/lxd/11 10.10.20.119 5672/tcp,15672/tcp Unit is ready and clustered
rabbitmq-server/2 active idle 5/lxd/11 10.10.20.106 5672/tcp,15672/tcp Unit is ready and clustered
vault/0* active idle 3/lxd/12 10.10.20.130 8200/tcp Unit is ready (active: true, mlock: disabled)
  vault-mysql-router/0* active idle 10.10.20.130 Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 10.10.70.101 lab-02 focal az-01 Deployed
0/lxd/0 started 10.10.70.115 juju-b7ff43-0-lxd-0 focal az-01 Container started
1 started 10.10.70.102 lab-05 focal az-02 Deployed
1/lxd/0 started 10.10.70.113 juju-b7ff43-1-lxd-0 focal az-02 Container started
2 started 10.10.70.103 lab-06 focal az-03 Deployed
2/lxd/0 started 10.10.70.112 juju-b7ff43-2-lxd-0 focal az-03 Container started
3 started 10.10.20.105 lab-03 focal az-01 Deployed
3/lxd/0 started 10.10.20.129 juju-b7ff43-3-lxd-0 focal az-01 Container started
3/lxd/1 started 10.10.20.125 juju-b7ff43-3-lxd-1 focal az-01 Container started
3/lxd/2 started 10.10.20.127 juju-b7ff43-3-lxd-2 focal az-01 Container started
3/lxd/3 started 10.10.20.136 juju-b7ff43-3-lxd-3 focal az-01 Container started
3/lxd/4 started 10.10.20.117 juju-b7ff43-3-lxd-4 focal az-01 Container started
3/lxd/5 started 10.10.20.137 juju-b7ff43-3-lxd-5 focal az-01 Container started
3/lxd/7 started 10.10.20.123 juju-b7ff43-3-lxd-7 focal az-01 Container started
3/lxd/10 started 10.10.20.139 juju-b7ff43-3-lxd-10 focal az-01 Container started
3/lxd/11 started 10.10.20.122 juju-b7ff43-3-lxd-11 focal az-01 Container started
3/lxd/12 started 10.10.20.130 juju-b7ff43-3-lxd-12 focal az-01 Container started
3/lxd/13 started 10.10.20.141 juju-b7ff43-3-lxd-13 focal az-01 Container started
3/lxd/14 started 10.10.20.134 juju-b7ff43-3-lxd-14 focal az-01 Container started
3/lxd/15 started 10.10.20.120 juju-b7ff43-3-lxd-15 focal az-01 Container started
4 started 10.10.20.102 lab-04 focal az-02 Deployed
4/lxd/0 started 10.10.20.131 juju-b7ff43-4-lxd-0 focal az-02 Container started
4/lxd/1 started 10.10.20.128 juju-b7ff43-4-lxd-1 focal az-02 Container started
4/lxd/2 started 10.10.20.138 juju-b7ff43-4-lxd-2 focal az-02 Container started
4/lxd/3 started 10.10.20.124 juju-b7ff43-4-lxd-3 focal az-02 Container started
4/lxd/4 started 10.10.20.133 juju-b7ff43-4-lxd-4 focal az-02 Container started
4/lxd/5 started 10.10.20.121 juju-b7ff43-4-lxd-5 focal az-02 Container started
4/lxd/7 started 10.10.20.118 juju-b7ff43-4-lxd-7 focal az-02 Container started
4/lxd/10 started 10.10.20.126 juju-b7ff43-4-lxd-10 focal az-02 Container started
4/lxd/11 started 10.10.20.119 juju-b7ff43-4-lxd-11 focal az-02 Container started
4/lxd/12 started 10.10.20.142 juju-b7ff43-4-lxd-12 focal az-02 Container started
4/lxd/13 started 10.10.20.135 juju-b7ff43-4-lxd-13 focal az-02 Container started
4/lxd/14 started 10.10.20.113 juju-b7ff43-4-lxd-14 focal az-02 Container started
5 started 10.10.20.104 lab-07 focal az-03 Deployed
5/lxd/0 started 10.10.20.112 juju-b7ff43-5-lxd-0 focal az-03 Container started
5/lxd/1 started 10.10.20.107 juju-b7ff43-5-lxd-1 focal az-03 Container started
5/lxd/2 started 10.10.20.110 juju-b7ff43-5-lxd-2 focal az-03 Container started
5/lxd/3 started 10.10.20.116 juju-b7ff43-5-lxd-3 focal az-03 Container started
5/lxd/4 started 10.10.20.114 juju-b7ff43-5-lxd-4 focal az-03 Container started
5/lxd/5 started 10.10.20.111 juju-b7ff43-5-lxd-5 focal az-03 Container started
5/lxd/7 started 10.10.20.108 juju-b7ff43-5-lxd-7 focal az-03 Container started
5/lxd/10 started 10.10.20.109 juju-b7ff43-5-lxd-10 focal az-03 Container started
5/lxd/11 started 10.10.20.106 juju-b7ff43-5-lxd-11 focal az-03 Container started
5/lxd/12 started 10.10.20.140 juju-b7ff43-5-lxd-12 focal az-03 Container started
5/lxd/13 started 10.10.20.115 juju-b7ff43-5-lxd-13 focal az-03 Container started
5/lxd/14 started 10.10.20.132 juju-b7ff43-5-lxd-14 focal az-03 Container started
6 started 10.10.20.103 lab-01 focal az-01 Deployed
8 started 10.10.20.101 lab-08 focal az-02 Deployed

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :
Download full text (6.0 KiB)

I was able to reproduce it although:

* JUJU_AVAILABILITY_ZONE is set correctly;
* the charm correctly renders the default_availability_zone config in the DEFAULT section of nova.conf.

So it does what it is supposed to do but the intended result is not achieved. We only check this functionality via a unit test that's why we haven't caught it in functional testing.

Based on the Nova code, unless the host aggregate overrides the AZ value, it is taken from nova.conf's default_availability_zone:
(stein)
https://opendev.org/openstack/nova/src/commit/e91540f96538c7ba568617db7c03cf3a195b87ff/nova/availability_zones.py#L93-L100
https://opendev.org/openstack/nova/src/commit/e91540f96538c7ba568617db7c03cf3a195b87ff/nova/availability_zones.py#L71-L109
https://opendev.org/openstack/nova/src/commit/e91540f96538c7ba568617db7c03cf3a195b87ff/nova/conf/availability_zone.py#L32-L34

juju config keystone openstack-origin
cloud:bionic-stein

openstack compute service list
+-----+------------------+----------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+-----+------------------+----------------------+----------+---------+-------+----------------------------+
| 1 | nova-scheduler | juju-554d51-27-lxd-6 | internal | enabled | up | 2020-07-24T10:50:55.000000 |
| 7 | nova-conductor | juju-554d51-27-lxd-6 | internal | enabled | up | 2020-07-24T10:50:55.000000 |
| 19 | nova-consoleauth | juju-554d51-27-lxd-6 | internal | enabled | up | 2020-07-24T10:50:57.000000 |
| 22 | nova-compute | server-maas-6.maas | nova | enabled | up | 2020-07-24T10:50:58.000000 |
| 25 | nova-compute | server-maas-11.maas | nova | enabled | up | 2020-07-24T10:50:59.000000 |
| 28 | nova-compute | server-maas-2.maas | nova | enabled | up | 2020-07-24T10:50:55.000000 |
| 31 | nova-compute | server-maas-8.maas | nova | enabled | up | 2020-07-24T10:50:53.000000 |
| 64 | nova-compute | server-maas-3.maas | nova | enabled | up | 2020-07-24T10:50:56.000000 |
| 67 | nova-compute | server-maas-10.maas | nova | enabled | up | 2020-07-24T10:50:54.000000 |
| 70 | nova-compute | server-maas-4.maas | nova | enabled | up | 2020-07-24T10:50:59.000000 |
| 73 | nova-consoleauth | juju-554d51-24-lxd-6 | internal | enabled | up | 2020-07-24T10:50:56.000000 |
| 76 | nova-conductor | juju-554d51-24-lxd-6 | internal | enabled | up | 2020-07-24T10:50:56.000000 |
| 82 | nova-scheduler | juju-554d51-24-lxd-6 | internal | enabled | up | 2020-07-24T10:50:57.000000 |
| 91 | nova-scheduler | juju-554d51-31-lxd-6 | internal | enabled | up | 2020-07-24T10:50:57.000000 |
| 100 | nova-consoleauth | juju-554d51-31-lxd-6 | internal | enabled | up | 2020-07-24T10:50:56.000000 |
| 103 | nova-conductor | juju-554d51-31-lxd-6 | internal | enabled | up | 2020-07-24T10:50:57.000000 |
+-----+------------------+----------------------+----------+---------+-------+----------------------------+

openstack aggregate list ; echo $?

0

juju config nova-compute-kvm ...

Read more...

Changed in charm-nova-compute:
importance: Undecided → High
status: New → Triaged
Revision history for this message
Nobuto Murata (nobuto) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.