host aggregates affecting on_maintenance status of hosts in a failover segment

[Syed Mohammad Adnan Karim]

The following was observed in an openstack stein cloud with 37 compute hosts:

Initially there was a single host aggregate called nova that contained all the compute hosts.
Then a failover segment called segment1 with an AUTO recovery method was created.
All 37 hosts were added to segment1 and all the hosts had on_maintenance=False.
Then 3 host aggregates corresponding to availability zones (AZ1, AZ2, AZ3) were created and the 37 hosts were distributed across the host aggregates as evenly as possible (12, 13, 12).
Then all hosts in segment1 had on_maintenance=True.

This becomes a problem because then I started running into this bug:
https://bugs.launchpad.net/charm-masakari/+bug/1884290

[Liam Young (gnuoy)]

I have tried to reproduce this on a bionic stein deploy but couldn't. Can you take a look at the steps I went through and see of they roughly match what you were doing when you hit the issue ?

Output below repeated in pastebin (https://paste.ubuntu.com/p/hBkzxy8cT4/):

$ openstack hypervisor list
+----+------------------------------------------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+------------------------------------------------------+-----------------+-------------+-------+
| 1 | juju-d21cc8-zaza-314b3c785bc4-17.project.serverstack | QEMU | 172.20.0.24 | up |
| 2 | juju-d21cc8-zaza-314b3c785bc4-21.project.serverstack | QEMU | 172.20.0.29 | up |
| 3 | juju-d21cc8-zaza-314b3c785bc4-19.project.serverstack | QEMU | 172.20.0.12 | up |
| 4 | juju-d21cc8-zaza-314b3c785bc4-22.project.serverstack | QEMU | 172.20.0.30 | up |
| 5 | juju-d21cc8-zaza-314b3c785bc4-23.project.serverstack | QEMU | 172.20.0.58 | up |
| 6 | juju-d21cc8-zaza-314b3c785bc4-16.project.serverstack | QEMU | 172.20.0.23 | up |
| 7 | juju-d21cc8-zaza-314b3c785bc4-24.project.serverstack | QEMU | 172.20.0.20 | up |
| 8 | juju-d21cc8-zaza-314b3c785bc4-20.project.serverstack | QEMU | 172.20.0.47 | up |
| 9 | juju-d21cc8-zaza-314b3c785bc4-18.project.serverstack | QEMU | 172.20.0.15 | up |
+----+------------------------------------------------------+-----------------+-------------+-------+
$ openstack segment list
+--------------------------------------+------+-------------+--------------+-----------------+
| uuid | name | description | service_type | recovery_method |
+--------------------------------------+------+-------------+--------------+-----------------+
| b20022ad-3e09-474...

[Syed Mohammad Adnan Karim (karimsye)]

The above steps match what was done on the cloud when I hit the issue.