Overlapping availability zones don't display properly

[Vish Ishaya]

If you create two overlapping availability zones, the get availability zones command returns odd information.

Repro:

$ nova --os-username admin --os-tenant-name admin --os-password secrete aggregate-create foo foo
+----+------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+-------+----------+
| 2 | foo | foo | | |
+----+------+-------------------+-------+----------+

$ nova --os-username admin --os-tenant-name admin --os-password secrete aggregate-add-host 2 node001-cont001
Aggregate 2 has been successfully updated.
+----+------+-------------------+----------------------+--------------------------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+----------------------+--------------------------------+
| 2 | foo | foo | [u'node001-cont001'] | {u'availability_zone': u'foo'} |
+----+------+-------------------+----------------------+--------------------------------+

$ nova availability-zone-list
+-------------+-----------+
| Name | Status |
+-------------+-----------+
| zone001 | available |
| zone001,foo | available |
+-------------+-----------+

Expected:
+---------+-----------+
| Name | Status |
+---------+-----------+
| zone001 | available |
| foo | available |
+---------+-----------+

The admin view is a little more useful:

$ nova --os-username admin --os-tenant-name admin --os-password secrete availability-zone-list
+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
...
| zone001 | available |
| |- node005-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:31.000000 |
| |- node007-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
| |- node009-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:25.000000 |
| |- node003-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
| zone001,foo | available |
| |- node001-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
+-----------------------+----------------------------------------+

but it could easily show two copies of the node that is in two zones:

+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
...
| zone001 | available |
| |- node005-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:31.000000 |
| |- node007-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
| |- node009-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:25.000000 |
| |- node003-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
| |- node001-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
| foo | available |
| |- node001-cont001 | |
| | |- nova-compute | enabled :-) 2014-02-06T19:18:27.000000 |
+-----------------------+----------------------------------------+

[Vish Ishaya (vishvananda)]

This line should be modified to return the zones as a list instead of joining them using ',':

https://github.com/openstack/nova/blob/e9788089173f03f9a25e69b61346e56af045eccf/nova/availability_zones.py#L73

The call sites of set_availability_zones should be converted to do the comma join if necessary, and to simply deal with processing a list in other cases.

For example, to fix the issue above:

https://github.com/openstack/nova/blob/e9788089173f03f9a25e69b61346e56af045eccf/nova/availability_zones.py#L118

and

https://github.com/openstack/nova/blob/e9788089173f03f9a25e69b61346e56af045eccf/nova/availability_zones.py#L129

can be modified to have another iteration through the zones when creating the list

for zone_list in zones:
--> for zone in zone_list:
-------> if zone not in not_available_zones:

[Sangeeta Singh (singhs)]

due to this issue the boot command fails in create instance during the validation of availability zone.

since instead of returning a list of all available zones we get a list like

[u'ZoneA,nova', u'ZoneB,nova', 'internal']"

and the boot zone is nova by default which does not match so the validation fails.

As suggested above a function returning a list is what is needed.

[Joe Breu (breu)]

This is affecting us w/ havana also. Can we get a backport when fixed in trunk?

[Sylvain Bauza (sylvain-bauza)]

I can't reproduce the bug on my devstack... When specifying an az in nova.conf for compute host, I can see it :

bauzas@climate-devstack-ctrl:~$ nova availability-zone-list
+--------------------------+----------------------------------------+
| Name | Status |
+--------------------------+----------------------------------------+
| internal | available |
(...)
| nova | available |
| |- climate-devstack-ctrl | |
| | |- nova-compute | enabled :-) 2014-04-03T12:00:30.000000 |
+--------------------------+----------------------------------------+

I then created an aggregate with AZ both set to foo :
bauzas@climate-devstack-ctrl:~$ nova aggregate-list
+----+----------+-------------------+
| Id | Name | Availability Zone |
+----+----------+-------------------+
| 10 | foo | foo |
| 12 | nova | nova |
+----+----------+-------------------+

But when adding the host in foo, it overrides the default AZ :
bauzas@climate-devstack-ctrl:~$ nova aggregate-add-host foo climate-devstack-ctrl
Aggregate 10 has been successfully updated.
+----+------+-------------------+-------------------------+-------------------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+-------------------------+-------------------------+
| 10 | foo | foo | 'climate-devstack-ctrl' | 'availability_zone=foo' |
+----+------+-------------------+-------------------------+-------------------------+

bauzas@climate-devstack-ctrl:~$ nova availability-zone-list
+--------------------------+----------------------------------------+
| Name | Status |
+--------------------------+----------------------------------------+
| internal | available |
(...)
| foo | available |
| |- climate-devstack-ctrl | |
| | |- nova-compute | enabled :-) 2014-04-03T12:08:30.000000 |
+--------------------------+----------------------------------------+

Besides this, Nova API prevents adding the same host in a second AZ :

bauzas@climate-devstack-ctrl:~$ nova aggregate-list
+----+----------+-------------------+
| Id | Name | Availability Zone |
+----+----------+-------------------+
| 10 | foo | foo |
| 11 | foo2 | foo2 |
| 12 | nova | nova |
+----+----------+-------------------+

bauzas@climate-devstack-ctrl:~$ nova aggregate-add-host foo2 climate-devstack-ctrl
ERROR: Cannot perform action 'add_host_to_aggregate' on aggregate 11. Reason: Host already in availability zone foo. (HTTP 409) (Request-ID: req-120e0b7a-8005-4137-97d3-952375ed6f5c)

This is due to https://github.com/openstack/nova/blob/9d45e9cef624a4a972c24c47c7abd57a72d74432/nova/compute/api.py#L3378

Could someone share his config with me to know how to add the same host in two AZs ?

[Daniele Venzano (venza)]

I can add an host to two availability zones by creating an aggregate without an availability zone, adding an host to it and then using aggregate-update to set the availability zone.

Probably it is a bug, but it works.

[Sylvain Bauza (sylvain-bauza)]

Thanks Daniele for the comment.

I'll open a discussion on openstack-dev because I need clarifications. If Nova API is preventing to have 2 AZs for the same host, we should fix the possibility to update an aggregate with setting an AZ if the hosts mapped to this AZ are already part of another AZ.

In other words, this bug should not be fixed, but rather the missing coverage when updating an aggregate.
Anyway, I can't reproduce the bug as reported.

Comments welcome.

[Sylvain Bauza (sylvain-bauza)]

http://lists.openstack.org/pipermail/openstack-dev/2014-April/031775.html for discussing about what to do

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/85961

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/88283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/85961
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0ad5a64dc9ac4b1cfb8038f9171b6385fdf07f28
Submitter: Jenkins
Branch: master

commit 0ad5a64dc9ac4b1cfb8038f9171b6385fdf07f28
Author: Sylvain Bauza <email address hidden>
Date: Tue Apr 8 11:00:40 2014 +0200

    Fix AvailabilityZone check for hosts in multiple aggregates

    Check for multiple AZ was distinct when adding an host to an aggregate
    and when updating an aggregate (or its metadata), which was leading to
    possible corner-cases where logics were different.

    This patch proposes a single way for checking for both API methods and
    also adds an extra control on the possibility to have an aggregate with
    its metadata AZ set to the configuration default value (eg. 'nova') and
    where it was possible to subsequently have one host in two disctinct AZs
    Fixed that scenerio as it is a gap, Nova as of now preventing to have
    one host in two distinct AZs.

    Change-Id: Ida6aa507f34f36a436adaafe1f7609963c59a9bc
    Partial-Bug: #1277230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/88283
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a2af1101fe94744ab87f30016def333405713287
Submitter: Jenkins
Branch: master

commit a2af1101fe94744ab87f30016def333405713287
Author: Sylvain Bauza <email address hidden>
Date: Thu Apr 17 14:37:00 2014 +0200

    Improve performance for checking hosts AZs

    When updating aggregate metadata or when adding a host into aggregate,
    a check is done to make sure that hosts can't be in multiple AZs.

    In order to do the check, the routine was looping over each host to look
    in the DB which AZs the host is part of.
    As it's quite hungry in terms of DB accesses, here the proposal is to
    extend the helper method for getting all AZs by also having their hosts,
    and so only do memory iterations.

    Partial-Bug: #1277230

    Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb

[Tracy Jones (tjones-i)]

Is this bug closed now? All reviews are merged though they are marked partial. what else is needed?

[Sylvain Bauza (sylvain-bauza)]

Well, sort of ;-) These patches are fixing the flaw where it was possible to add a host into 2 AZs, so now it's no longer possible to have the reporter's scenario.