nova aggregate-add-host should limit user to set different availability zone for a host

[David Jia]

user can use "nova aggregate-add-host" to set different availability zone for a host, then the host will be available for multi availability zone. When user try to boot instance in the host using second or third availability zone, the instance will still be assigned to the first availability zone which set at first time the host added to a host aggregate of the availability zone. It works as design, but it confuse the user, it make user to think that a host can be assigned to multi availability zone. So I think we should add checking to limit user to add host to host aggregates in different availability zone.

Here are steps to recreate the issue:

1)   create one AZ
nova aggregate-create test-az test-az
the AZ test-az id is 1

2)   add host to AZ
nova aggregate-add-host 1 WIN-UJAK7KN9LRA

3)  boot one instance
 nova boot --flavor 2 --image  26c6ddd1-3db7-440e-930c-f3dbc146c7f2   --availability-zone  test-az test-vm-1

In this steps, it prints | OS-EXT-AZ:availability_zone         | nova                                 |

4) check test-vm-1

nova show d752aa55-b2dd-4966-a770-4ab2972c6dd7

OS-EXT-AZ:availability_zone         | test-az                                                  |

5)
then create another AZ
             nova aggregate-create test-az-2 test-az-2

Add host to AZ
             nova aggregate-add-host 2 WIN-UJAK7KN9LRA

check AZs
            nova  availability-zone-list
                     | test-az-2,test-az     | available                              |
                             | |- WIN-UJAK7KN9LRA    |                                        |
                             |  | |- nova-compute     | enabled :-) 2013-06-26T03:16:47.764046 |

boot one instance
           nova boot --flavor 2 --image  26c6ddd1-3db7-440e-930c-f3dbc146c7f2   --availability-zone  test-az-2 test-vm-4

    still the print is | OS-EXT-AZ:availability_zone         | nova                                 |

check instance info
            | OS-EXT-AZ:availability_zone         | test-az                                                  |

 The new instance is in test-az, the passed option test-az-2 seems not work. We may should add checking logic to limit users to do operations like " nova aggregate-add-host 2 WIN-UJAK7KN9LRA" in step #5

[Haomeng,Wang (whaom)]

This /opt/stack/nova/nova/availability_zones.py get_host_availability_zone method returns the first zone if there are multiple zones for one single host for this:

def get_host_availability_zone(context, host, conductor_api=None):
    if conductor_api:
        metadata = conductor_api.aggregate_metadata_get_by_host(
            context, host, key='availability_zone')
    else:
        metadata = db.aggregate_metadata_get_by_host(
            context, host, key='availability_zone')
    if 'availability_zone' in metadata:
        return list(metadata['availability_zone'])[0] -- return first one
    else:
        return CONF.default_availability_zone

But our "nova aggregate-add-host" command supports to add one same host into multi-zones, this should be conflict between the 'add host to zone' and 'get host zone' two actions.

[David Jia (jiataotj)]

for fixing the issue, we can limit users to add more availability zone to a host when the host belongs to an availability zone, and allow users to add the host to different host aggregate of same availability zone

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/36651

[David Jia (jiataotj)]

Bug for adding limitation in aggregate metadata updating:https://bugs.launchpad.net/nova/+bug/1200479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/36651
Committed: http://github.com/openstack/nova/commit/40986bc17c5fccac74df5edad13bb70ef16f7987
Submitter: Jenkins
Branch: master

commit 40986bc17c5fccac74df5edad13bb70ef16f7987
Author: jiataotj <email address hidden>
Date: Thu Jul 11 21:04:41 2013 +0800

    Fix multi availability zone issue part 1

    Current aggregate APIs allow user to add multi
    AZs to a host, but Openstack support single AZ
    of a host. When booting an instance, nova uses
    the first AZ assigned to a host,even if user
    assigned multi AZs to the host and user chose
    the second AZ to boot an instance. So multi AZs
    aren't supported by current Openstack design,
    we should fix aggregate APIs to align the design
    . This patch is for limit user to add multi AZs
    using aggregate-add-host API.

    The part 2 limits user to add multi AZs using aggregate
    update and aggregate metadata update API, here is the
    link:
    I18ad1ac22c6aee298ccc75c978b85241ae1ac616

    Fixes: bug #1196893

    Change-Id: I788782a9f21ec2672551f75123753175bb268586

[Meghal Gosalia (meghalgosalia)]

so, what it means is --

if I have hostX in zoneA and zoneB and user passes zoneB to boot a vm Y,
if user runs nova show vmY, it will show that vm is in zoneA even though he had passed zoneB.

So, should we not focus on solving the problem and showing correct zone to the user,
rather than avoiding addition of hostX in zoneA and zoneB.

Because there could be scenarios where hostX needs to be part of two zones.
Lets say that I have zones based on PDUs (power distribution) - zoneA and zoneB.
I want hostX to be part of zoneA, host Y to be part of zoneB.

But, I would also want hostX and hostY to be part of default scheduling zone called default_zone for example.
So, that when user does not pass any zone, vm placement is evenly distributed across two zones A and B.

[David Jia (jiataotj)]

Meghal,
Here is some misunderstanding in your description about the issue

“if I have hostX in zoneA and zoneB and user passes zoneB to boot a vm Y,
if user runs nova show vmY, it will show that vm is in zoneA even though he had passed zoneB.”

The issue isn't how nova show vmY's zone, the issue is which zone the nova actually chose to boot your vmY, without the fixing, although you pass zoneB to boot vmY, Openstack always use zoneA to boot vmY

[Meghal Gosalia (meghalgosalia)]

If you look at https://github.com/openstack/nova/blob/2013.2.2/nova/scheduler/filters/availability_zone_filter.py#L46

and relate it to my example in #7,
the scheduler is making this check for hostX,
metadata['availability_zone'] will have ['zoneA', 'zoneB']
and zone passed by user during boot is zoneB which will make the check true.
And hence, hostX will be selected.

That is what I see in my environment as well.
Only problem is if you do a nova show vmY, which calls extension "OS-EXT-AZ:availability_zone" which calls the function for hostX - https://github.com/openstack/nova/blob/2013.2.2/nova/availability_zones.py#L93
Because of that function call, nova show vmY might show zoneA based on which element it picks up as the first one from the list.

So, scheduling works fine, but issue is with nova show since that function referenced in #1 (get_host_availability_zones) is used only during nova show and not by availability zone scheduler.