different availability zone for nova and cinder when AZ is not explicitly given

[Jiri Suchomel]

When booting an instance from image, which should create new volume, when AZ zone is not specified, instance could end in different AZ than the image.

That doesn't hurt with cross_az_attach=true, but if this is set to False, creating the volume will fail with

" Instance %(instance)s and volume %(vol)s are not in the same availability_zone...." error.

Nova actually decides at some point, which AZ it should use (when none was specified), so I think we just need to move this decision before the point when the volume is created, so nova can pass correct AZ value to cinder API.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225119

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/226977

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226977
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8d2f02f3014e8d7e0ef9a61723f5a10f050cc81
Submitter: Jenkins
Branch: master

commit d8d2f02f3014e8d7e0ef9a61723f5a10f050cc81
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 23 13:21:06 2015 -0700

    Deprecate cinder.cross_az_attach option

    Introduced in 1504cbc5d4a27695fa663f0b0f3f7b48745bdb45 in Grizzly, the
    use cases around this option are unclear but there have been several
    bugs related to it.

    For example, 6060888b58db42eea826939852838f9e1c204d2c changed boot from
    volume to pass the server instance availability zone to cinder when
    creating the volume. However, if that same AZ does not exist in Cinder,
    the volume create request will fail - which is bug 1496235.

    Cinder works around this with b85d2812a8256ff82934d150dbc4909e041d8b31
    using a config option to allow falling back to a default Cinder AZ if
    the one requested does not exist. By default this fallback is disabled
    though.

    It also sounds like availability zones in Cinder were an artifact from
    when Cinder was split out from the nova-volume service, but Cinder AZ
    use cases are also unclear. And it's also unclear what the relationship
    is between AZs in Nova and Cinder.

    So given the problems here and the lack of a clear use case to justify
    having this configuration option in tree, along with it's added
    complexity, deprecate the option for removal.

    Related-Bug: #1496235
    Related-Bug: #1380780
    Related-Bug: #1489575
    Related-Bug: #1497253

    Change-Id: I52cd3d8867d3b35f5caba377302bfc52c112f1d6

[Jiri Suchomel (jsuchome)]

I should have been more clear in the bug report, so here's another attempt:

1. User has cinder AZ's and nova AZ of the same names

2. AZ's physical location is different, and user wants instances to have same AZ as their volumes

3. This is generally achieved by setting cross_az_attach option to False, because since
https://review.openstack.org/#/c/157041/ volumes are created in the same AZ as instances.

4. However, what if user doesn't explicitly provide AZ when creating the instance (so the scheduler can distribute the load evenly and according to available resources)?
This is the situation possibly requiring a fix. In such case, nova uses the None value for availability zone at the time it calls volume_api.create. Cinder creates the volume in some AZ it has available, and when nova finishes creating the instance it creates it in some of its available AZ. There's no relation between these two, so if they end up being different, we'll hit the error " Instance %(instance)s and volume %(vol)s are not in the same availability_zone...."

So, my proposal (as expressed in https://review.openstack.org/225119) is that:

- if cross_az_attach is set to false, nova should ensure cinder and nova AZ's are matching, AND it should make sure this rule is true also in the case when AZ was not specified by user. Thus I propose to look for instance's real AZ BEFORE actually trying to create the volume, and use this value also for volume.

[Marcin Zbik (zbikmarc+launchpad)]

I confirm this as of https://bugs.launchpad.net/heat/+bug/1580582

If there is no computes in nova AZ, booting from volume will fail if AZ is not explicitly set - it will try to create volume in nova AZ.

[Jiri Suchomel (jsuchome)]

Marcin, does the proposed solution for this (https://review.openstack.org/225119) solve your problem?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/225119
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[Sean Dague (sdague)]

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/366724 in branch: master

[Matt Riedemann (mriedem)]

Related mailing list thread about this:

http://lists.openstack.org/pipermail/openstack-operators/2017-June/013676.html

[Matt Riedemann (mriedem)]

Is this still a problem after change https://review.openstack.org/#/c/446053/ in Pike?

[Matt Riedemann (mriedem)]

I've confirmed this with a devstack setup that this was fixed indirectly in pike with change https://review.openstack.org/#/c/446053/.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/366724