Boot from volume faild with availability_zone option, in case of cinder do not have availability_zone

[Ikuo Kumagai]

My envirement ,Nova has 2 availability_zone and Cinder has no availability_zone.
When I run two command "cinder create" and "nova boot" separately , it is finished normally.
But run at one time, and an error occurs as below.(It is the same on the dashboard)

----
$ nova boot --flavor m1.small --block-device source=image,id=4014a3f7-507b-4692-86c8-8224bbcc7102,dest=volume,size=10,shutdown=delete,bootindex=0 --nic net-id=4d9e9847-80b5-46ca-8439-344930a59825 --availability_zone az2 test1

$ nova list
+--------------------------------------+-------+--------+----------------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+----------------------+-------------+-----------------------+
| a0dc0f03-155c-422b-b0ac-996fc17e0989 | test1 | ERROR | block_device_mapping | NOSTATE | |
+--------------------------------------+-------+--------+----------------------+-------------+-----------------------+

$ nova show
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | az1 |
| OS-EXT-SRV-ATTR:host | compute011-az1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute011-az1.maas |
| OS-EXT-SRV-ATTR:instance_name | instance-00000514 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-09-16T02:47:04Z |
| fault | {"message": "Build of instance a0dc0f03-155c-422b-b0ac-996fc17e0989 aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2219, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2330, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 85, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2303, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2439, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-09-16T02:47:08Z"} |
| flavor | m1.small (090955df-7dbe-4102-bcfb-4d64d0cc5105) |
| hostId | bf381a5d2d9136fe4669218bcda1467156b71227023610e0311f9e86 |
| id | a0dc0f03-155c-422b-b0ac-996fc17e0989 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | test1 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 3296a2ca68a04a139258927447956d95 |
| updated | 2015-09-16T02:47:09Z |
| user_id | ab2a6867991f4cd785c75935ea80f0fb |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Version : Kilo

[Matt Riedemann (mriedem)]

Is there more stacktrace to that error? I'd expect to see something failing down in the nova.virt.block_device attach code somewhere, or maybe an error in the cinder logs?

[Matt Riedemann (mriedem)]

I assume this is the call that fails, right?

https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L381

            vol = volume_api.create(context, self.volume_size,
                                    '', '', image_id=self.image_id,
                                    availability_zone=av_zone)

And since that az2 doesn't exist in cinder, the volume create request fails. I guess I wouldn't consider this a bug.

We could probably determine from the az list in cinder if the volume we're about to create with the given AZ exists, but if it doesn't, then what do we do? We can either fail (which we're already doing), or not send in the AZ to cinder, but I'm not sure if that's what should happen - although in talking with the cinder devs it sounds like AZs aren't really used in cinder anyway.

[Matt Riedemann (mriedem)]

The alternative here is you create the volume in cinder first, then just provide that as the source when booting from volume on the nova side.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/226977

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226977
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8d2f02f3014e8d7e0ef9a61723f5a10f050cc81
Submitter: Jenkins
Branch: master

commit d8d2f02f3014e8d7e0ef9a61723f5a10f050cc81
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 23 13:21:06 2015 -0700

    Deprecate cinder.cross_az_attach option

    Introduced in 1504cbc5d4a27695fa663f0b0f3f7b48745bdb45 in Grizzly, the
    use cases around this option are unclear but there have been several
    bugs related to it.

    For example, 6060888b58db42eea826939852838f9e1c204d2c changed boot from
    volume to pass the server instance availability zone to cinder when
    creating the volume. However, if that same AZ does not exist in Cinder,
    the volume create request will fail - which is bug 1496235.

    Cinder works around this with b85d2812a8256ff82934d150dbc4909e041d8b31
    using a config option to allow falling back to a default Cinder AZ if
    the one requested does not exist. By default this fallback is disabled
    though.

    It also sounds like availability zones in Cinder were an artifact from
    when Cinder was split out from the nova-volume service, but Cinder AZ
    use cases are also unclear. And it's also unclear what the relationship
    is between AZs in Nova and Cinder.

    So given the problems here and the lack of a clear use case to justify
    having this configuration option in tree, along with it's added
    complexity, deprecate the option for removal.

    Related-Bug: #1496235
    Related-Bug: #1380780
    Related-Bug: #1489575
    Related-Bug: #1497253

    Change-Id: I52cd3d8867d3b35f5caba377302bfc52c112f1d6

[Ikuo Kumagai (i-kumagai)]

Hi Matt.

Thank you for introduce that information.

At first point, I wanted that I can select the sync and async az between nova and cinder .
The case of async , cinder uses default az , in spite of designation of Nova.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/227483

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227564

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227483
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0b737f24bef9d00e266202af94ff402a503b52fb
Submitter: Jenkins
Branch: master

commit 0b737f24bef9d00e266202af94ff402a503b52fb
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 24 12:10:47 2015 -0700

    Add more help text to the cinder.cross_az_attach option

    Try to provide some more detail on how the cross_az_attach option is
    used and what setting it to False implies.

    Related-Bug: #1496235

    Change-Id: I588a058711154b7857375bcf166ac8a74f8af448

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227564
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f9a51b970f688b90baf0ae3ef31d79b3fec02ed1
Submitter: Jenkins
Branch: master

commit f9a51b970f688b90baf0ae3ef31d79b3fec02ed1
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 24 14:47:49 2015 -0700

    Only create volumes with instance.az if cinder.cross_az_attach is False

    Commit 6060888b58db42eea826939852838f9e1c204d2c changed the boot from
    volume flow where source != 'volume' (where Nova creates the volume and
    then attaches it to the instance during the server create operation)
    such that the instance.availability_zone is always passed to the volume
    create API. That was done because if the cinder.cross_az_attach config
    option is False then Nova requires the instance and volume AZs to be the
    same when attaching the volume.

    However, telling Cinder to create a volume based on the instance AZ can
    fail if that AZ doesn't exist in Cinder.

    Cinder commit b85d2812a8256ff82934d150dbc4909e041d8b31 works around this
    by allowing Cinder to be configured to basically not fail if the
    requested AZ does not exist. This is not a great solution though if you
    have cinder.cross_az_attach=False in Nova because Cinder ignored your
    request. It's also terrible UX for the end user because they get a
    NoValidHost when boot from volume fails. There is a TODO in this change
    to address that later (because exactly how cross-service AZs should work
    is still being discussed).

    This change addresses the gap introduced by always telling Cinder to
    create the volume with the instance AZ. We should only do that when
    cinder.cross_az_attach is False, because that's the strict case where
    Nova expects them to match. If cross_az_attach is True, Nova doesn't
    care what AZ the volume is in so we shouldn't try and force
    Cinder to accommodate the Nova AZ when creating the volume.

    Closes-Bug: #1496235

    Change-Id: I3d60c084fd71162a0d4835394f8e819723c1b8a7

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/253082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253082
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ec146e0d75b7c907082227221835dad8291134b6
Submitter: Jenkins
Branch: stable/kilo

commit ec146e0d75b7c907082227221835dad8291134b6
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 24 14:47:49 2015 -0700

    Only create volumes with instance.az if cinder.cross_az_attach is False

    Commit 6060888b58db42eea826939852838f9e1c204d2c changed the boot from
    volume flow where source != 'volume' (where Nova creates the volume and
    then attaches it to the instance during the server create operation)
    such that the instance.availability_zone is always passed to the volume
    create API. That was done because if the cinder.cross_az_attach config
    option is False then Nova requires the instance and volume AZs to be the
    same when attaching the volume.

    However, telling Cinder to create a volume based on the instance AZ can
    fail if that AZ doesn't exist in Cinder.

    Cinder commit b85d2812a8256ff82934d150dbc4909e041d8b31 works around this
    by allowing Cinder to be configured to basically not fail if the
    requested AZ does not exist. This is not a great solution though if you
    have cinder.cross_az_attach=False in Nova because Cinder ignored your
    request. It's also terrible UX for the end user because they get a
    NoValidHost when boot from volume fails. There is a TODO in this change
    to address that later (because exactly how cross-service AZs should work
    is still being discussed).

    This change addresses the gap introduced by always telling Cinder to
    create the volume with the instance AZ. We should only do that when
    cinder.cross_az_attach is False, because that's the strict case where
    Nova expects them to match. If cross_az_attach is True, Nova doesn't
    care what AZ the volume is in so we shouldn't try and force
    Cinder to accommodate the Nova AZ when creating the volume.

    Closes-Bug: #1496235

    Change-Id: I3d60c084fd71162a0d4835394f8e819723c1b8a7
    (cherry picked from commit f9a51b970f688b90baf0ae3ef31d79b3fec02ed1)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 2015.1.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 2015.1.4 release.