API documentation and behavior do not match for booting with attached volumes

[Doug Hellmann]

Description
===========

The documentation for block device mapping in http://docs.openstack.org/developer/nova/block_device_mapping.html#block-device-mapping-v2 indicates that to attach a volume to a new instance without booting from that volume a boot_index value of None can be passed. Shade was doing this, and it works against some clouds (at least DreamHosts's iad2 cloud, "dreamcompute", which is running either mitaka or newton). It does not work against nova at least as of 9ae5b2306b7a7cc9e77c77292256b13926920ead launched with devstack.

Steps to reproduce
==================

1. Run devstack.
2. Download the downpour git repo from https://github.com/dhellmann/downpour
3. Use the tiny.yml playbook in the demo directory to launch an instance (see the README for some setup instructions).

Expected Result
===============

One new instance named downpour-demo-tiny booted ephemeral with a new volume named downpour-demo-tiny attached to it.

Actual Result
=============

Error message: Invalid input for field/attribute boot_index. Value: None. None is not of type 'integer'

Additional Info
===============

IRC logs from 7 Feb 2017 in #openstack-nova

[Tue 05:29:35 PM] <mordred> mriedem: so - I've got this question about boot from volume
[Tue 05:30:51 PM] <mordred> mriedem: http://developer.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server says that block_device_mapping_v2.boot_index takes a string and also that None should be used for volumes not used as boot volumes
[Tue 05:32:00 PM] <mriedem> was just packing up...
[Tue 05:32:20 PM] <mordred> mriedem: but we just got an error "Invalid input for field/attribute boot_index. Value: None. None is not of type 'integer"
[Tue 05:32:24 PM] <mriedem> mordred: http://docs.openstack.org/developer/nova/block_device_mapping.html might be helpful
[Tue 05:32:33 PM] <mordred> mriedem: yup. it also says None is valid
[Tue 05:34:01 PM] <mriedem> i feel like something was just recently changed in the json schema there, is this master?
[Tue 05:34:17 PM] <mordred> dhellmann: you were doing master devstack?
[Tue 05:34:30 PM] <mordred> mriedem: I think so yeah?
[Tue 05:34:43 PM] <dhellmann> mordred : yes, master devstack (from a few hours ago)
[Tue 05:34:45 PM] mordred goes to look at json schema
[Tue 05:34:51 PM] <mriedem> https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/block_device_mapping.py#L56
[Tue 05:35:28 PM] <mordred> fantastic
[Tue 05:36:06 PM] <mriedem> what's the image and dest type?
[Tue 05:36:19 PM] <mriedem> source_type and destination_type i mean
[Tue 05:37:01 PM] <mriedem> because https://github.com/openstack/nova/blob/master/nova/block_device.py#L198
[Tue 05:37:02 PM] <dhellmann> it's a qcow2 ubuntu image
[Tue 05:37:16 PM] <mordred> destination_type would be 'volume'
[Tue 05:37:22 PM] <mriedem> and source_type is?
[Tue 05:37:23 PM] <mriedem> image?
[Tue 05:37:24 PM] <dhellmann> I want to boot it ephemeral and attach a volume
[Tue 05:37:47 PM] <mordred> yah- source_type also volume
[Tue 05:38:01 PM] <mriedem> have you tried omitting boot_index
[Tue 05:38:02 PM] <mriedem> ?
[Tue 05:38:05 PM] <mordred> yes. that works
[Tue 05:38:12 PM] <mordred> as does providing -1 or '-1'
[Tue 05:38:46 PM] <mordred> however, None worked in the past and is still referneced in the docs ... so maybe there's a bug somewhere, I'm just not sure what to file/write up
[Tue 05:38:59 PM] <mordred> (we can make shade dtrt by omitting or using -1
[Tue 05:39:14 PM] <mriedem> oomichi: Kevin_Zheng: gmann: alex_xu: do you remember anything recently related to this? ^
[Tue 05:39:40 PM] <mriedem> https://review.openstack.org/#/c/410006/
[Tue 05:40:50 PM] <mriedem> mordred: i wouldn't put a ton of faith in the api-ref docs on this one
[Tue 05:40:57 PM] <mordred> mriedem: ok
[Tue 05:41:03 PM] <mriedem> i'd be interested in knowing if it's different on like a liberty install
[Tue 05:41:29 PM] <mordred> mriedem: None worked against dreamhost - whatever version is running there
[Tue 05:41:43 PM] <mriedem> mordred: because my guess is the wording in the api-ref was pulled from http://docs.openstack.org/developer/nova/block_device_mapping.html and that's more about the actual object code in nova rather than the API
[Tue 05:42:05 PM] <mordred> ah - k. so it's likely just a docs bug
[Tue 05:42:06 PM] <dhellmann> dreamhost is either mitaka or newton
[Tue 05:42:15 PM] <mriedem> i'd file a bug either way so we can investigate
[Tue 05:42:27 PM] <dhellmann> I can do that
[Tue 05:42:37 PM] <mriedem> plus, if the api is ok but the docs are wrong, we can use the bug to fix the docs
[Tue 05:42:50 PM] <mordred> cool.
[Tue 05:42:52 PM] <mordred> thanks dhellmann !
[Tue 05:43:10 PM] <dhellmann> thanks, mordred & mriedem
[Tue 05:45:00 PM] <mriedem> also,
[Tue 05:45:07 PM] <mriedem> i wonder if you were hitting a nova v2 vs v2.1 endpoint before...
[Tue 05:45:11 PM] <mriedem> v2 didn't have schema validation
[Tue 05:45:22 PM] <mriedem> mordred: dhellmann: ^
[Tue 05:45:39 PM] <mordred> oh - it's entirely possible
[Tue 05:46:06 PM] <mordred> shade is (currently) just piggybacking novaclient which grabs latest microversion aiui
[Tue 05:46:12 PM] <mordred> I haven't gotten to doing explicit microversion code myself
[Tue 05:46:29 PM] <mordred> (that'll be next month - expect plenty of questions from me)
[Tue 05:47:02 PM] <mriedem> there is where the legacy v2 api bdm v2 stuff happened https://github.com/openstack/nova/blob/liberty-eol/nova/api/openstack/compute/legacy_v2/servers.py#L424
[Tue 05:47:54 PM] <mriedem> which only checked boot_index here https://github.com/openstack/nova/blob/liberty-eol/nova/block_device.py#L201
[Tue 05:48:00 PM] <mriedem> if source_type == 'image' and destination_type == 'local':
[Tue 05:48:33 PM] <mriedem> and then that dict was probably turned into an object https://github.com/openstack/nova/blob/liberty-eol/nova/objects/block_device.py#L73
[Tue 05:48:36 PM] <mriedem> where boot_index can be None
[Tue 05:49:18 PM] <mriedem> the compute api did some validation here too https://github.com/openstack/nova/blob/liberty-eol/nova/compute/api.py#L1315
[Tue 05:49:20 PM] <mriedem> on the object
[Tue 05:49:22 PM] <mriedem> SO,
[Tue 05:49:38 PM] <mriedem> i think it just means that the v2.1 API json schema validation regressed from the v2 API validation
[Tue 05:49:56 PM] <mriedem> would be cool if you guys could confirm that it was working when you were talking to a v2 compute endpoint cloud vs v2.1
[Tue 05:50:03 PM] <mriedem> but that'd be my professional opinion
[Tue 05:50:09 PM] <mriedem> dr debug
[Tue 05:51:06 PM] <mriedem> dhellmann: i've got to run but just throw ^ into the bug report and i'll check it out later when i'm home

[Doug Hellmann (doug-hellmann)]

Changing shade to pass -1, '-1', or to leave the boot_index value out entirely all makes it work. See https://review.openstack.org/430465

[Matt Riedemann (mriedem)]

Fix is here: https://review.openstack.org/#/c/430497/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/430497
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e34f05edb2efc79bfdd8e73cca8fa02ea6ef2d60
Submitter: Jenkins
Branch: master

commit e34f05edb2efc79bfdd8e73cca8fa02ea6ef2d60
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 7 20:28:13 2017 -0500

    Allow None for block_device_mapping_v2.boot_index

    The legacy v2 API allowed None for the boot_index [1]. It
    allowed this implicitly because the API code would convert
    the block_device_mapping_v2 dict from the request into a
    BlockDeviceMapping object, which has a boot_index field that
    is nullable (allows None).

    The API reference documentation [2] also says:

    "To disable a device from booting, set the boot index
    to a negative value or use the default boot index value,
    which is None."

    It appears that with the move to v2.1 and request schema
    validation, the boot_index schema was erroneously set to
    not allow None for a value, which is not backward compatible
    with the v2 API behavior.

    This change fixes the schema to allow boot_index=None again
    and adds a test to show it working.

    This should not require a microversion bump since it's fixing
    a regression in the v2.1 API which worked in the v2 API and
    is already handled throughout Nova's block device code.

    Closes-Bug: #1662699

    [1] https://github.com/openstack/nova/blob/13.0.0/nova/compute/api.py#L1268
    [2] http://developer.openstack.org/api-ref/compute/#create-server

    Change-Id: Ice78a0982bcce491f0c9690903ed2c6b6aaab1be

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/431595

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/431610

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/431628

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431595
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ff1925ae47245f056690889a6e063ebce1bb7a80
Submitter: Jenkins
Branch: stable/ocata

commit ff1925ae47245f056690889a6e063ebce1bb7a80
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 7 20:28:13 2017 -0500

    Allow None for block_device_mapping_v2.boot_index

    The legacy v2 API allowed None for the boot_index [1]. It
    allowed this implicitly because the API code would convert
    the block_device_mapping_v2 dict from the request into a
    BlockDeviceMapping object, which has a boot_index field that
    is nullable (allows None).

    The API reference documentation [2] also says:

    "To disable a device from booting, set the boot index
    to a negative value or use the default boot index value,
    which is None."

    It appears that with the move to v2.1 and request schema
    validation, the boot_index schema was erroneously set to
    not allow None for a value, which is not backward compatible
    with the v2 API behavior.

    This change fixes the schema to allow boot_index=None again
    and adds a test to show it working.

    This should not require a microversion bump since it's fixing
    a regression in the v2.1 API which worked in the v2 API and
    is already handled throughout Nova's block device code.

    Closes-Bug: #1662699

    [1] https://github.com/openstack/nova/blob/13.0.0/nova/compute/api.py#L1268
    [2] http://developer.openstack.org/api-ref/compute/#create-server

    Change-Id: Ice78a0982bcce491f0c9690903ed2c6b6aaab1be
    (cherry picked from commit e34f05edb2efc79bfdd8e73cca8fa02ea6ef2d60)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431610
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dce8618166d80dc6cf2854920f6275eee73b8d84
Submitter: Jenkins
Branch: stable/newton

commit dce8618166d80dc6cf2854920f6275eee73b8d84
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 7 20:28:13 2017 -0500

    Allow None for block_device_mapping_v2.boot_index

    The legacy v2 API allowed None for the boot_index [1]. It
    allowed this implicitly because the API code would convert
    the block_device_mapping_v2 dict from the request into a
    BlockDeviceMapping object, which has a boot_index field that
    is nullable (allows None).

    The API reference documentation [2] also says:

    "To disable a device from booting, set the boot index
    to a negative value or use the default boot index value,
    which is None."

    It appears that with the move to v2.1 and request schema
    validation, the boot_index schema was erroneously set to
    not allow None for a value, which is not backward compatible
    with the v2 API behavior.

    This change fixes the schema to allow boot_index=None again
    and adds a test to show it working.

    This should not require a microversion bump since it's fixing
    a regression in the v2.1 API which worked in the v2 API and
    is already handled throughout Nova's block device code.

    Closes-Bug: #1662699

    [1] https://github.com/openstack/nova/blob/13.0.0/nova/compute/api.py#L1268
    [2] http://developer.openstack.org/api-ref/compute/#create-server

    Change-Id: Ice78a0982bcce491f0c9690903ed2c6b6aaab1be
    (cherry picked from commit e34f05edb2efc79bfdd8e73cca8fa02ea6ef2d60)
    (cherry picked from commit ff1925ae47245f056690889a6e063ebce1bb7a80)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431628
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9b3c4736a35b0db6ceff38786fb706a6a312a7ab
Submitter: Jenkins
Branch: stable/mitaka

commit 9b3c4736a35b0db6ceff38786fb706a6a312a7ab
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 7 20:28:13 2017 -0500

    Allow None for block_device_mapping_v2.boot_index

    The legacy v2 API allowed None for the boot_index [1]. It
    allowed this implicitly because the API code would convert
    the block_device_mapping_v2 dict from the request into a
    BlockDeviceMapping object, which has a boot_index field that
    is nullable (allows None).

    The API reference documentation [2] also says:

    "To disable a device from booting, set the boot index
    to a negative value or use the default boot index value,
    which is None."

    It appears that with the move to v2.1 and request schema
    validation, the boot_index schema was erroneously set to
    not allow None for a value, which is not backward compatible
    with the v2 API behavior.

    This change fixes the schema to allow boot_index=None again
    and adds a test to show it working.

    This should not require a microversion bump since it's fixing
    a regression in the v2.1 API which worked in the v2 API and
    is already handled throughout Nova's block device code.

    Closes-Bug: #1662699

    [1] https://github.com/openstack/nova/blob/13.0.0/nova/compute/api.py#L1268
    [2] http://developer.openstack.org/api-ref/compute/#create-server

    Change-Id: Ice78a0982bcce491f0c9690903ed2c6b6aaab1be
    (cherry picked from commit e34f05edb2efc79bfdd8e73cca8fa02ea6ef2d60)
    (cherry picked from commit ff1925ae47245f056690889a6e063ebce1bb7a80)
    (cherry picked from commit dce8618166d80dc6cf2854920f6275eee73b8d84)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.