nova api v2.1 does not allow to use autodetection of volume device path

[Valeriy Ponomaryov]

Using API v2.1 we are forced to provide device path attaching a volume to an instance.

using API v2.0 it allowed to provide 'auto' and in this case Nova calculated it by itself.

It is very useful when we do not care about exact device path.

using APi v2.1 Nova, at first verifies request body [1] and only then have logic to autodetect "device path". So, either autodetect is dead code now or request validation should be changed.

For the moment, this bug is blocker for Manila project.

We get one of two errors:

Returning 400 to user: Invalid input for field/attribute device. Value: None. None is not of type 'string' __call__

or

Returning 400 to user: Invalid input for field/attribute device. Value: auto. u'auto' does not match '(^/dev/x{0,1}[a-z]{0,1}d{0,1})([a-z]+)[0-9]*$'

Where Nova client says explicitly:

$ nova help volume-attach
usage: nova volume-attach <server> <volume> [<device>]

Attach a volume to a server.

Positional arguments:
  <server> Name or ID of server.
  <volume> ID of the volume to attach.
  <device> Name of the device e.g. /dev/vdb. Use "auto" for autoassign (if supported)

That "device" is optional and can be set to 'auto'.

[1] https://github.com/openstack/nova/blob/b7c8a73824211db9627962abd31b8801cc2c2880/nova/api/openstack/compute/volumes.py#L270

[Valeriy Ponomaryov (vponomaryov)]

Also, official docs for v2.1 says that "device" is optional, see http://developer.openstack.org/api-ref-compute-v2.1.html#attachVolume

[Valeriy Ponomaryov (vponomaryov)]

But novaclient always provides key "device" that makes this key "required" de facto, see: https://github.com/openstack/python-novaclient/blob/f3fef4ab/novaclient/v2/volumes.py#L144

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219696

[Matt Riedemann (mriedem)]

This is also a bug in kilo:

https://github.com/openstack/nova/blob/stable/kilo/nova/api/openstack/compute/schemas/v3/volumes.py#L71

[Matt Riedemann (mriedem)]

And a bug in juno:

https://github.com/openstack/nova/blob/stable/juno/nova/api/openstack/compute/schemas/v3/extended_volumes.py#L47

We never caught any of this because every tempest test for volume attachment specifies the device name.

[Matt Riedemann (mriedem)]

Actually this isn't a problem in nova, device is optional, it's not nullable though. I have a change in tempest that attaches a volume and I don't specify device, it works fine:

http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-nova-v21-full/03cdd7a/logs/tempest.txt.gz#_2015-09-02_17_32_14_204

So the bug is in novaclient passing device=None in the request.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219835

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/219696

[Matt Riedemann (mriedem)]

Example failure in the gate for manila:

http://logs.openstack.org/33/219333/1/gate/gate-manila-tempest-dsvm-neutron-multibackend/4c4572b/logs/screen-m-shr.txt.gz?level=TRACE#_2015-09-02_06_11_03_047

2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher [req-93761677-88c3-4b18-8b60-021e06ee204c ] Exception during message handling: Invalid input received: Invalid input for field/attribute device. Value: None. None is not of type 'string' (HTTP 400) (Request-ID: req-5134ee5b-d445-431f-b507-edbea510f03b).
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 445, in create_share_instance
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 416, in create_share_instance
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher context, share_instance, share_server=share_server)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 125, in wrap
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kwargs)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 209, in create_share
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher volume)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 376, in _attach_volume
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher return do_attach(volume)
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-09-02 06:11:03.047 2334 ERROR oslo_messaging.rpc.dispatcher ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/219835
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=eaf1e56b210c94e7d26ca99b5931f3688cef984c
Submitter: Jenkins
Branch: master

commit eaf1e56b210c94e7d26ca99b5931f3688cef984c
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 2 17:05:46 2015 -0700

    Update path to subunit2html in post_test_hook

    Per:

    http://lists.openstack.org/pipermail/openstack-dev/2015-August/072982.html

    The location of subunit2html changed on the images in the gate
    so update the path used in the post_test_hook.

    Long-term we should just use what's in devstack-gate.

    Change-Id: I5e50e7d7ad845aba26403df1df412c0a139a6dc7
    Closes-Bug: #1491646

    --------------
    squashed with:
    --------------

    Don't pass null device when attaching a volume

    The v2.1 API schema rejects null device values in an
    os-volume_attachments request, so only include the device in the request
    if one is specified on the command line.

    Closes-Bug: #1491325

    Change-Id: I4fa4019f19f9af6ff350db2fb6e524fa8570a6f3

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW52YWxpZCBpbnB1dCBmb3IgZmllbGQvYXR0cmlidXRlIGRldmljZS4gVmFsdWU6IE5vbmUuIE5vbmUgaXMgbm90IG9mIHR5cGUgJ3N0cmluZydcIiBBTkQgdGFnczpcInNjcmVlbi1tLXNoci50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MTMyMTU3NzQ2NywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/219696
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=354e972f9fbcc9bbb8cb618f7fb8e00c8fbb2d95
Submitter: Jenkins
Branch: master

commit 354e972f9fbcc9bbb8cb618f7fb8e00c8fbb2d95
Author: Davanum Srinivas <email address hidden>
Date: Wed Sep 2 06:13:54 2015 -0700

    Allow to use autodetection of volume device path

    API contract says, device is optional. We should allow the
    device property to be nullable
    http://developer.openstack.org/api-ref-compute-v2.1.html#attachVolume

    Note that in python nova client "auto" is turned into None for the
    device before the call is made to Nova API.

    Closes-Bug: #1491325
    Change-Id: Ibfd83b6abdfeec328019246a372363cada53869e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/220567

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220567
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=17b5052f0b399237e4d8074d12f2075422bcf57d
Submitter: Jenkins
Branch: stable/kilo

commit 17b5052f0b399237e4d8074d12f2075422bcf57d
Author: Davanum Srinivas <email address hidden>
Date: Wed Sep 2 06:13:54 2015 -0700

    Allow to use autodetection of volume device path

    API contract says, device is optional. We should allow the
    device property to be nullable
    http://developer.openstack.org/api-ref-compute-v2.1.html#attachVolume

    Note that in python nova client "auto" is turned into None for the
    device before the call is made to Nova API.

    Conflicts:
            nova/tests/unit/api/openstack/compute/test_volumes.py

    NOTE(mriedem): The tests were moved in liberty, otherwise nothing
    changed.

    Closes-Bug: #1491325
    Change-Id: Ibfd83b6abdfeec328019246a372363cada53869e
    (cherry picked from commit 354e972f9fbcc9bbb8cb618f7fb8e00c8fbb2d95)

[Matt Riedemann (mriedem)]

Since the v2.1 API was still experimental in Juno I don't think this is worth fixing there.