attachments API's(create and update) fails with 500 InternalServerError if user don't pass 'initiator' parameter in connector object

[Pooja Jadhav]

When user passes request body to attachments 'create' and 'update' API's without 'initiator' parameter in connector object
then cinder volume service raises 'InvalidConnectorException' exception with error message as saying "Connector doesn't have required information: initiator".

The raised 'InvalidConnectorException' exception is not caught at cinder-api side which results into 500 InternalServerError.

Steps to reproduce:

command:

curl -g -i -X POST http://10.232.48.206/volume/v3/b36aa91b00ac4c9091fdec0ef4dbbcd9/attachments -H "User-Agent: python-cinderclient" -H "OpenStack-API-Version: volume 3.27" -H "X-OpenStack-Cinder-API-Version: 3.27" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: e7ad96cff28449fd98176390144141a6" -d '{"attachment": {"instance_uuid": "462dcc2d-130d-4654-8db1-da0df2da6a0d", "volume_uuid": "936d5fb7-5521-41c3-b64a-4d7e7ebcc49b", "connector": {"multipath": false, "mountpoint": "/dev/vdb", "mode": "ro"}}}'

Output:

HTTP/1.1 500 Internal Server Error
Date: Tue, 31 Oct 2017 06:59:26 GMT
Server: Apache/2.4.18 (Ubuntu)
OpenStack-API-Version: volume 3.27
Vary: OpenStack-API-Version
Content-Length: 85
Content-Type: application/json
x-compute-request-id: req-6a53f599-77aa-45d1-a413-2fe2a33f626f
x-openstack-request-id: req-6a53f599-77aa-45d1-a413-2fe2a33f626f
Connection: close

{"computeFault": {"message": "Unable to create attachment for volume.", "code": 500}}

c-api logs:

Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments [None req-485a1100-f8d0-4353-87a9-11b053c2ae11 admin admin] Unable to create attachment for volume.: RemoteError: Remote error: InvalidInput Invalid input received: Connector doesn't have required information: initiator
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/cinder/cinder/volume/manager.py", line 4334, in attachment_update\n connector)\n', u' File "/opt/stack/cinder/cinder/volume/manager.py", line 4261, in _connection_create\n raise exception.InvalidInput(reason=six.text_type(err))\n', u"InvalidInput: Invalid input received: Connector doesn't have required information: initiator\n"].
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/api/v3/attachments.py", line 179, in create
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments connector=connector))
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/api.py", line 2010, in attachment_create
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments if connector:
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/rpc.py", line 187, in _wrapper
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments return f(self, *args, **kwargs)
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/rpcapi.py", line 442, in attachment_update
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments attachment_id=attachment_id)
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments retry=self.retry)
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments timeout=timeout, retry=retry)
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments retry=retry)
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments raise result
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments RemoteError: Remote error: InvalidInput Invalid input received: Connector doesn't have required information: initiator
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/cinder/cinder/volume/manager.py", line 4334, in attachment_update\n connector)\n', u' File "/opt/stack/cinder/cinder/volume/manager.py", line 4261, in _connection_create\n raise exception.InvalidInput(reason=six.text_type(err))\n', u"InvalidInput: Invalid input received: Connector doesn't have required information: initiator\n"].
Oct 31 12:55:27 openstack-VirtualBox <email address hidden>[31203]: ERROR cinder.api.v3.attachments

Environment details:

devstack:
commit 2c9343e5db44fa7a41ca6924737331dd9088ef8f
Merge: 464d6e0 e8db867
Author: Zuul <email address hidden>
Date: Fri Oct 27 08:28:49 2017 +0000

    Merge "Change ENABLE_KSM to $ENABLE_KSM"

Cinder:
commit 9dc606da524ac3dfd7f2f27d6df28e365939cad6
Merge: 8cbee98 104d6fc
Author: Zuul <email address hidden>
Date: Sun Oct 29 23:36:56 2017 +0000

    Merge "ibm-storage: Fix create_volume_from_snapshot"

cinder backend used in cinder.conf:

enabled_backends = lvmdriver-1

[lvmdriver-1]
image_volume_cache_enabled = True
volume_clear = zero
lvm_type = auto
iscsi_helper = tgtadm
volume_group = stack-volumes-lvmdriver-1
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

[Gorka Eguileor (gorka)]

The problem is that we are not unwrapping the InvalidInput exception from the RemoteError. We should do that in the middleware so we handle all exceptions of the same kind and not just this one.

[Vishakha Agarwal (vishakha.agarwal)]

Hi ,

Is anyone working on this bug?

Thanks

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572302

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572674
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c25a1485f2976bec871e804cb5aacd3fe095b05e
Submitter: Zuul
Branch: master

commit c25a1485f2976bec871e804cb5aacd3fe095b05e
Author: TommyLike <email address hidden>
Date: Wed Jun 6 13:36:12 2018 +0800

    Remove kwargs 'message' when exception is created

    Now when rpc client rebuild exception at the middleware,
    the process will always fail because we have duplicate
    keyword 'message' there. Considering we already have a bunch
    of exceptions who need format key 'message' as below:

    ```
    class VolumeDriverException(CinderException):
        message = _("Volume driver reported an error: %(message)s")
    ```
    Instead of refactor all of them, this patch just removes
    the kwargs 'message' when exception is initialized.

    Closes-Bug: #1728826

    Change-Id: I6bfa34a42df1f44e16b7536246def610821b7cba

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vishakha Agarwal (<email address hidden>) on branch: master
Review: https://review.openstack.org/572302
Reason: https://review.openstack.org/#/c/572674/

resolves this issue.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/592699

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/592699
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2ecaad46a737c24965ae81fc19947db9264de684
Submitter: Zuul
Branch: stable/queens

commit 2ecaad46a737c24965ae81fc19947db9264de684
Author: TommyLike <email address hidden>
Date: Wed Jun 6 13:36:12 2018 +0800

    Remove kwargs 'message' when exception is created

    Now when rpc client rebuild exception at the middleware,
    the process will always fail because we have duplicate
    keyword 'message' there. Considering we already have a bunch
    of exceptions who need format key 'message' as below:

    ```
    class VolumeDriverException(CinderException):
        message = _("Volume driver reported an error: %(message)s")
    ```
    Instead of refactor all of them, this patch just removes
    the kwargs 'message' when exception is initialized.

    Closes-Bug: #1728826

    Change-Id: I6bfa34a42df1f44e16b7536246def610821b7cba
    (cherry picked from commit c25a1485f2976bec871e804cb5aacd3fe095b05e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.4 release.