can't attach a read only volume to an instance

[Yogev Rabl]

Description of problem:

An attachment of a read only volume to an instance failed. The openstack was installed as AIO, Cinder was configured with Netapp back end. The following error from the nova compute log:

014-04-13 11:28:17.838 25176 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Invalid input received: Invalid attaching mode 'rw' for volume 3f5828e1-77b2-4302-9cdf-486f7
0834c31.
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher payload)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 244, in decorated_function
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher pass
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 230, in decorated_function
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 272, in decorated_function
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 259, in decorated_function
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3876, in attach_volume
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher bdm.destroy(context)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3873, in attach_volume
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return self._attach_volume(context, instance, driver_bdm)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3894, in _attach_volume
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher self.volume_api.unreserve_volume(context, bdm.volume_id)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3886, in _attach_volume
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher do_check_attach=False, do_driver_attach=True)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 44, in wrapped
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher ret_val = method(obj, context, *args, **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 251, in attach
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher instance['uuid'], self['mount_device'])
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 174, in wrapper
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher res = method(self, ctx, volume_id, *args, **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 263, in attach
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher mountpoint)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 266, in attach
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher 'mode': mode})
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 250, in _action
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return self.api.client.post(url, body=body)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 210, in post
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher return self._cs_request(url, 'POST', **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 174, in _cs_request
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 157, in request
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher raise exceptions.from_response(resp, body)
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher InvalidInput: Invalid input received: Invalid attaching mode 'rw' for volume 3f5828e1-77b2-4302-9cdf-486f70834c31.
2014-04-13 11:28:17.838 25176 TRACE oslo.messaging.rpc.dispatcher
2014-04-13 11:28:17.858 25176 ERROR oslo.messaging._drivers.common [-] Returning exception Invalid input received: Invalid attaching mode 'rw' for volume 3f5828e1-77b2-4302-9cdf-486f70834c31. to caller

Version-Release number of selected component (if applicable):
openstack-nova-conductor-2014.1-0.13.b3.el7.noarch
openstack-swift-object-1.12.0-1.el7.noarch
openstack-glance-2014.1-0.4.b3.el7.noarch
openstack-packstack-puppet-2014.1.1-0.7.dev1018.el7.noarch
openstack-nova-cert-2014.1-0.13.b3.el7.noarch
python-django-openstack-auth-1.1.4-1.el7.noarch
openstack-swift-1.12.0-1.el7.noarch
openstack-keystone-2014.1-0.4.b3.el7.noarch
openstack-utils-2013.2-3.p1.el7.noarch
openstack-nova-api-2014.1-0.13.b3.el7.noarch
openstack-nova-compute-2014.1-0.13.b3.el7.noarch
openstack-nova-novncproxy-2014.1-0.13.b3.el7.noarch
openstack-dashboard-2014.1-0.5.b3.el7.noarch
openstack-swift-account-1.12.0-1.el7.noarch
openstack-swift-proxy-1.12.0-1.el7.noarch
openstack-puppet-modules-2014.1-5.3.el7.noarch
openstack-cinder-2014.1-0.6.b3.el7.noarch
openstack-nova-common-2014.1-0.13.b3.el7.noarch
openstack-nova-console-2014.1-0.13.b3.el7.noarch
openstack-nova-network-2014.1-0.13.b3.el7.noarch
openstack-swift-container-1.12.0-1.el7.noarch
openstack-packstack-2014.1.1-0.7.dev1018.el7.noarch
openstack-nova-scheduler-2014.1-0.13.b3.el7.noarch
openstack-swift-plugin-swift3-1.7-3.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create a volume from a qcow2 image.
2. Attach the volume to an instance and make changes.
3. Detach the volume from the instance.
4. Set the volume as read only.
5. Attach the volume to an instance.

Actual results:
the attachment fails.

Expected results:
the attachment should succeed, but the volume cannot be changed.

[Thang Pham (thang-pham)]

The bug is generic across all volume backends. I can also reproduce this with glusterfs. The bug is caused by a missing option passed to cinderclient, i.e. mode. By default, if mode is not passed to cinderclient.volumes.attach, the mode defaults to 'rw'. The structure of this method found in the Volume class under python-cinderclient/cinderclient/v2/volumes.py is as follows:
     def attach(self, instance_uuid, mountpoint, mode='rw'):

When nova attaches a volume to an instance, the mode is ignored and not passed in, hence every time you attempt to attach a read-only volume, the mode gets set to 'rw' and fails with InvalidInput.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/89684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/89684
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=de05c2994212e83eba3cedf2544cdc549d9f8184
Submitter: Jenkins
Branch: master

commit de05c2994212e83eba3cedf2544cdc549d9f8184
Author: Thang Pham <email address hidden>
Date: Tue Apr 22 15:00:06 2014 -0400

    Set the volume access mode during volume attach

    cinderclient supports an optional keyword argument named 'mode'
    for volumes.attach(), which allows you to set the access mode of
    a volume at attach time. The keyword argument is not passed in
    by nova when it attaches a read-only volume, making the volume
    access mode default to 'rw' instead of the desired 'ro'. This
    patch finds the access mode of the volume and passes it to
    cinderclient, so the correct access mode is used at attach
    time.

    Change-Id: Ib62f454292e60114f96debbad96c4a356dc32a21
    Closes-Bug: #1307088

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/98253

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98253
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c104f2f0eb78a8ea7a58eebdbc510e1f1d1deb9e
Submitter: Jenkins
Branch: stable/icehouse

commit c104f2f0eb78a8ea7a58eebdbc510e1f1d1deb9e
Author: Thang Pham <email address hidden>
Date: Tue Apr 22 15:00:06 2014 -0400

    Set the volume access mode during volume attach

    cinderclient supports an optional keyword argument named 'mode'
    for volumes.attach(), which allows you to set the access mode of
    a volume at attach time. The keyword argument is not passed in
    by nova when it attaches a read-only volume, making the volume
    access mode default to 'rw' instead of the desired 'ro'. This
    patch finds the access mode of the volume and passes it to
    cinderclient, so the correct access mode is used at attach
    time.

    Change-Id: Ib62f454292e60114f96debbad96c4a356dc32a21
    Closes-Bug: #1307088
    (cherry picked from commit de05c2994212e83eba3cedf2544cdc549d9f8184)