upload-to-image fails with the NetApp NFS backend

[Ala Rezmerita]

Hi all

OS : Ubuntu 12.04
Cinder Backend : NetApp NFS
Options used for NFS : rw,relatime,rsize=65536,wsize=65536,hard,timeo=600,addr=10.32.99.1
Environment: Openstack Grizzly 2013.1.2 + manually backported patch https://review.openstack.org/#/c/28396/

The command fails:

$cinder upload-to-image --disk-format=raw c690b911-3582-4704-9a7b-a59fbdd51798 "image-from-volume"

The log cinder-volume:
2013-09-05 12:20:24 WARNING [cinder.volume.drivers.nfs] d-nstcls-0000.sto.dev1.val.cw-labs.net:/sata_01/cinder is already mounted
2013-09-05 12:20:24 DEBUG [cinder.volume.drivers.nfs] Available shares ['d-nstcls-0000.sto.dev1.val.cw-labs.net:/sata_01/cinder']
2013-09-05 12:20:24 DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf df -P -B 1 /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63
2013-09-05 12:20:24 DEBUG [cinder.manager] Notifying Schedulers of capabilities ...
2013-09-05 12:20:24 DEBUG [cinder.openstack.common.rpc.amqp] Making asynchronous fanout cast...
2013-09-05 12:20:24 DEBUG [cinder.openstack.common.rpc.amqp] UNIQUE_ID is 64d408ec837442069d20d66f114b5b3f.
2013-09-05 12:20:24 DEBUG [cinder.openstack.common.rpc.amqp] Pool creating new connection
2013-09-05 12:20:24 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on d-msqcld-0000.usr.dev1.val.cw-labs.net:5672
2013-09-05 12:20:24 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on d-msqcld-0000.usr.dev1.val.cw-labs.net:5672
2013-09-05 12:20:24 DEBUG [cinder.service] Creating Consumer connection for Service cinder-volume
2013-09-05 12:20:48 DEBUG [cinder.openstack.common.rpc.amqp] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-31fd38b6-e341-4bd3-bfe2-1bd36524a8a0', u'_context_quota_class': None, u'_unique_id': u'560e5ed3a5a94d87a8b7d2d6eaf4bda4', u'_context_read_deleted': u'no', u'args': {u'image_meta': {u'status': u'queued', u'name': u'Image-from-volume-ubuntu-raw777', u'deleted': False, u'container_format': u'bare', u'created_at': u'2013-09-05T12:20:48.000000', u'disk_format': u'raw', u'updated_at': u'2013-09-05T12:20:48.000000', u'id': u'78027737-3989-48e4-ba4c-423dac01c6a4', u'owner': u'462d7cbd479f499fa75ad9f14571d553', u'min_ram': 0, u'checksum': None, u'min_disk': 0, u'is_public': False, u'deleted_at': None, u'properties': {}, u'size': 0}, u'volume_id': u'c690b911-3582-4704-9a7b-a59fbdd51798'}, u'_context_tenant': u'462d7cbd479f499fa75ad9f14571d553', u'_context_auth_token': '<SANITIZED>', u'_context_is_admin': True, u'version': u'1.3', u'_context_project_id': u'462d7cbd479f499fa75ad9f14571d553', u'_context_timestamp': u'2013-09-05T12:20:48.314450', u'_context_user': u'bbdcffe9d3944c35bcf6a875b6d3d235', u'_context_user_id': u'bbdcffe9d3944c35bcf6a875b6d3d235', u'method': u'copy_volume_to_image', u'_context_remote_address': u'10.32.5.1'}
2013-09-05 12:20:48 DEBUG [cinder.openstack.common.rpc.amqp] unpacked context: {'user_id': u'bbdcffe9d3944c35bcf6a875b6d3d235', 'roles': [u'admin'], 'timestamp': u'2013-09-05T12:20:48.314450', 'auth_token': '<SANITIZED>', 'remote_address': u'10.32.5.1', 'quota_class': None, 'is_admin': True, 'user': u'bbdcffe9d3944c35bcf6a875b6d3d235', 'request_id': u'req-31fd38b6-e341-4bd3-bfe2-1bd36524a8a0', 'project_id': u'462d7cbd479f499fa75ad9f14571d553', 'read_deleted': u'no', 'tenant': u'462d7cbd479f499fa75ad9f14571d553'}
2013-09-05 12:20:48 DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63
2013-09-05 12:20:48 DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o rw,relatime,rsize=65536,wsize=65536,hard,timeo=600 d-nstcls-0000.sto.dev1.val.cw-labs.net:/sata_01/cinder /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63
2013-09-05 12:20:48 DEBUG [cinder.utils] Result was 32
2013-09-05 12:20:48 WARNING [cinder.volume.drivers.nfs] d-nstcls-0000.sto.dev1.val.cw-labs.net:/sata_01/cinder is already mounted
2013-09-05 12:20:48 DEBUG [cinder.image.image_utils] 78027737-3989-48e4-ba4c-423dac01c6a4 was raw, no need to convert to raw
2013-09-05 12:20:48 DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 106 /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63/volume-c690b911-3582-4704-9a7b-a59fbdd51798
2013-09-05 12:20:48 DEBUG [cinder.utils] Result was 1
2013-09-05 12:20:48 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 637, in copy_volume_to_image
    payload['message'] = unicode(error)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 632, in copy_volume_to_image
    image_meta)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nfs.py", line 374, in copy_volume_to_image
    self.local_path(volume))
  File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 258, in upload_volume
    with utils.temporary_chown(volume_path):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 1097, in temporary_chown
    execute('chown', owner_uid, path, run_as_root=True)
  File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 106 /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63/volume-c690b911-3582-4704-9a7b-a59fbdd51798
Exit code: 1
Stdout: ''
Stderr: "/bin/chown: changing ownership of `/var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63/volume-c690b911-3582-4704-9a7b-a59fbdd51798': Operation not permitted\n"

[Jeff Applewhite (ajeffrey)]

I suspect that this is related to permissions on the export on the share from the Netapp controller. Please check what export options were used and make sure that appropriate permissions on the volume are in place.

[Ala Rezmerita (arezmerita)]

For security resons, we can't change the export options used, but the user cinder has already read/write permissions :

# ls -la /var/lib/cinder/netapp-standard/e6403532daa49d571abc394dae312a63
total 1614188
drwxrwxrwx 2 root root 4096 Sep 5 11:17 .
drwxr-xr-x 3 cinder cinder 4096 Aug 27 12:10 ..
drwxrwxrwx 2 root root 4096 Aug 26 09:30 .snapshot
-rw-rw-rw- 1 nobody nogroup 1073741824 Sep 5 11:17 volume-4209e57d-cb63-426e-8932-25dd89f280e2
-rw-rw-rw- 1 nobody nogroup 1073741824 Aug 30 15:37 volume-97b3c2b7-3658-411f-bae4-2570c4f46f1f
-rw-rw-rw- 1 nobody nogroup 2147483648 Sep 4 15:42 volume-9ca4e129-bb07-4e2b-86dd-f4c1dec6c19a
-rw-rw-rw- 1 nobody nogroup 2147483648 Sep 4 14:29 volume-c690b911-3582-4704-9a7b-a59fbdd51798
root@d-ovcclc-0005:/usr/lib/python2.7/dist-packages/cinder#

So why change the ownership by default, and not try first to open the file and change the ownership only if the open fails.

[Ben Swartzlander (bswartz)]

I suggest we fix this by adding a check to upload_volume() that skips the temporary chown if the file mode bits have 004 (world readable) set.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/48901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/48901
Committed: http://github.com/openstack/cinder/commit/1498958299ccb807f406e9d257848cb4259e49e4
Submitter: Jenkins
Branch: master

commit 1498958299ccb807f406e9d257848cb4259e49e4
Author: Navneet Singh <email address hidden>
Date: Sat Aug 3 18:16:44 2013 +0530

    Fix chown fail for nfs file without necessary permission

    chown failed in case of volumes on nfs export when the
    permission is not granted on the storage system. Upload
    volume only requires read access on the nfs file and hence
    chown is not neccessary when current user has read permission.
    Chown only executed when read is not granted to the user.

    Change-Id: Ifa8eff7e79c6e4f15a1feb7cabec8cae8a0b0a83
    Closes-Bug:#1221205

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/49688

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49688
Committed: http://github.com/openstack/cinder/commit/6909dcb061f138843636778c52484acdb4b8679b
Submitter: Jenkins
Branch: stable/grizzly

commit 6909dcb061f138843636778c52484acdb4b8679b
Author: Navneet Singh <email address hidden>
Date: Sat Aug 3 18:16:44 2013 +0530

    Fix chown fail for nfs file without necessary permission

    chown failed in case of volumes on nfs export when the
    permission is not granted on the storage system. Upload
    volume only requires read access on the nfs file and hence
    chown is not neccessary when current user has read permission.
    Chown only executed when read is not granted to the user.

    Conflicts:

     cinder/image/image_utils.py

    Change-Id: Ifa8eff7e79c6e4f15a1feb7cabec8cae8a0b0a83
    Closes-Bug:#1221205
    (cherry picked from commit 1498958299ccb807f406e9d257848cb4259e49e4)