Creating an image from a cinder volume runs as root

[Pieter Halkema]

Implementing secure nfs for cinder we have run into an issue. Using the deployment guide, root has been squashed and setuid disabled on the exports.

The problem results when trying to generate a cqow2 image from a boot cinder volume. This is run by the service cinder-volume.

What we see in the log is:

1. We generate a cqow2 image from a boot cinder volume: "volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72"

extracted from (/var/log/cinder/volume.log):

2016-10-28 11:01:40.419 12178 TRACE oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2016-10-28 11:01:40.419 12178 TRACE oslo_messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72
2016-10-28 11:01:40.419 12178 TRACE oslo_messaging.rpc.dispatcher Exit code: 1
2016-10-28 11:01:40.419 12178 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2016-10-28 11:01:40.419 1
2178 TRACE oslo_messaging.rpc.dispatcher Stderr: u"qemu-img: Could not open '/var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72': Could not open '/var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72': Permission denied\n"
2016-10-28 11:01:40.419 12178 TRACE oslo_messaging.rpc.dispatcher

2. We observe the following permissions:
[root@cvs1lco03 mnt]# find /var/lib/cinder/mnt -name "*volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72*"
/var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b- 07dc46d0ec72
[root@cvs1lco03 mnt]# ls -ld /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72
-rw-rw----. 1 cinder cinder 21474836480 Oct 28 07:48 /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72
[root@cvs1lco03 mnt]# ls -ld /var/lib/cinder/mnt/
drwxr-xr-x. 5 cinder cinder 4096 Aug 19 12:39 /var/lib/cinder/mnt/
[root@cvs1lc
o03 mnt]# ls -ld /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc
drwxr-xr-x. 2 cinder cinder 4096 Oct 28 07:48 /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc

3. The problem is that the process that converts to the qcow2 image, uses the tool qemu-img which runs as root:
root 20776 20773 4 11:32 ? 00:00:00 /usr/bin/python2 /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O qcow2 /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72 /var/lib/cinder/conversion/tmpqMzQ4L
root 20777 20776 24 11:32 ? 00:00:00 /bin/qemu-img convert -O qcow2 /var/lib/cinder/mnt/c257fa540e38bdfafef6dc8bb505b8dc/volume-5d0f7b70-f573-43dd-a78b-07dc46d0ec72 /var/lib/cinder/conversion/tmpqMzQ4L

We find that cinder uses “/usr/bin/cinder-rootwrap” to launch the process as root.

( https://wiki.openstack.org/wiki/Rootwrap ) more info.

[root@cvs1lco03 mnt]# cat /etc/sudoers.d/cinder
Defaults:cinder !requiretty

cinder ALL = (root) NOPASSWD: /us
r/bin/cinder-rootwrap /etc/cinder/rootwrap.conf *

To make the process run as cinder we have done the following:

We introduce a new optional parameter in a method in cinder python code:
 "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py"
...
  def copy_volume_to_image(self, context, volume, image_service, image_meta):
        """Copy the volume to the specified image."""
        image_utils.upload_volume(context,
                            image_service,
                                  image_meta,
 self.local_path(volume),
                                  run_as_root=False)
.........

We added the last line "run_as_root=False".

Including this optional parameter, we can upload cinder-volumes to glance-images as "cinder":

ps -fe
cinder 27448 20682 0 14:08 ? 00:00:00 qemu-img convert -O qcow2 /var/lib/cinder/mnt/785dce26ac9aad019162c8de535ba4e6/volume-88f53fb3-3514-4e82-bf5b-0e53ebcd87f8 /var/lib/cinder/conversion/tmphsyBEL

Can this be added to the driver?

[Goutham Pacha Ravi (gouthamr)]

As noted, nas_secure_operations has been set to True. In a general workflow with nas_secure_operations, run_as_root is disallowed. In this case it isn't.

[Goutham Pacha Ravi (gouthamr)]

Revisiting this bug. It's been fixed in Cinder's master branch since:

https://review.openstack.org/#/c/147186/

Iae35c722eb4b6b7d02a95690abbc07a63da77ce7

Pieter: What release are you using?

[Pieter Halkema (erantd)]

Kilo and Liberty

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/432337

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/432337
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7d2afb298a11788a52817ad929fcbf53a8e72a60
Submitter: Jenkins
Branch: stable/newton

commit 7d2afb298a11788a52817ad929fcbf53a8e72a60
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Feb 2 22:50:34 2017 -0500

    Fix execution of upload-volume-to-image

    While executing the image utils methods, we need to take
    care not to use root privileges if the environment is running
    in 'secure' mode.

    This is a partial back-port of the fix committed in
    Iae35c722eb4b6b7d02a95690abbc07a63da77ce7 in ocata.

    Change-Id: I8a4912d10797ac8521acc27d02b97b00f8efdb7a
    Closes-Bug: #1649209

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 9.1.4 release.