ceph rbd username and secret should be configured in nova-compute, not passed from nova-volume/cinder

Bug #1065883 reported by James Page
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
James Page
cinder (Ubuntu)
nova (Ubuntu)
Fix Released

Bug Description

I'm testing using ceph RADOS block devices to back nova volumes; however I've hit an issue which limits its usefulness in environments where cephx authentication is required.

Configuration is directly taken from http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-cinder-nova-volume. Note that nova-volume and nova-compute are running on different hosts.

The problem is as follows:

The rbd_user and rbd_secret_uuid must be configured in nova-volume to ensure that when the nova-compute nodes attach volumes to instances, they will use the libvirt stored secret.

However, the libvirt secret UUID when created on each of the compute nodes is going to be different; and nova-compute will try to attach using the secret provided from nova-volume - for which it has no knowledge.

I also want to configure nova-compute with a different username to nova-volume/cinder to provide more granular access control to ceph.

The user and secret_uuid should be configured in nova-compute; not provided by nova-volume.

I've worked around this using this patch/hack:

=== modified file 'nova/virt/libvirt/volume.py'
--- nova/virt/libvirt/volume.py 2012-08-27 15:37:18 +0000
+++ nova/virt/libvirt/volume.py 2012-10-12 08:37:38 +0000
@@ -88,9 +88,11 @@
         conf.serial = connection_info.get('serial')
         netdisk_properties = connection_info['data']
         if netdisk_properties.get('auth_enabled'):
- conf.auth_username = netdisk_properties['auth_username']
+ conf.auth_username = FLAGS.rbd_user or \
+ netdisk_properties['auth_username']
             conf.auth_secret_type = netdisk_properties['secret_type']
- conf.auth_secret_uuid = netdisk_properties['secret_uuid']
+ conf.auth_secret_uuid = FLAGS.rbd_secret_uuid or \
+ netdisk_properties['secret_uuid']
         return conf

Which basically allows me to override the auth_username and auth_secret_uuid through the nova-compute configuration file.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: nova-compute (not installed)
ProcVersionSignature: Ubuntu 3.5.0-17.27-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.1-0ubuntu2
Architecture: amd64
Date: Fri Oct 12 09:38:32 2012
SourcePackage: nova
UpgradeStatus: Upgraded to quantal on 2012-06-11 (122 days ago)

Revision history for this message
James Page (james-page) wrote :
James Page (james-page)
description: updated
Revision history for this message
James Page (james-page) wrote :

Revised patch which ensures sheepdog handling does not get interfered with

tags: added: patch
Chuck Short (zulcss)
Changed in nova (Ubuntu):
status: New → Invalid
status: Invalid → Confirmed
Revision history for this message
Josh Durgin (jdurgin) wrote :

You can actually specify the uuid for the secret when you add it to libvirt, so it can be the same on all compute hosts.


<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.volumes secret</name>

Libvirt only generates a random uuid if you don't specify one. I'll update the Ceph docs to clarify this.

Your patch does make sense if you want to control more finely which rados user's your using on the compute nodes. It's easier than running multiple (cinder|nova)-volume processes, but the long term solution probably involves changing the volume driver to use different rados pools and users based on volume_type or some other configuration.

However, with the current rbd volume driver using only a single pool, I'm not sure how much finer-grained the compute node permissions could be compared to the volume service permissions. What do you have in mind?

BTW, sheepdog and nbd don't have auth support through libvirt, so you don't need to check specifically for rbd in your patch.

Revision history for this message
James Page (james-page) wrote :

Hi Josh

I was aware that was possible; however I'm deploying openstack automatically and I don't really want to pass the uuid around between nova-volume and nova-compute nodes.

I simply want to provide each of the compute nodes with the cephx key it needs to use and a generated username - and it will just configure its own set of secrets and configure nova appropriately, overriding the config that nova-volume may/will have sent.

My finer grained access control requirement was really around having different keys for volume/cinder and compute - so if I add/remove additional compute farms I can easily manage the keys on a per role basis.

I guess I was just being hyper-cautious with the rbd check in the patch :-)

Revision history for this message
Josh Durgin (jdurgin) wrote :

I see, that makes sense now. It'd be good to get this patch upstream for grizzly.

Revision history for this message
James Page (james-page) wrote :

I'll work on doing that as my first code contribution to OpenStack!

James Page (james-page)
Changed in nova:
assignee: nobody → James Page (james-page)
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2012.2-0ubuntu5

nova (2012.2-0ubuntu5) quantal-proposed; urgency=low

  [ Adam Gandelman ]
  * Move management of /var/lib/nova/volumes from nova-common to
    nova-volume. Ensure it has proper permissions. (LP: #1065320)
  * debian/patches/avoid_setuptools_git_dependency.patch: Remove
    setuptools_git from tools/pip-requires to avoid it being automatically
    added to python-nova's runtime dependencies. (LP: #1059907)

  [ Chuck Short ]
  * debian/patches/rbd-security.patch: Support override of ceph rbd
    user and secret in nova-compute. (LP: #1065883)
  * debian/patches/ubuntu/fix-libvirt-firewall-slowdown.patch: Fix
    refreshing of security groups in libvirt not to block on RPC calls.
    (LP: #1062314)
  * debian/patches/ubuntu/fix-ec2-volume-id-mappings.patch: Read deleted
    snapshot and volume id mappings. (LP: #1065785)
 -- Chuck Short <email address hidden> Fri, 12 Oct 2012 12:35:01 -0500

Changed in nova (Ubuntu):
status: Confirmed → Fix Released
James Page (james-page)
Changed in cinder (Ubuntu):
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/14458
Committed: http://github.com/openstack/nova/commit/af51b46b1e08b26c07bd32019e54b9c521cb7813
Submitter: Jenkins
Branch: master

commit af51b46b1e08b26c07bd32019e54b9c521cb7813
Author: James Page <email address hidden>
Date: Mon Oct 15 13:21:55 2012 +0100

    Allow local rbd user and secret_uuid configuration

    By default, the rbd_user and rbd_secret_uuid are specified in the
    nova-volume/cinder configuration and passed to nova-compute when
    volumes are attached to instances.

    This change allows these values to be specified locally in
    nova-compute which means access control to RADOS devices in ceph
    can be managed independently from nova-volume/cinder with no
    requirement for consistent uuid's for libvirt secrets.

    Fixes bug 1065883.

    Change-Id: I9f07d040ae267bfbe8f794a5d22d327106314cc6

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Josh Durgin (jdurgin)
Changed in cinder:
status: New → Invalid
Revision history for this message
Paul Collins (pjdc) wrote :

Is there an essex variant of this patch available?

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments