Bug #1635008 “rbd credentials in nova.conf overwrite connection_...” : Bugs : OpenStack Compute (nova)

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-10-20:

#1

It looks like the problem with the volumes stuff is that the libvirt volume driver that handles rbd connections isn't checking the connection_info for the rbd credentials, it's just using the nova config:

https://github.com/openstack/nova/blob/14.0.0.0rc2/nova/virt/libvirt/volume/net.py#L70

So that should check the connection_info first, and if nothing there use the nova config values.

tags:	added: ceph
Changed in nova:
status:	New → Confirmed

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-10-20:

#2

Here is an example of the connection_info we get back from cinder for rbd volume connections:

http://logs.openstack.org/21/386221/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/54c38f9/logs/screen-n-cpu.txt.gz#_2016-10-13_21_08_12_361

{
   "connection_info":{
      "driver_volume_type":"rbd",
      "data":{
         "secret_type":"ceph",
         "name":"volumes/volume-ebf06f1b-df7f-47c7-8d7b-1473a8b04db1",
         "encrypted":false,
         "cluster_name":"ceph",
         "secret_uuid":null,
         "qos_specs":null,
         "hosts":[
            "127.0.0.1"
         ],
         "volume_id":"ebf06f1b-df7f-47c7-8d7b-1473a8b04db1",
         "auth_enabled":true,
         "access_mode":"rw",
         "auth_username":"cinder",
         "ports":[
            "6789"
         ]
      }
   }
}

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-10-20:

#3

This is the cinder rbd initialize_connection method:

https://github.com/openstack/cinder/blob/9.0.0/cinder/volume/drivers/rbd.py#L823

So on the cinder side, rbd_user in this case is 'cinder'.

I guess in this test, rbd_user in nova is also 'cinder':

http://logs.openstack.org/21/386221/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/54c38f9/logs/etc/nova/nova.conf.txt.gz

In this case, the secret uuid is set in nova.conf:

rbd_secret_uuid = d032268a-4382-480d-b505-528b8af5d96e
rbd_user = cinder

but not cinder.conf:

[ceph]
rbd_max_clone_depth = 5
rbd_flatten_volume_from_snapshot = False
rbd_uuid = d032268a-4382-480d-b505-528b8af5d96e
rbd_user = cinder
rbd_pool = volumes
rbd_ceph_conf =
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph

Still, seems better to use what's in the connection_info if auth is enabled.

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)
importance:	Undecided → Low

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-20: Fix proposed to nova (master)

#4

Fix proposed to branch: master
Review: https://review.openstack.org/389389

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-21:

#5

Fix proposed to branch: master
Review: https://review.openstack.org/389399

OpenStack Infra (hudson-openstack) on 2016-11-28

Changed in nova:
assignee:	Matt Riedemann (mriedem) → Michael Still (mikal)

OpenStack Infra (hudson-openstack) on 2016-11-30

Changed in nova:
assignee:	Michael Still (mikal) → Matt Riedemann (mriedem)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-30: Fix merged to nova (master)

#6

Reviewed: https://review.openstack.org/389389
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5595b4613a81908433776fae6fb4b34a4b3ab666
Submitter: Jenkins
Branch: master

commit 5595b4613a81908433776fae6fb4b34a4b3ab666
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 20 19:05:53 2016 -0400

libvirt: cleanup network volume driver auth config

    The LibvirtNetVolumeDriver is handling both rbd and
    sheepdog (iscsi) connections. The auth config logic
    is mingling both backends which is really confusing.

    For example, the iscsi protocol only defines auth_method,
    auth_username and auth_password. It does not set an
    auth_enabled value in the connection_info['data'] dict.

This change simplifies the logic involved for setting
the auth config by decoupling the rbd/iscsi handlers.

    A follow-up change will build on this to fix the
    rbd auth config to prefer the cinder volume connection_info
    auth data over the local config for nova in the case
    that different cinder backends are used for ephemeral
    and block storage.

Change-Id: I8a55d87f75ecad757ce81b1f5f77c3a551154a17
Partial-Bug: #1635008

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-30:

#7

Reviewed: https://review.openstack.org/389399
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b89efa3ef611a1932df0c2d6e6f30315b5111a57
Submitter: Jenkins
Branch: master

commit b89efa3ef611a1932df0c2d6e6f30315b5111a57
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 4 10:39:57 2016 -0400

libvirt: prefer cinder rbd auth values over nova.conf

    In the case that the ceph storage backing volumes is different
    from the one backing ephemeral storage in nova, the auth values
    in the rbd connection_info could be different and not work if
    we are using the nova.conf values for ephemeral storage.

    This change makes the volume connection config code for rbd
    prefer the cinder connection_info values if they exist, and
    only falls back to nova config values if cinder doesn't have
    anything set.

Depends-On: I4655cae3212d589177d2570403b563a83aad529a

Change-Id: Idcbada705c1d38ac5fd7c600141c2de7020eae25
Closes-Bug: #1635008

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

John McEleney (launchpa6) wrote on 2016-12-03:

#8

I've already posted this comment on Github, so I hope it's okay to double-post it here, as in retrospect this might be the better forum.

The upgrade note in the patch states that the setting in cinder.conf will be "pulled" and take precedence over the rbd_user etc. from nova.conf. I'm not sure that this is strictly true. Am I right in thinking that the credentials are (and always have been) pulled from the cinder.conf at volume creation time, and that this metadata is stored in the block_device_mapping table in the nova database?

If I'm right then a user that has pre-existing volumes, who previously omitted ceph credentials from their cinder.conf will not have any credentials stored in block_device_mapping (presumably the source of connection_info).

I myself have ended up with incomplete credentials stored in block_device_mapping. This is due to my use of Ubuntu's cinder-ceph juju charms, so this is not an isolated case. The juju charms write the rbd user, but not the secret_uuid to the cinder.conf on the cinder-api nodes.

When I initially investigated the issue that this patch addresses, on an unpatched Liberty, I tried adding the ceph credentials to the cinder.conf on my compute hosts, and removed the credentials in nova.conf. The result was that a "null" secret uuid was pulled from the database, and the volume did not work. The secret uuid in my cinder.conf was ignored at this point.

To retrospectively fix this problem for myself, I ran the following SQL update:
update block_device_mapping set connection_info = REPLACE(connection_info,'"secret_uuid": null','"secret_uuid": "62003c88-92c4-4986-92a2-7f9245ed3bb7"') WHERE connection_info LIKE '%"secret_uuid": null%cinder-ceph%';
(cinder-ceph was the name of my pool)

Can anyone confirm that changes (not obvious in this patch) have been made that will pull the credentials from cinder.conf when they are absent in the block_device_mapping table (connection_info)?

If those credentials are not pulled from cinder.conf at run-time, then lots of rbd volumes might fail when the legacy nova.conf settings are pulled in the 'P' release.

Thanks,
John