[SRU] nova rbd auth fallback uses cinder user with libvirt secret

[Corey Bryant]

[Impact]
From David Ames (thedac), originally posted to https://bugs.launchpad.net/charm-nova-compute/+bug/1671422/comments/25:

Updating this bug. We may decide to move this elsewhere it at some point.

We have a deployment that was upgraded through to pike at which point it was noticed that nova instances with ceph backed volumes would not start.

The cinder key was manually added to the nova-compute nodes in /etc/ceph and with:
sudo virsh secret-define --file /tmp/cinder.secret

However, this did not resolve the problem. It appeared libvirt was trying to use a mixed pair of usernames and keys. It was using the cinder username but the nova-compute key.

Looking at nova's code it falls back to nova.conf when it does not have a secret_uuid from cinder but it was not setting the username correctly.
https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/volume/net.py#L74

The following seems to mitigate this as a temporary fix on nova-compute until we can come up with a complete plan:

https://pastebin.ubuntu.com/p/tGm7C7fpXT/

diff --git a/nova/virt/libvirt/volume/net.py b/nova/virt/libvirt/volume/net.py
index cec43ce93b..8b0148df0b 100644
--- a/nova/virt/libvirt/volume/net.py
+++ b/nova/virt/libvirt/volume/net.py
@@ -71,6 +71,7 @@ class LibvirtNetVolumeDriver(libvirt_volume.LibvirtBaseVolumeDriver):
             else:
                 LOG.debug('Falling back to Nova configuration for RBD auth '
                           'secret_uuid value.')
               + conf.auth_username = CONF.libvirt.rbd_user
                 conf.auth_secret_uuid = CONF.libvirt.rbd_secret_uuid
             # secret_type is always hard-coded to 'ceph' in cinder
             conf.auth_secret_type = netdisk_properties['secret_type']

Apply to /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/net.py

We still need a migration plan to get from the topology with nova-compute directly related to ceph to the topology with cinder-ceph related to nova-compute using ceph-access which would populate cinder's secret_uuid.

It is possible we will need to carry the patch for existing instances. It may be worth getting that upstream as master has the same problem.

[Test Case]
Upgrade a juju-deployed cloud with ceph backend for nova and cinder from pre-ocata to ocata or above. Ensure that nova instances with ceph backed volumes successfully start.

[Regression Potential]
The fix is minimal and will not be fixed in Ubuntu until it has been approved upstream.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/626897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/626897
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=47b7c4f3cc582bf463fd0c796df84736a0074f48
Submitter: Zuul
Branch: master

commit 47b7c4f3cc582bf463fd0c796df84736a0074f48
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500

    Ensure rbd auth fallback uses matching credentials

    As of Ocata, cinder config is preferred for rbd auth values with a
    fallback to nova values [1]. The fallback path, for the case when
    rbd_user is configured in cinder.conf and rbd_secret_uuid is not
    configured in cinder.conf, results in the mismatched use of cinder
    rbd_user with nova rbd_secret_uuid.

    This fixes that fallback path to use nova rbd_user from nova.conf
    with rbd_secret_uuid from nova.conf.

    [1] See commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3

    Thanks to David Ames for this fix.

    Change-Id: Ieba216275c07ab16414065ee47e66915e9e9477d
    Co-Authored-By: David Ames <email address hidden>
    Closes-Bug: #1809454

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627009
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f5d8ee1bfc3b7b9f1a25f85b42e207db0c9f4b04
Submitter: Zuul
Branch: stable/rocky

commit f5d8ee1bfc3b7b9f1a25f85b42e207db0c9f4b04
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500

    Ensure rbd auth fallback uses matching credentials

    As of Ocata, cinder config is preferred for rbd auth values with a
    fallback to nova values [1]. The fallback path, for the case when
    rbd_user is configured in cinder.conf and rbd_secret_uuid is not
    configured in cinder.conf, results in the mismatched use of cinder
    rbd_user with nova rbd_secret_uuid.

    This fixes that fallback path to use nova rbd_user from nova.conf
    with rbd_secret_uuid from nova.conf.

    [1] See commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3

    Thanks to David Ames for this fix.

    Change-Id: Ieba216275c07ab16414065ee47e66915e9e9477d
    Co-Authored-By: David Ames <email address hidden>
    Closes-Bug: #1809454
    (cherry picked from commit 47b7c4f3cc582bf463fd0c796df84736a0074f48)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=accef50f9648dc40f1a6f457f83f5359e9dd2a24
Submitter: Zuul
Branch: stable/queens

commit accef50f9648dc40f1a6f457f83f5359e9dd2a24
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500

    Ensure rbd auth fallback uses matching credentials

    As of Ocata, cinder config is preferred for rbd auth values with a
    fallback to nova values [1]. The fallback path, for the case when
    rbd_user is configured in cinder.conf and rbd_secret_uuid is not
    configured in cinder.conf, results in the mismatched use of cinder
    rbd_user with nova rbd_secret_uuid.

    This fixes that fallback path to use nova rbd_user from nova.conf
    with rbd_secret_uuid from nova.conf.

    [1] See commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3

    Thanks to David Ames for this fix.

    Change-Id: Ieba216275c07ab16414065ee47e66915e9e9477d
    Co-Authored-By: David Ames <email address hidden>
    Closes-Bug: #1809454
    (cherry picked from commit 47b7c4f3cc582bf463fd0c796df84736a0074f48)
    (cherry picked from commit f5d8ee1bfc3b7b9f1a25f85b42e207db0c9f4b04)

[Corey Bryant (corey.bryant)]

New packages have been uploaded to Ubuntu disco, cosmic and bionic with patches cherry-picked from upstream nova. cosmic and bionic are awaiting SRU review.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:19.0.0~b1~git2018120609.c9dca64fa6-0ubuntu2

---------------
nova (2:19.0.0~b1~git2018120609.c9dca64fa6-0ubuntu2) disco; urgency=medium

  * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
    picked from upstream to ensure ceph backend continues to work for upgrades
    from pre-Ocata (LP: #1809454).

 -- Corey Bryant <email address hidden> Fri, 21 Dec 2018 09:20:12 -0500

[Brian Murray (brian-murray)]

Hello Corey, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.0.3-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Corey, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.7-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted nova into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted nova into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627011
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a7e25aa3d2088e2726988c03e84b3b5ea47bfb7e
Submitter: Zuul
Branch: stable/pike

commit a7e25aa3d2088e2726988c03e84b3b5ea47bfb7e
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500

    Ensure rbd auth fallback uses matching credentials

    As of Ocata, cinder config is preferred for rbd auth values with a
    fallback to nova values [1]. The fallback path, for the case when
    rbd_user is configured in cinder.conf and rbd_secret_uuid is not
    configured in cinder.conf, results in the mismatched use of cinder
    rbd_user with nova rbd_secret_uuid.

    This fixes that fallback path to use nova rbd_user from nova.conf
    with rbd_secret_uuid from nova.conf.

    [1] See commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3

    Thanks to David Ames for this fix.

    Change-Id: Ieba216275c07ab16414065ee47e66915e9e9477d
    Co-Authored-By: David Ames <email address hidden>
    Closes-Bug: #1809454
    (cherry picked from commit 47b7c4f3cc582bf463fd0c796df84736a0074f48)
    (cherry picked from commit f5d8ee1bfc3b7b9f1a25f85b42e207db0c9f4b04)
    (cherry picked from commit accef50f9648dc40f1a6f457f83f5359e9dd2a24)

[Xav Paice (xavpaice)]

How do we go about getting this moving forward from cloud-archive:queens-proposed to stable so we can run this in production?

[David Ames (thedac)]

The newton-proposed, ocata-proposed, pike-proposed, queens-proposed, bionic-proposed and rocky-proposed packages have all been tested.

Newton to pike upgrades were performed. The bug no longer exists. The fix is verified in the packages.

[David Ames (thedac)]

Verified on cosmic.

[David Ames (thedac)]

Adding a bit more context.

The original break only occurred with instances launched on Newton with a subsequent upgrade to Ocata.
The required fix needs to be in every Ubuntu/OpenStack combination we support from xenial-ocata to cosmic-rocky.

I tested the upgrade from xenail-newton to xenial-ocata.

I have also tested that no regressions occur with deployments from xenial-pike to cosmic-rocky.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:18.0.3-0ubuntu2

---------------
nova (2:18.0.3-0ubuntu2) cosmic; urgency=medium

  * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
    picked from upstream to ensure ceph backend continues to work for upgrades
    from pre-Ocata (LP: #1809454).

 -- Corey Bryant <email address hidden> Mon, 07 Jan 2019 14:49:48 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 2:18.0.3-0ubuntu2~cloud0
---------------

 nova (2:18.0.3-0ubuntu2~cloud0) bionic-rocky; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:18.0.3-0ubuntu2) cosmic; urgency=medium
 .
   * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
     picked from upstream to ensure ceph backend continues to work for upgrades
     from pre-Ocata (LP: #1809454).

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:17.0.7-0ubuntu2

---------------
nova (2:17.0.7-0ubuntu2) bionic; urgency=medium

  * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
    picked from upstream to ensure ceph backend continues to work for upgrades
    from pre-Ocata (LP: #1809454).

 -- Corey Bryant <email address hidden> Mon, 07 Jan 2019 14:54:42 -0500

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 2:17.0.7-0ubuntu2~cloud0
---------------

 nova (2:17.0.7-0ubuntu2~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:17.0.7-0ubuntu2) bionic; urgency=medium
 .
   * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
     picked from upstream to ensure ceph backend continues to work for upgrades
     from pre-Ocata (LP: #1809454).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 2:16.1.6-0ubuntu1~cloud1
---------------

 nova (2:16.1.6-0ubuntu1~cloud1) xenial-pike; urgency=medium
 .
   * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
     picked from upstream to ensure ceph backend continues to work for upgrades
     from pre-Ocata (LP: #1809454).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package nova - 2:15.1.5-0ubuntu1~cloud1
---------------

 nova (2:15.1.5-0ubuntu1~cloud1) xenial-ocata; urgency=medium
 .
   * d/p/ensure-rbd-auth-fallback-uses-matching-credentials.patch: Cherry-
     picked from upstream to ensure ceph backend continues to work for upgrades
     from pre-Ocata (LP: #1809454).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627012
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=779e58e97612d9fcb15d8a926fa8e9ddcf88feba
Submitter: Zuul
Branch: stable/ocata

commit 779e58e97612d9fcb15d8a926fa8e9ddcf88feba
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500

    Ensure rbd auth fallback uses matching credentials

    As of Ocata, cinder config is preferred for rbd auth values with a
    fallback to nova values [1]. The fallback path, for the case when
    rbd_user is configured in cinder.conf and rbd_secret_uuid is not
    configured in cinder.conf, results in the mismatched use of cinder
    rbd_user with nova rbd_secret_uuid.

    This fixes that fallback path to use nova rbd_user from nova.conf
    with rbd_secret_uuid from nova.conf.

    [1] See commit f2d27f6a8afb62815fb6a885bd4f8ae4ed287fd3

    Thanks to David Ames for this fix.

    Change-Id: Ieba216275c07ab16414065ee47e66915e9e9477d
    Co-Authored-By: David Ames <email address hidden>
    Closes-Bug: #1809454
    (cherry picked from commit 47b7c4f3cc582bf463fd0c796df84736a0074f48)
    (cherry picked from commit f5d8ee1bfc3b7b9f1a25f85b42e207db0c9f4b04)
    (cherry picked from commit accef50f9648dc40f1a6f457f83f5359e9dd2a24)
    (cherry picked from commit a7e25aa3d2088e2726988c03e84b3b5ea47bfb7e)

[Lee Yarwood (lyarwood)]

Released in 17.0.10.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.8 release.

[Corey Bryant (corey.bryant)]

For some reason this wasn't automatically marked "Fix Released" for upstream stable/pike. It's available in the 16.1.8 release for nova. https://opendev.org/openstack/nova/commit/779e58e97612d9fcb15d8a926fa8e9ddcf88feba

[Corey Bryant (corey.bryant)]

This was fix-released in ocata via nova 2:15.1.5-0ubuntu1~cloud1.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova ocata-eol release.