[SRU] Key-pair is not updated during the rebuild

[Quentin GROLLEAU]

[Impact]

During rebuilds, the customer was unable to update the instance's keypair.

[Test Case]

- create a bionic openstack test env

- choose the key 'testkey' to create an instance

openstack keypair create mykey --public-key ~/.ssh/id_rsa.pub
openstack keypair create testkey --public-key /home/ubuntu/testkey.pub
openstack server create --flavor m1.small --image jammy --key-name testkey --network=$(openstack network show private -f value -c id) i1

- create a new instance from the snapshot and choose a different keypair 'mykey' at rebuild time

openstack --os-compute-api-version 2.54 server rebuild --image jammy --key-name mykey --name i1 i1
sudo ip netns exec qrouter-xxx ssh ubuntu@192.168.21.4 -i ~/testkey.priv -v
sudo ip netns exec qrouter-xxx ssh ubuntu@192.168.21.4 -i ~/id_rsa -v

the new instance should accept the new key and reject the old key, but the result is the new instance rejects the new key but old key still works.

[Regression Potential]

This fix 6a7a78a44 is already in stable/queens and all versions since queens, bionic uses 17.0.13 rather than stable/queens, we just SRU this fix to 17.0.13 so there can't be any regression theoretically. On the other hand, code change is limited to _save_keypairs according to https://review.opendev.org/c/openstack/nova/+/683043/19/nova/objects/instance.py so the regressions is also limited in _save_keypairs . The test will also ensure that other logic beyond _save_keypairs. I have tested this fix, it works. so I think it's safe.

[Others]

Original Bug Description Below
===========

When we want to rebuild an instance and change the keypair we can specified it with :
openstack --os-compute-api-version 2.54 server rebuild --image "Debian 10" --key-name key1 instance1

This comes from this implementation :
https://review.opendev.org/#/c/379128/
https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/rebuild-keypair-reset.html

But when rebuilding the instance, Cloud-Init will set the key in authorized_keys from
http://169.254.169.254/openstack/latest/meta_data.json

And this meta_data.json uses the keys from instance_extra tables
But the keypair will be updated in the 'instances' table but not in the 'instance_extra' table.

So the keypair is not updated inside the VM

May be this is the function for saving the keypair, but the save() do nothing :
https://opendev.org/openstack/nova/src/branch/master/nova/objects/instance.py#L714

Steps to reproduce
==================

- Deploy a DevStack
- Boot an instance with keypair key1
- Rebuild it with key2
- A nova show will show the key_name key2, keypairs object in table instance_extra is not updated and you cannot connect with key2 to the instance

Expected result
===============
Connecte to the Vm with the new keypair added during the rebuild call

Actual result
=============
The keypair added during the rebuild call is not set in the VM

Environment
===========
I tested it on a Devstack from master and we have the behaviour.
NOVA : commit 5fa49cd0b8b6015aa61b4312b2ce1ae780c42c64

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/683043

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/738583

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.opendev.org/738583

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/683043
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=086796021b189c3ac64805ed8f6bde833906d284
Submitter: Zuul
Branch: master

commit 086796021b189c3ac64805ed8f6bde833906d284
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742631

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/742709

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742631
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aed86ee5d6289edf1baf9fe0b2a9e509031fdd25
Submitter: Zuul
Branch: stable/ussuri

commit aed86ee5d6289edf1baf9fe0b2a9e509031fdd25
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>
    (cherry picked from commit 086796021b189c3ac64805ed8f6bde833906d284)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/743264

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/743268

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/743271

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742709
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b971dc82cb524fe86284c95ec671e2bad1c2874f
Submitter: Zuul
Branch: stable/train

commit b971dc82cb524fe86284c95ec671e2bad1c2874f
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Conflicts:
            nova/tests/unit/fake_instance.py

    NOTE(stephenfin): Conflicts are due to change
    I44ad826f0edb39d770bb3201c675dff78154cbf3 ("partial support for live
    migration with specific resources"), which initialized the
    'migration_context' attribute of the instance created as part of the
    'fake_instance_obj' function.

    Changes:
            nova/tests/functional/regressions/test_bug_1843708.py

    NOTE(stephenfin): The 'IntegratedTestBase' and 'InstanceHelperMixin'
    base classes for functional tests are not fully integrated yet in
    stable/train, which requires reinventing the wheel somewhat.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>
    (cherry picked from commit 086796021b189c3ac64805ed8f6bde833906d284)
    (cherry picked from commit aed86ee5d6289edf1baf9fe0b2a9e509031fdd25)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743264
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0bc5a4ecb524a73aacb5d0dd2887799885bdbb14
Submitter: Zuul
Branch: stable/stein

commit 0bc5a4ecb524a73aacb5d0dd2887799885bdbb14
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Conflicts:
            nova/tests/unit/fake_instance.py
            nova/tests/unit/objects/test_instance.py

    NOTE(stephenfin): Conflicts in 'fake_instance.py' are due to change
    If7f48933db10fcca3b9a05e1e978dfc51f6dabd0 ("Claim resources in resource
    tracker"), which is related to the vPMEM work and shouldn't be
    backported, while the conflicts in 'test_instance.py' are due to change
    Ic89352a9900515484bffe961475feb1cefc6b2a9 ("Remove
    'instance_update_at_top', 'instance_destroy_at_top'") which removed some
    cells v1 tests but shouldn't be removed here where cells v1 is
    technically still a thing.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>
    (cherry picked from commit 086796021b189c3ac64805ed8f6bde833906d284)
    (cherry picked from commit aed86ee5d6289edf1baf9fe0b2a9e509031fdd25)
    (cherry picked from commit b971dc82cb524fe86284c95ec671e2bad1c2874f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aa7a6939d5177c0dd8c9f5a7bf7975264d2f5a2a
Submitter: Zuul
Branch: stable/rocky

commit aa7a6939d5177c0dd8c9f5a7bf7975264d2f5a2a
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Changes:
            nova/tests/functional/regressions/test_bug_1843708.py

    NOTE(stephenfin): Changes are necessary since 'PlacementFixture' moved
    in stable/stein with change Idaed39629095f86d24a54334c699a26c218c6593
    but it's still in the old place here.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>
    (cherry picked from commit 086796021b189c3ac64805ed8f6bde833906d284)
    (cherry picked from commit aed86ee5d6289edf1baf9fe0b2a9e509031fdd25)
    (cherry picked from commit b971dc82cb524fe86284c95ec671e2bad1c2874f)
    (cherry picked from commit 0bc5a4ecb524a73aacb5d0dd2887799885bdbb14)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743271
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6a7a78a44ea19497ea3e331ddd672f95cd49c50e
Submitter: Zuul
Branch: stable/queens

commit 6a7a78a44ea19497ea3e331ddd672f95cd49c50e
Author: Takashi NATSUME <email address hidden>
Date: Thu Sep 19 15:57:44 2019 +0900

    objects: Update keypairs when saving an instance

    The keypair of a server is updated when rebuilding the server with a
    keypair. This function has been added since API microversion 2.54.
    However the 'keypairs' of the instance object is not saved when saving
    the instance object currently.

    Make the instance object update the 'keypairs' field when saving the
    instance object.

    Change-Id: I8a2726b39d0444de8c35480024078a97430f5d0c
    Closes-Bug: #1843708
    Co-authored-by: Stephen Finucane <email address hidden>
    (cherry picked from commit 086796021b189c3ac64805ed8f6bde833906d284)
    (cherry picked from commit aed86ee5d6289edf1baf9fe0b2a9e509031fdd25)
    (cherry picked from commit b971dc82cb524fe86284c95ec671e2bad1c2874f)
    (cherry picked from commit 0bc5a4ecb524a73aacb5d0dd2887799885bdbb14)
    (cherry picked from commit aa7a6939d5177c0dd8c9f5a7bf7975264d2f5a2a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova rocky-eol release.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "bionic.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Thank you Hua. Uploaded to the bionic unappproved queue: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=nova

[Steve Langasek (vorlon)]

"see the original bug description below" - please update the bug description to use the standard template so the SRU team does not have to dig for this information and guess what you think is relevant.

[Hua Zhang (zhhuabj)]

hi @vorlon, I have updated the bug description to remove the part "see the original bug description below", thanks

[Łukasz Zemczak (sil2100)]

An upload of nova to bionic-proposed has been rejected from the upload queue for the following reason: "SRU reverts a newer security update (2:17.0.13-0ubuntu5.3). Please re-submit after re-basing on the latest package version in -security and -updates.".

[Łukasz Zemczak (sil2100)]

Also, before I actually accept this into -proposed (after a new rebased version is uploaded), a quick question about the actual SRU. I don't have enough context as I am not a huge nova user and I totally agree this is a bug, but I just wanted to make sure if, by any chance, the current behaviour couldn't now be the 'expected behaviour' on bionic systems?

[Trent Lloyd (lathiat)]

The original debdiff was correctly based on 2:17.0.13-0ubuntu5.3, but during the sponsored upload it was applied against 2:17.0.13-0ubuntu5.2 instead.

Needs a sponsor to re-upload correctly applied on top of 2:17.0.13-0ubuntu5.3

Reference: http://launchpadlibrarian.net/666459685/nova_2%3A17.0.13-0ubuntu5_2%3A17.0.13-0ubuntu6.diff.gz

[Trent Lloyd (lathiat)]

> a quick question about the actual SRU. I don't have enough context as I am not a huge nova user and I totally agree this is a bug, but I just wanted to make sure if, by any chance, the current behaviour couldn't now be the 'expected behaviour' on bionic systems?

When executing the "openstack server rebuild" command (which, generally, destroys the current VM disk and rebuilds it from the image), the user is explicitly specifying this new key on the commandline. In the current version, this request is ignored.

This parameter is optional, so I think it should be reasonably safe to assume that if the user really wanted to use the old key, they would simply not specify a new key and would leave it absent instead.

While in theory someone could have accidentally relied on this behaviour, it seems highly unlikely to me and if they did, it would be having explicitly asked for the behaviour they will now get. So this feels OK to me.

[Hua Zhang (zhhuabj)]

Hi Łukasz, after rebuilding a VM with different keypairs on bionic, we can use the old keypair for ssh login, but we cannot use the new keypair. This is not 'expected behaviour' for nova rebuild. bionic uses 17.0.13, which has this issue.

[Dariusz Gadomski (dgadomski)]

Based on comments #25 and #26 I have reuploaded the patch rebased on top of the security update.

[Łukasz Zemczak (sil2100)]

Thank you for all the comments and for the re-upload. This feels like the way to go.

[Łukasz Zemczak (sil2100)]

Hello Quentin, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.13-0ubuntu5.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Hua Zhang (zhhuabj)]

I test nova/2:17.0.13-0ubuntu5.4, it works, pls see https://paste.ubuntu.com/p/76Nf9VYzVJ/

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:17.0.13-0ubuntu5.4

---------------
nova (2:17.0.13-0ubuntu5.4) bionic; urgency=medium

  * Update keypairs when saving an instance (LP: #1843708)
    - d/p/objects-Update-keypairs-when-saving-an-instance.patch

 -- Zhang Hua <email address hidden> Thu, 27 Apr 2023 04:21:53 +0800

[Robie Basak (racb)]

The verification of the Stable Release Update for nova has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.