NFS volume snapshot does not update volume attachment format to qcow2

[melanie witt]

This was reported downstream when a customer hit the issue [1].

They found that after performing a volume snapshot on a stopped instance, that instance could no longer boot.

The instance XML had the correct disk format "qcow2" after the snapshot while it was still stopped. However, when someone then tries to start the instance, the regenerated XML contains the wrong disk format "raw" (nova regenerates fresh XML on hard reboot, boot from shutoff) and that prevents the instance from booting.

While tracing the issue I noticed that after the snapshot, the volume attachment still showed the format "raw". It should have been "qcow2" after the snapshot, I think.

The customer found that rebuilding the instance resulted in a booting instance despite the underlying volume (snapshot) being the same. Rebuild generates a new volume attachment and that volume attachment reflected the correct format "qcow2".

Based on this, it appeared to me that volume attachment format was not being updated from "raw" to "qcow2" after the volume snapshot was created as qcow2 and was made the active volume for the instance.

AFAICT, there is not a way for nova to update the volume attachment format.

Repro steps:

  $ openstack volume create --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --size 1 --bootable my_volume

  $ openstack server create --flavor c1 --volume my_volume --network private --wait my_server

  $ openstack server stop d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance has disk format "raw" and volume attachment also shows format "raw")
  (To check this:

    $ virsh dumpxml d97a6178-112d-487b-ba21-1af2a4a78456

    $ openstack --os-volume-api-version 3.27 volume attachment show 2f43581f-ba30-423e-ba5f-7b0ac7768fa5 -f json
  )

  $ openstack volume snapshot create --volume d5c71e24-46ac-49be-bf33-667598b6058b --force my_volume_snapshot

  (Instance has correct disk format "qcow2" at this point, volume attachment shows format "raw")

  $ openstack server start d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance now has incorrect disk format "raw" and volume attachment still shows format "raw")

To see the effect of a rebuild:

  $ openstack server rebuild d97a6178-112d-487b-ba21-1af2a4a78456 --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --wait

  (Instance has disk format "qcow2" and volume attachment also shows format "qcow2")
  (Volume attachment is newly created with new ID etc)

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2118150

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/857528

[melanie witt (melwitt)]

Hi, just FYI I am already working on this, I spent a LOT of time figuring out what's going on here this week and last :)

And I've proposed https://review.opendev.org/c/openstack/cinder/+/857528 yesterday (shown in comment #1).

I am going to assign this bug to myself to try to prevent confusion.

If my approach in the above proposed patch is a completely wrong direction, please let me know and I would abandon it.

[Sofia Enriquez (lsofia-enriquez)]

Sure! Sorry about assigning me the bug. I'll review your patch. Thank you so much for working on this.

[Sofia Enriquez (lsofia-enriquez)]

I just want to confirm that I've reproduced this bug. I'll review the patch proposed.

[melanie witt (melwitt)]

I've added the nova project to this bug in case it is more appropriate to address it there.

I'm going to try a nova patch as well to see if that might be the more proper way to fix this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836
Reason: We talked about this problem with the cinder team at the PTG and the fix is going to be in cinder: https://review.opendev.org/c/openstack/cinder/+/857528

[melanie witt (melwitt)]

This bug is Invalid for Nova and needs to be fixed in Cinder.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/857528
Committed: https://opendev.org/openstack/cinder/commit/56abc9d5b9a07ca31791e385d93ca5b56b6bd74e
Submitter: "Zuul (22348)"
Branch: master

commit 56abc9d5b9a07ca31791e385d93ca5b56b6bd74e
Author: melanie witt <email address hidden>
Date: Thu Sep 15 00:00:31 2022 +0000

    NFS update volume attachment format during volume snapshot

    During a NFS volume snapshot of an attached volume, a QCOW2 snapshot is
    created and is made the active volume for the instance. The associated
    volume attachment is however not updated, resulting in an unbootable
    instance after a stop/start.

    This adds an update to the volume format and attachment connection_info
    format during the snapshot.

    Closes-Bug: #1989514

    Change-Id: I3c638dab2f0b5a2128eaeeea533450d7cdfa1577

[Luigi Toscano (ltoscano)]

A tempest test to cover this use case was proposed here: https://review.opendev.org/c/openstack/tempest/+/939329

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/cinder/+/939349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/cinder/+/939350

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/cinder/+/939351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/939349
Committed: https://opendev.org/openstack/cinder/commit/9d1a7de850ad06e9f8e242ecfeb070da22c688c4
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 9d1a7de850ad06e9f8e242ecfeb070da22c688c4
Author: melanie witt <email address hidden>
Date: Thu Sep 15 00:00:31 2022 +0000

    NFS update volume attachment format during volume snapshot

    During a NFS volume snapshot of an attached volume, a QCOW2 snapshot is
    created and is made the active volume for the instance. The associated
    volume attachment is however not updated, resulting in an unbootable
    instance after a stop/start.

    This adds an update to the volume format and attachment connection_info
    format during the snapshot.

    Closes-Bug: #1989514

    Change-Id: I3c638dab2f0b5a2128eaeeea533450d7cdfa1577
    (cherry picked from commit 56abc9d5b9a07ca31791e385d93ca5b56b6bd74e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/939351
Committed: https://opendev.org/openstack/cinder/commit/a54442aef0860876db85b29f8df1b2c1245aecd9
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit a54442aef0860876db85b29f8df1b2c1245aecd9
Author: melanie witt <email address hidden>
Date: Thu Sep 15 00:00:31 2022 +0000

    NFS update volume attachment format during volume snapshot

    During a NFS volume snapshot of an attached volume, a QCOW2 snapshot is
    created and is made the active volume for the instance. The associated
    volume attachment is however not updated, resulting in an unbootable
    instance after a stop/start.

    This adds an update to the volume format and attachment connection_info
    format during the snapshot.

    Closes-Bug: #1989514

    Change-Id: I3c638dab2f0b5a2128eaeeea533450d7cdfa1577
    (cherry picked from commit 56abc9d5b9a07ca31791e385d93ca5b56b6bd74e)
    (cherry picked from commit 9d1a7de850ad06e9f8e242ecfeb070da22c688c4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/939350
Committed: https://opendev.org/openstack/cinder/commit/0480073b9ac64f6897fd6bcea72a8d1cc6d474bd
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 0480073b9ac64f6897fd6bcea72a8d1cc6d474bd
Author: melanie witt <email address hidden>
Date: Thu Sep 15 00:00:31 2022 +0000

    NFS update volume attachment format during volume snapshot

    During a NFS volume snapshot of an attached volume, a QCOW2 snapshot is
    created and is made the active volume for the instance. The associated
    volume attachment is however not updated, resulting in an unbootable
    instance after a stop/start.

    This adds an update to the volume format and attachment connection_info
    format during the snapshot.

    Closes-Bug: #1989514

    Change-Id: I3c638dab2f0b5a2128eaeeea533450d7cdfa1577
    (cherry picked from commit 56abc9d5b9a07ca31791e385d93ca5b56b6bd74e)
    (cherry picked from commit 9d1a7de850ad06e9f8e242ecfeb070da22c688c4)
    (cherry picked from commit a54442aef0860876db85b29f8df1b2c1245aecd9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 25.1.0 Dalmatian release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 24.3.0 Caracal release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 23.4.0 Bobcat release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 26.0.0.0rc1 Epoxy release candidate.