NFS volume snapshot does not update volume attachment format to qcow2

[melanie witt]

This was reported downstream when a customer hit the issue [1].

They found that after performing a volume snapshot on a stopped instance, that instance could no longer boot.

The instance XML had the correct disk format "qcow2" after the snapshot while it was still stopped. However, when someone then tries to start the instance, the regenerated XML contains the wrong disk format "raw" (nova regenerates fresh XML on hard reboot, boot from shutoff) and that prevents the instance from booting.

While tracing the issue I noticed that after the snapshot, the volume attachment still showed the format "raw". It should have been "qcow2" after the snapshot, I think.

The customer found that rebuilding the instance resulted in a booting instance despite the underlying volume (snapshot) being the same. Rebuild generates a new volume attachment and that volume attachment reflected the correct format "qcow2".

Based on this, it appeared to me that volume attachment format was not being updated from "raw" to "qcow2" after the volume snapshot was created as qcow2 and was made the active volume for the instance.

AFAICT, there is not a way for nova to update the volume attachment format.

Repro steps:

  $ openstack volume create --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --size 1 --bootable my_volume

  $ openstack server create --flavor c1 --volume my_volume --network private --wait my_server

  $ openstack server stop d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance has disk format "raw" and volume attachment also shows format "raw")
  (To check this:

    $ virsh dumpxml d97a6178-112d-487b-ba21-1af2a4a78456

    $ openstack --os-volume-api-version 3.27 volume attachment show 2f43581f-ba30-423e-ba5f-7b0ac7768fa5 -f json
  )

  $ openstack volume snapshot create --volume d5c71e24-46ac-49be-bf33-667598b6058b --force my_volume_snapshot

  (Instance has correct disk format "qcow2" at this point, volume attachment shows format "raw")

  $ openstack server start d97a6178-112d-487b-ba21-1af2a4a78456

  (Instance now has incorrect disk format "raw" and volume attachment still shows format "raw")

To see the effect of a rebuild:

  $ openstack server rebuild d97a6178-112d-487b-ba21-1af2a4a78456 --image f8e5b23e-b1d3-44e8-bc71-ac7aca4f6f8e --wait

  (Instance has disk format "qcow2" and volume attachment also shows format "qcow2")
  (Volume attachment is newly created with new ID etc)

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2118150

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/857528

[melanie witt (melwitt)]

Hi, just FYI I am already working on this, I spent a LOT of time figuring out what's going on here this week and last :)

And I've proposed https://review.opendev.org/c/openstack/cinder/+/857528 yesterday (shown in comment #1).

I am going to assign this bug to myself to try to prevent confusion.

If my approach in the above proposed patch is a completely wrong direction, please let me know and I would abandon it.

[Sofia Enriquez (lsofia-enriquez)]

Sure! Sorry about assigning me the bug. I'll review your patch. Thank you so much for working on this.

[Sofia Enriquez (lsofia-enriquez)]

I just want to confirm that I've reproduced this bug. I'll review the patch proposed.

[melanie witt (melwitt)]

I've added the nova project to this bug in case it is more appropriate to address it there.

I'm going to try a nova patch as well to see if that might be the more proper way to fix this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/858836
Reason: We talked about this problem with the cinder team at the PTG and the fix is going to be in cinder: https://review.opendev.org/c/openstack/cinder/+/857528