[rbd] VM created from snapshot becomes a child of that RBD image

[Marius L]

When I create an instance from a snapshot, the underlying RBD image of the new instance is actually a child of the snapshot in RBD.
Then I can't delete that snapshot because in Ceph it has children and is protected, thus Glance will throw this error: "Image <ID> could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance."

Steps to reproduce:
1. Create a server
    $ openstack server create server-1

2. Create a snapshot from that server
    $ openstack server image create $SERVER_1_ID

3. Create a new server from the snapshot
    $ openstack server create --image $SNAPSHOT_ID server-2

4. Try to delete the snapshot, it will throw the above error from Glance (it is in use through the backend store outside of Glance.)
    $ openstack image delete $SNAPSHOT_ID

5. Check the snapshot image in RBD, it will have the new instance's disk as children.
    $ rbd children images/$SNAPSHOT_ID
    > vms/<server-2>_disk

This doesn't happen when creating instances from images uploaded through glance, but only when using images created with `openstack server image create`

Environment
- Openstack Xena (Nova 17.6.0)
- Libvirt with KVM virt_type
- RBD for ephemeral storage
- Ceph pacific

[Marius L (marius-leus)]

Up.

[sean mooney (sean-k-mooney)]

this is a known limitation of ceph backed glance images.
a snapshot is just a special case of this.
it not a nova bug as this is expected to happen when we do a shallow clone.

there have been some feature request in the past tho allow breaking this chain but to me this is not a bug. we will discuss this in the nova meeting in the open discussion section shortly.

so i will not update this yet.

[sean mooney (sean-k-mooney)]

based on our irc meeting we determin we do not have enough info to fully triage this form our side since we don't know the image
types that are in use when you careted the VM and snapshots vs uploaded an image directly.

i suspect the issue is that the snapstho is a shallow image create by a direct snapshot in ceph and
we are doing a shallow copy of that when we boot the new image.

that is what we would expect and is a performance optimization that may operator depend on.
so this does not appear to be a bug form a nova perspective but is perhaps a feature request.

like the ability to flaten on snapshot or simialr.

[Marius L (marius-leus)]

Yes, this is a feature request then.

Nova needs an option flatten RBD snapshots, similar to how cinder does:

https://opendev.org/openstack/cinder/src/commit/7a312ab1a847ff55728cca677eaee11518f6f920/cinder/volume/drivers/rbd.py#L78-L81

On Tue, May 17, 2022 at 8:12 PM sean mooney <email address hidden>
wrote:

> based on our irc meeting we determin we do not have enough info to fully
> triage this form our side since we don't know the image
> types that are in use when you careted the VM and snapshots vs uploaded an
> image directly.
>
>
> i suspect the issue is that the snapstho is a shallow image create by a
> direct snapshot in ceph and
> we are doing a shallow copy of that when we boot the new image.
>
> that is what we would expect and is a performance optimization that may
> operator depend on.
> so this does not appear to be a bug form a nova perspective but is perhaps
> a feature request.
>
> like the ability to flaten on snapshot or simialr.
>
> ** Also affects: glance
> Importance: Undecided
> Status: New
>
> ** Changed in: nova
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1959186
>
> Title:
> [rbd] VM created from snapshot becomes a child of that RBD image
>
> Status in Glance:
> New
> Status in OpenStack Compute (nova):
> Invalid
>
> Bug description:
> When I create an instance from a snapshot, the underlying RBD image of
> the new instance is actually a child of the snapshot in RBD.
> Then I can't delete that snapshot because in Ceph it has children and is
> protected, thus Glance will throw this error: "Image <ID> could not be
> deleted because it is in use: The image cannot be deleted because it is in
> use through the backend store outside of Glance."
>
> Steps to reproduce:
> 1. Create a server
> $ openstack server create server-1
>
> 2. Create a snapshot from that server
> $ openstack server image create $SERVER_1_ID
>
> 3. Create a new server from the snapshot
> $ openstack server create --image $SNAPSHOT_ID server-2
>
> 4. Try to delete the snapshot, it will throw the above error from Glance
> (it is in use through the backend store outside of Glance.)
> $ openstack image delete $SNAPSHOT_ID
>
> 5. Check the snapshot image in RBD, it will have the new instance's disk
> as children.
> $ rbd children images/$SNAPSHOT_ID
> > vms/<server-2>_disk
>
> This doesn't happen when creating instances from images uploaded
> through glance, but only when using images created with `openstack
> server image create`
>
> Environment
> - Openstack Xena (Nova 17.6.0)
> - Libvirt with KVM virt_type
> - RBD for ephemeral storage
> - Ceph pacific
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance/+bug/1959186/+subscriptions
>
>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/884595

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Michel Nederlof <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/884595
Reason: Wait for glance team to implement the rbd trash feature, which is better.

[Michel Nederlof (mnederlof)]

Glance rbd trash feature: https://review.opendev.org/c/openstack/glance_store/+/884524

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 4.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 4.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 4.7.0 release.