VM reboot issues

[Aravindh]

Problem Statement: Creating VM from image work fine, but reboot of that instance from openstack results in Error state.

Steps to re-create:

1. Create a VM from Openstack with Image as the source. VM is created successfully.
2. Now issue a openstack server reboot <VM>
3. Softreboot fails after sometime, and hard reboot is attempted, but it fails and moves the instance to an Error state.

Observations.

1. Weird thing I noticed is, when the instance is being created the nova log shows this (Note the VOLUME ID)

2024-01-19 06:45:06.536 7 INFO nova.virt.block_device [req-da372940-2784-4aba-881f-7636460efb46 req-bffdffd6-7521-4450-9ab9-93bac32789e1 a828b4ad4d794e18ac9c6238e893522d 1f4d24639d564e40816d90be4cac8ecd - - default default] [instance: 330922ac-8333-4ee7-a634-0075a00f1fd7] Booting with volume-backed-image 0e1aa7ba-23cc-4ced-8aa2-c9f78535d435 at /dev/vda

2. The create volume (c22d487d-ab7a-4373-b69f-91cd02f32cc8) in openstack does not match this ID at all.

3. openstack server show VM shows the correct volume ID as in 2.

4. During reboot issued though openstack, the multipath just fails and nova logs are stuck at cleaning stale cinder volumes.

5. Reboot command from inside the VM work fine. (sudo reboot)

----

6. When I use volume as the source to create instance, I have none of these problems. Nova log shows the correct volume ID during the creation phase and reboot works great with no issues.

I can consistently reproduce this error.

Host OS - Ubuntu 22.04
Deployment - Kolla Ansible with Docker containers
Storage - Netapp with iSCSI
Hypervisor - KVM
Nova Version - 18.4.0

[Aravindh (highlow7)]

This seems to affect on Bobcat. Antelope works fine.

I tried to update os-brick package to the latest version, and tried to downgrade to the same version as on antelope (6.2.0), the issue persisted on both occasions.

[Amit Uniyal (auniyal)]

I could not reproduce the main issue - i.e. create server from image; reboot, and on reboot server went to error state, I tested this in stable/2023.2 bobcat.

but as from observation if somehow server did created from volume instead of image (which is another bug), server can go to an error state as per this bug https://bugs.launchpad.net/nova/+bug/2048184

the bug is fixed in the master and is yet to be backported in Bobcat.

[Amit Uniyal (auniyal)]

as you are able reproduce it, can please provide full compute logs for vm creation and reboot.
this will help us to understand why vm created from volume instead of image.

[Aravindh (highlow7)]

Is there any specific log you need? I can provide you everything related to
Nova, is it alright?

On Sun, Jan 28, 2024, 3:05 PM Amit Uniyal <email address hidden>
wrote:

> as you are able reproduce it, can please provide full compute logs for vm
> creation and reboot.
> this will help us to understand why vm created from volume instead of
> image.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2049852
>
> Title:
> VM reboot issues
>
> Status in OpenStack Compute (nova):
> New
>
> Bug description:
> Problem Statement: Creating VM from image work fine, but reboot of
> that instance from openstack results in Error state.
>
>
> Steps to re-create:
>
> 1. Create a VM from Openstack with Image as the source. VM is created
> successfully.
> 2. Now issue a openstack server reboot <VM>
> 3. Softreboot fails after sometime, and hard reboot is attempted, but it
> fails and moves the instance to an Error state.
>
>
> Observations.
>
> 1. Weird thing I noticed is, when the instance is being created the
> nova log shows this (Note the VOLUME ID)
>
> 2024-01-19 06:45:06.536 7 INFO nova.virt.block_device [req-
> da372940-2784-4aba-881f-7636460efb46 req-
> bffdffd6-7521-4450-9ab9-93bac32789e1 a828b4ad4d794e18ac9c6238e893522d
> 1f4d24639d564e40816d90be4cac8ecd - - default default] [instance:
> 330922ac-8333-4ee7-a634-0075a00f1fd7] Booting with volume-backed-image
> 0e1aa7ba-23cc-4ced-8aa2-c9f78535d435 at /dev/vda
>
> 2. The create volume (c22d487d-ab7a-4373-b69f-91cd02f32cc8) in
> openstack does not match this ID at all.
>
> 3. openstack server show VM shows the correct volume ID as in 2.
>
> 4. During reboot issued though openstack, the multipath just fails and
> nova logs are stuck at cleaning stale cinder volumes.
>
> 5. Reboot command from inside the VM work fine. (sudo reboot)
>
> ----
>
> 6. When I use volume as the source to create instance, I have none of
> these problems. Nova log shows the correct volume ID during the
> creation phase and reboot works great with no issues.
>
>
> I can consistently reproduce this error.
>
>
> Host OS - Ubuntu 22.04
> Deployment - Kolla Ansible with Docker containers
> Storage - Netapp with iSCSI
> Hypervisor - KVM
> Nova Version - 18.4.0
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/2049852/+subscriptions
>
>