Cinder

Fail to migrate, resize, evacuate VM with volume state backing-up

Bug #2034677 reported by Sang Tran on 2023-09-07

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	Cinder	In Progress	Undecided	Sang Tran

Bug Description

Environment: OpenStack Yoga version

Reproduce steps:
1. Backup VM volume with cinder-backup, volume transit to state Backing-up
2. Do one of the following actions: Live migrate, cold migrate, resize, evacuate VM.
3. VM returns ERROR status with logs below

Expect result:
- Action should succeed or at least the VM evacuation process.

2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server [req-d2867ed1-6372-457c-8114-4eb3831ee3fd d3802c30ff814feaa3197d29e8a3a731 4bce93cd1dd34919961adce9957d4a24 - default default] Exception during message handling: nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 830dbeeb-df51-4724-ace4-86564eb092fc status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-7bb4e5e8-b48c-463a-a3c1-528111c651c3)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 397, in wrapper
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 433, in wrapper
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 803, in attachment_create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server LOG.error('Create attachment failed for volume '
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise value
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 795, in attachment_create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server attachment_ref = cinderclient(context, '3.44').attachments.create(
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 38, in create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment')
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/base.py", line 302, in _create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 218, in post
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 206, in _cs_request
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 192, in request
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.BadRequest: Invalid volume: Volume 830dbeeb-df51-4724-ace4-86564eb092fc status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-7bb4e5e8-b48c-463a-a3c1-528111c651c3)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise value
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise value
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 8141, in pre_live_migration
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server bdm.save()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise value
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 8084, in pre_live_migration
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server attach_ref = self.volume_api.attachment_create(
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 405, in wrapper
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server _reraise(exception.InvalidInput(reason=err_msg))
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 483, in _reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise desired_exc.with_traceback(sys.exc_info()[2])
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 397, in wrapper
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 433, in wrapper
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 803, in attachment_create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server LOG.error('Create attachment failed for volume '
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise value
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 795, in attachment_create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server attachment_ref = cinderclient(context, '3.44').attachments.create(
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 38, in create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server retval = self._create('/attachments', body, 'attachment')
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/base.py", line 302, in _create
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 218, in post
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 206, in _cs_request
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 192, in request
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 830dbeeb-df51-4724-ace4-86564eb092fc status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-7bb4e5e8-b48c-463a-a3c1-528111c651c3)
2023-08-28 12:11:17.142 7 ERROR oslo_messaging.rpc.server

Revision history for this message

Jean Pierre Roquesalane (jproque15130) wrote on 2023-09-13:

Hello,

what kind of backend do you use for cinder-backup?

Revision history for this message

Sang Tran (sangtq8) wrote on 2023-09-14:

Dear Jean,

I use cinder-backup with cinder.backup.drivers.nfs.NFSBackupDriver driver.

Revision history for this message

Jean Pierre Roquesalane (jproque15130) wrote on 2023-09-14:

Thank you.

When you create your backup can you confirm it's marked as available or it remains stuck in creating state?

openstack volume backup list

Can you also provide the logs from cinder-backup service?

Revision history for this message

Sang Tran (sangtq8) wrote on 2023-09-15:

Download full text (6.4 KiB)

Hi Jean,

Yes, the state of backup is "Creating", time to transition to available depends on the actual usage size of the volume.

The volume status while waiting for finishing backup

Hi Jean,

Yes, the state of backup is "Creating", time to transition to available depends on the actual usage size of the volume.

The volume status while waiting for finishing backup

+--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
|             Field              |                                                                 Value                                                                  |
+--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| attachments                    | [{'id': '6ce07797-9e44-43f3-aae5-775c9c08747e', 'attachment_id': '55163a8e-c00a-478d-b363-575700e576ce', 'volume_id':                  |
|                                | '6ce07797-9e44-43f3-aae5-775c9c08747e', 'server_id': '5fc40d0a-95c3-4600-8430-f6501ff7fb5b', 'host_name': '', 'device': |
|                                | '/dev/vda', 'attached_at': '2023-09-06T17:40:26.000000'}]                                                                              |
| availability_zone              | nova                                                                                                                                   |
| bootable                       | true                                                                                                                                   |
| consistencygroup_id            | None                                                                                                                                   |
| created_at                     | 2023-09-06T17:24:20.000000                                                                                                             |
| description                    |                                                                                                                                        |
| encrypted                      | False                                                                                                                                  |
| id                             | 6ce07797-9e44-43f3-aae5-775c9c08747e                                                                                                   |
| migration_status               | None                                                                                                                                   |
| multiattach                    | False                                                                                                                                  |
| name                           |                                                                                                                                        |                                                                              
| os-vol-mig-status-attr:migstat | None                                                                                                                                   |
| os-vol-mig-status-attr:name_id | None                                                                                                                                   |
| os-vol-tenant-attr:tenant_id   | 2a60631b7920497b84ff86550d826ef5                                                                                                       |
| properties                     |                                                                                                                                        |
| replication_status             | None                                                                                                                                   |
| size                           | 5                                                                                                                                      |
| snapshot_id                    | None                                                                                                                                   |
| source_volid                   | None                                                                                                                                   |
| status                         | backing-up                                                                                                                             |
| type                           | Premium-SSD                                                                                                                            |
| updated_at                     | 2023-09-15T02:21:42.000000                                                                                                             |
| user_id                        | 3358468ef73546d5965c39e495c78df9                                                                                                       |
| volume_image_metadata          | {'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.0', 'owner_specified.openstack.sha256':   |
|                                | '', 'image_id': 'd24e9503-7e93-4b04-b0e4-42f4ef3ee10c', 'image_name': 'cirros-0.5.0', 'checksum': '7734eb3945297adc90ddc6cebe8bb082',  |
|                                | 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '1', 'min_ram': '0', 'size': '16344576'}                               |
+--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+

In addition, there no log error for the cinder-backup accordingly.

Revision history for this message

Jean Pierre Roquesalane (jproque15130) wrote on 2023-09-15:

Hello,

as long as the backup is not finished yet, the source volume will remain in backing-up status.

Therefore and as a result of the current design, it's perfectly normal that any VM operation which implies the volume fails as cinder requires the volume to be in available state.

Revision history for this message

Sang Tran (sangtq8) wrote on 2023-09-15:

Hi Jean,

Thank for your response,
I've checked the logic code from Cinder API which allows state available, in-use, and downloading
https://github.com/openstack/cinder/blob/8796522c992759cba3a8f8f4381730874a933ff9/cinder/volume/api.py#L2393

The backup process will create a clone of the original volume and mount it to cinder-backup instance for performing backup. Its "backing up" state only indicates the current state and has no impact on the volume operation (same as the downloading state). So I think it should be ignored from VM normal operation or at least the evacuation and resize process.

Revision history for this message

Jean Pierre Roquesalane (jproque15130) wrote on 2023-09-20:

Can you provide the commands in order that you ran when you hit the issue?

Revision history for this message

Jean Pierre Roquesalane (jproque15130) wrote on 2023-09-20:

I don't think the function you are looking at is called during your test, we can clearly see that this function is called:

Sep 20 13:58:15 ubuntu2004-devstack nova-compute[1101]: ERROR nova.compute.manager [instance: b678ae50-9218-4234-8bf1-2a91215de34f] File "/opt/stack/nova/nova/volume/cinder.py", line 810, in attachment_create Sep 20 13:58:15 ubuntu2004-devstack nova-compute[1101]: ERROR nova.compute.manager [instance: b678ae50-9218-4234-8bf1-2a91215de34f] LOG.error('Create attachment failed for volume '

Revision history for this message

Sang Tran (sangtq8) wrote on 2023-10-03:

Hi Jean

The command is as below:

cinder backup-create --force <volume>
nova live-migration <server> (volume backup attached to this server)

Sang Tran (sangtq8) on 2023-11-03

Changed in cinder:
assignee:	nobody → Sang Tran (sangtq8)

OpenStack Infra (hudson-openstack) on 2023-11-03

Changed in cinder:
status:	New → In Progress

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.