kolla-ansible

Cannot delete instance after evacuation

Bug #1891462 reported by Jegor van Opdorp
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Radosław Piliszek
kolla-ansible 11.0.0 "victoria"
Train
Fix Released
Medium
Mark Goddard
Ussuri
Fix Released
Medium
Radosław Piliszek
Victoria
Fix Released
Medium
Radosław Piliszek
kolla-ansible 11.0.0 "victoria"

Bug Description

**Environment**:
* OS: CentOS 8
* Kernel: 4.18.0-147.8.1.el8_1.x86_64
* Docker version: 19.03.12
* Kolla-Ansible version: train/stable
* Docker image Install type (source/binary): source
* Docker image distribution: CentOS 8
* If self built - Kolla version and environment used to build: stable/train, CentOS 8
* Share your inventory file, globals.yml and other configuration files if relevant

I get the following error trying to delete an instance after evacuation:

2020-08-07 16:38:48.226 6 INFO nova.compute.manager [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] [instance: fb5e23be-32e5-4dae-a820-4be37235908e] Destroying instance as it has been evacuated from this host but still exists in the hypervisor
2020-08-07 16:38:49.466 6 INFO nova.virt.libvirt.driver [-] [instance: fb5e23be-32e5-4dae-a820-4be37235908e] Instance destroyed successfully.
2020-08-07 16:38:49.477 6 INFO oslo.privsep.daemon [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp__mpf1x9/privsep.sock']
2020-08-07 16:38:50.201 6 INFO oslo.privsep.daemon [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] Spawned new privsep daemon via rootwrap
2020-08-07 16:38:50.133 61 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-08-07 16:38:50.136 61 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-08-07 16:38:50.138 61 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2020-08-07 16:38:50.139 61 INFO oslo.privsep.daemon [-] privsep daemon running as pid 61
2020-08-07 16:38:50.493 6 INFO os_vif [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:5a:70:65,bridge_name='qbre55f84ee-c8',has_traffic_filtering=True,id=e55f84ee-c81a-443f-8ef2-f96e5d29ff4b,network=Network(ea7550eb-73f1-480f-b5c9-05729684e65f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tape55f84ee-c8')
2020-08-07 16:38:50.497 6 ERROR nova.volume.cinder [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] The [cinder] section of your nova configuration file must be configured for authentication with the block-storage service endpoint.
2020-08-07 16:38:50.497 6 ERROR os_brick.encryptors [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] Failed to retrieve encryption metadata for volume eea07cca-37b5-4d00-a073-712395e8a9a5: Unknown auth type: None (HTTP 401): cinderclient.exceptions.Unauthorized: Unknown auth type: None (HTTP 401)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service [req-21baa3a8-33cd-4114-9abc-df6a75372ec7 - - - - -] Error starting thread.: cinderclient.exceptions.Unauthorized: Unknown auth type: None (HTTP 401)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service Traceback (most recent call last):
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_service/service.py", line 810, in run_service
2020-08-07 16:38:50.498 6 ERROR oslo_service.service service.start()
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/service.py", line 174, in start
2020-08-07 16:38:50.498 6 ERROR oslo_service.service self.manager.init_host()
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 1348, in init_host
2020-08-07 16:38:50.498 6 ERROR oslo_service.service evacuated_instances = self._destroy_evacuated_instances(context)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 725, in _destroy_evacuated_instances
2020-08-07 16:38:50.498 6 ERROR oslo_service.service bdi, destroy_disks)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1312, in destroy
2020-08-07 16:38:50.498 6 ERROR oslo_service.service destroy_disks)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1402, in cleanup
2020-08-07 16:38:50.498 6 ERROR oslo_service.service instance=instance)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-07 16:38:50.498 6 ERROR oslo_service.service self.force_reraise()
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-07 16:38:50.498 6 ERROR oslo_service.service six.reraise(self.type_, self.value, self.tb)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-08-07 16:38:50.498 6 ERROR oslo_service.service raise value
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1389, in cleanup
2020-08-07 16:38:50.498 6 ERROR oslo_service.service self._disconnect_volume(context, connection_info, instance)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1628, in _disconnect_volume
2020-08-07 16:38:50.498 6 ERROR oslo_service.service self._detach_encryptor(context, connection_info, encryption=encryption)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1740, in _detach_encryptor
2020-08-07 16:38:50.498 6 ERROR oslo_service.service encryption = self._get_volume_encryption(context, connection_info)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1673, in _get_volume_encryption
2020-08-07 16:38:50.498 6 ERROR oslo_service.service self._volume_api, volume_id, connection_info)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/encryptors/__init__.py", line 113, in get_encryption_metadata
2020-08-07 16:38:50.498 6 ERROR oslo_service.service volume_id)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2020-08-07 16:38:50.498 6 ERROR oslo_service.service res = method(self, ctx, *args, **kwargs)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 745, in get_volume_encryption_metadata
2020-08-07 16:38:50.498 6 ERROR oslo_service.service return cinderclient(context).volumes.get_encryption_metadata(volume_id)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 249, in cinderclient
2020-08-07 16:38:50.498 6 ERROR oslo_service.service auth, service_parameters, url = _get_cinderclient_parameters(context)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 198, in _get_cinderclient_parameters
2020-08-07 16:38:50.498 6 ERROR oslo_service.service auth = _get_auth(context)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 100, in _get_auth
2020-08-07 16:38:50.498 6 ERROR oslo_service.service _ADMIN_AUTH = _load_auth_plugin(CONF)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 80, in _load_auth_plugin
2020-08-07 16:38:50.498 6 ERROR oslo_service.service raise cinder_exception.Unauthorized(401, message=err_msg)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service cinderclient.exceptions.Unauthorized: Unknown auth type: None (HTTP 401)
2020-08-07 16:38:50.498 6 ERROR oslo_service.service

and another trace:

2020-08-07 16:43:00.554 6 ERROR oslo_service.service Traceback (most recent call last):
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2020-08-07 16:43:00.554 6 ERROR oslo_service.service res = method(self, ctx, *args, **kwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 745, in get_volume_encryption_metadata
2020-08-07 16:43:00.554 6 ERROR oslo_service.service return cinderclient(context).volumes.get_encryption_metadata(volume_id)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 249, in cinderclient
2020-08-07 16:43:00.554 6 ERROR oslo_service.service auth, service_parameters, url = _get_cinderclient_parameters(context)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/volume/cinder.py", line 214, in _get_cinderclient_parameters
2020-08-07 16:43:00.554 6 ERROR oslo_service.service url = _SESSION.get_endpoint(auth, **service_parameters)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1200, in get_endpoint
2020-08-07 16:43:00.554 6 ERROR oslo_service.service return auth.get_endpoint(self, **kwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2020-08-07 16:43:00.554 6 ERROR oslo_service.service allow_version_hack=allow_version_hack, **kwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2020-08-07 16:43:00.554 6 ERROR oslo_service.service service_catalog = self.get_access(session).service_catalog
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-08-07 16:43:00.554 6 ERROR oslo_service.service self.auth_ref = self.get_auth_ref(session)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2020-08-07 16:43:00.554 6 ERROR oslo_service.service return self._plugin.get_auth_ref(session, **kwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 188, in get_auth_ref
2020-08-07 16:43:00.554 6 ERROR oslo_service.service authenticated=False, log=False, **rkwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1106, in post
2020-08-07 16:43:00.554 6 ERROR oslo_service.service return self.request(url, 'POST', **kwargs)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request
2020-08-07 16:43:00.554 6 ERROR oslo_service.service raise exceptions.from_response(resp, method, url)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in user. The server could not comply with the request since it is either malformed o
r otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-74807db4-f9dd-4afc-a479-1c22ee32c4d6)
2020-08-07 16:43:00.554 6 ERROR oslo_service.service

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/746039

Changed in kolla-ansible:
assignee: nobody → Jegor van Opdorp (jopdorp)
status: New → In Progress
Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
OpenStack Infra (hudson-openstack)
Changed in kolla-ansible:
assignee: Jegor van Opdorp (jopdorp) → Radosław Piliszek (yoctozepto)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/746039
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=de16013bd630019c4febdb618a3e0c5bf93f7e6f
Submitter: Zuul
Branch: master

commit de16013bd630019c4febdb618a3e0c5bf93f7e6f
Author: Jegor van Opdorp <email address hidden>
Date: Thu Aug 13 11:45:26 2020 +0200

    Add cinder auth config to nova-cell nova.conf.j2

    Fixes an issue during deleting evacuated instances with encrypted block
    devices.

    Change-Id: I9b9b689ef7e1e41b597e2c5f6b96f3ed011193c5
    Closes-Bug: 1891462
    Related-Bug: 1850279

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/747438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/747439

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/747439
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ad42c70f97bf29779b06f565bfdac12f8bc6ec61
Submitter: Zuul
Branch: stable/train

commit ad42c70f97bf29779b06f565bfdac12f8bc6ec61
Author: Jegor van Opdorp <email address hidden>
Date: Thu Aug 13 11:45:26 2020 +0200

    Add cinder auth config to nova-cell nova.conf.j2

    Fixes an issue during deleting evacuated instances with encrypted block
    devices.

    Change-Id: I9b9b689ef7e1e41b597e2c5f6b96f3ed011193c5
    Closes-Bug: 1891462
    Related-Bug: 1850279
    (cherry picked from commit de16013bd630019c4febdb618a3e0c5bf93f7e6f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/747438
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=652d3eeb5fd2aad1a7951405df2b63847167140b
Submitter: Zuul
Branch: stable/ussuri

commit 652d3eeb5fd2aad1a7951405df2b63847167140b
Author: Jegor van Opdorp <email address hidden>
Date: Thu Aug 13 11:45:26 2020 +0200

    Add cinder auth config to nova-cell nova.conf.j2

    Fixes an issue during deleting evacuated instances with encrypted block
    devices.

    Change-Id: I9b9b689ef7e1e41b597e2c5f6b96f3ed011193c5
    Closes-Bug: 1891462
    Related-Bug: 1850279
    (cherry picked from commit de16013bd630019c4febdb618a3e0c5bf93f7e6f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 10.2.0

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.3.0

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.