Failed to start instances with encrypted volumes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Lee Yarwood | ||
Queens |
Fix Committed
|
Medium
|
Lee Yarwood | ||
Rocky |
Fix Released
|
Medium
|
Lee Yarwood | ||
Stein |
Fix Released
|
Medium
|
Lee Yarwood | ||
kolla-ansible |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Description
===========
We hit this bug after doing a complete cluster shutdown due to server room maintenance. The bug is however more easily reproducible.
When cold starting an instance with an encrypted volume attached, it fails so start with a VolumeEncryptio
https:/
Steps to reproduce
==================
* Deploy Openstack with Barbican support using Kolla.
* Create an encrypted volume type
* Create an encrypted volume
* Create an instance and attach the encrypted folder
* Enjoy your new instance and volume, install software and store data
* In our case, we shut down the entire cluster and restarted it again. First all instances were stopped in Horizon using Shut down instance command. We use Ceph so we then stopped that using these procedures https:/
* Instances without encrypted volumes started fine.
* Instances with encrypted volumes fail to start with VolumeEncryptio
Note: It is possible to recreate the problem by using a Hard Reboot (possibly related https:/
Expected results
================
Instances with encrypted volumes should start fine, even after a Hard Reboot or a complete cluster shutdown.
Actual results
==============
Instances with encrypted volumes failed to start with VolumeEncryptio
Environment
===========
1. Openstack version
Environment is established by Kolla (Rocky release).
2. Hypervisor
KVM on RHEL
3. Storage type
Ceph using Kolla (Rocky release)
Analysis
========
There seems to be a problem related to this code not behaving as expected:
https:/
It seems that it is expected that the exception should be ignored and logged, but for some reason, the `ctxt.reraise = False` does not work as expected:
self.force_
We did some hacking and just swallowed the exception by commenting out the `excutils.
Then the instance booted - but it could not boot from the image. But, it was then possible to remove the encrypted volume attachment, reboot the server and then reattach the encrypted volume.
description: | updated |
tags: | added: volumes |
tags: | added: encryption |
Changed in nova: | |
status: | New → Confirmed |
assignee: | nobody → pandatt (pandatt) |
affects: | kolla → kolla-ansible |
Changed in kolla-ansible: | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Mark Goddard (mgoddard) |
Changed in nova: | |
importance: | Undecided → Medium |
tags: | added: libvirt |
Changed in kolla-ansible: | |
status: | Fix Committed → In Progress |
Changed in kolla-ansible: | |
status: | In Progress → Triaged |
milestone: | 8.0.0 → none |
assignee: | Mark Goddard (mgoddard) → nobody |
Changed in kolla-ansible: | |
status: | Triaged → Invalid |
no longer affects: | kolla-ansible/rocky |
no longer affects: | kolla-ansible/stein |
Changed in kolla-ansible: | |
importance: | High → Undecided |
Note that it was not possible to remove the encrypted volume attachment from the affected hosts - that would also yield a VolumeEncryptio nNotSupported error.