Cold migration/resize failure with encrypted volumes can leave instance in error and volumes attaching

[Mark Goddard]

Description
===========
Due to the differences in nova, cinder and barbican policies described in bug 1895848, a user cannot migrate an instance with an encrypted volume (using barbican) that belongs to a user in a different project. Furthermore, if a cold migration or resize is attempted and fails when accessing the encryption key, the instance will go to an 'error' state, and the volumes will get stuck in the 'attaching' state.

Steps to reproduce
==================
Prerequisites: users A & B, where B has the admin role.

As user A in project A, create an instance with an encrypted volume.
As user B in project B, attempt to cold migrate the instance.

Expected result
===============
Cold migration is unsuccessful. Instance remains active with volume attached.

Actual result
=============
Cold migration is unsuccessful. Instance is in ERROR state and shutoff. The volume appears to be attached from the nova perspective, but in Cinder its status is attaching. The volume has lost the attachment record.

Environment
===========
Seen in Stein, CentOS 7, deployed via Kolla Ansible.

Logs
====
Will follow up with more info.

[Mark Goddard (mgoddard)]

I have a patch for this, need to propose it.

[Mark Goddard (mgoddard)]

https://review.opendev.org/c/openstack/nova/+/781079