commit 3f8daf080411b84ec0669f0642524ce8a7d19057
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 21 15:29:30 2016 +0000
libvirt: Re-initialise volumes, encryptors, and vifs on hard reboot
We call _hard_reboot during reboot, power_on, and
resume_state_on_host_boot. It functions essentially by tearing as much
of an instance as possible before recreating it, which additionally
makes it useful to operators for attempting automated recovery of
instances in an inconsistent state.
The Libvirt driver would previously only call _destroy and
_undefine_domain when hard rebooting an instance. This would leave vifs
plugged, volumes connected, and encryptors attached on the host. It
also means that when we try to restart the instance, we assume all
these things are correctly configured. If they are not, the instance
may fail to start at all, or may be incorrectly configured when
starting.
For example, consider an instance with an encrypted volume after a
compute host reboot. When we attempt to start the instance, power_on
will call _hard_reboot. The volume will be coincidentally re-attached
as a side-effect of calling _get_guest_xml(!), but when we call
_create_domain_and_network we pass reboot=True, which tells it not to
reattach the encryptor, as it is assumed to be already attached. We
are therefore left presenting the encrypted volume data directly to
the instance without decryption.
The approach in this patch is to ensure we recreate the instance as
fully as possible during hard reboot. This means not passing
vifs_already_plugged and reboot to _create_domain_and_network, which
in turn requires that we fully destroy the instance first. This
addresses the specific problem given in the example, but also a whole
class of potential volume and vif related issues of inconsistent
state.
Because we now always tear down volumes, encryptors, and vifs, we are
relying on the tear down of these things to be idempotent. This
highlighted that detach of the luks and cryptsetup encryptors were not
idempotent. We depend on the fixes for those os-brick drivers.
Reviewed: https:/ /review. openstack. org/400384 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=3f8daf08041 1b84ec0669f0642 524ce8a7d19057
Committed: https:/
Submitter: Zuul
Branch: master
commit 3f8daf080411b84 ec0669f0642524c e8a7d19057
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 21 15:29:30 2016 +0000
libvirt: Re-initialise volumes, encryptors, and vifs on hard reboot
We call _hard_reboot during reboot, power_on, and state_on_ host_boot. It functions essentially by tearing as much
resume_
of an instance as possible before recreating it, which additionally
makes it useful to operators for attempting automated recovery of
instances in an inconsistent state.
The Libvirt driver would previously only call _destroy and domain when hard rebooting an instance. This would leave vifs
_undefine_
plugged, volumes connected, and encryptors attached on the host. It
also means that when we try to restart the instance, we assume all
these things are correctly configured. If they are not, the instance
may fail to start at all, or may be incorrectly configured when
starting.
For example, consider an instance with an encrypted volume after a domain_ and_network we pass reboot=True, which tells it not to
compute host reboot. When we attempt to start the instance, power_on
will call _hard_reboot. The volume will be coincidentally re-attached
as a side-effect of calling _get_guest_xml(!), but when we call
_create_
reattach the encryptor, as it is assumed to be already attached. We
are therefore left presenting the encrypted volume data directly to
the instance without decryption.
The approach in this patch is to ensure we recreate the instance as already_ plugged and reboot to _create_ domain_ and_network, which
fully as possible during hard reboot. This means not passing
vifs_
in turn requires that we fully destroy the instance first. This
addresses the specific problem given in the example, but also a whole
class of potential volume and vif related issues of inconsistent
state.
Because we now always tear down volumes, encryptors, and vifs, we are
relying on the tear down of these things to be idempotent. This
highlighted that detach of the luks and cryptsetup encryptors were not
idempotent. We depend on the fixes for those os-brick drivers.
Depends-On: I31d72357c89db5 3a147c2d986a28c 9c6870efad0 3699cfd5c0e32c6 72c934cd6fb
Depends-On: I9f52f89b8466d0
Closes-bug: #1724573 2d14e16c7f61142 91d547a8986
Change-Id: Id188d48609f3d2