Comment 3 for bug 2019190

Revision history for this message
Alexander Käb (alexander-kaeb) wrote : Re: [RBD] Retyping of in-use boot volumes renders instances unusable (possible data corruption)

Hi Sofia,

all the tested instances were created from an image, with the option `Create New Volume`
checked, when creating an instance via the dashboard. The steps performed to retype the
volumes are as follows:

- Either via the Dashboard or the CLI (`cinder retype --migration-policy on-demand [...]`) retype the volume from either slow to fast or fast to slow
- rebooting the instance using i.e. a soft reboot

Just these two steps are enough to bring the instance to an error state, as libvirt will
try to load the instance's volume from the pre-retype location which will fail.
Sometimes live-migrating the instance after the retype can lead to the instance working again, but
if the instance performs some IO-Operations, there is a great chance, that the FS is broken
after an reboot:

```
[[0;32m OK [0m] Stopped target [0;1;39mBasic System[0m.
[[0;32m OK [0m] Reached target [0;1;39mInitrd File Systems[0m.
[[0;32m OK [0m] Stopped target [0;1;39mSystem Initialization[0m.
[[0;32m OK [0m] Stopped [0;1;39mdracut pre-mount hook[0m.
[[0;32m OK [0m] Stopped [0;1;39mdracut initqueue hook[0m.
[[0;32m OK [0m] Stopped [0;1;39mdracut pre-trigger hook[0m.
[[0;32m OK [0m] Stopped [0;1;39mdracut pre-udev hook[0m.
[[0;32m OK [0m] Stopped [0;1;39mdracut cmdline hook[0m.
[[0;32m OK [0m] Started [0;1;39mEmergency Shell[0m.
[[0;32m OK [0m] Reached target [0;1;39mEmergency Mode[0m.

Generating "/run/initramfs/rdsosreport.txt"

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

[?2004h:/#
```

Attached you will find the cinder-volume log and the nova-compute log during an earlier
test. (debug log enabled)