After upgrading to Focal:
Nova-Compute 21.0.0
Charm Rev 317
openstack-origin="distro-proposed"
Glance Image backend is swift/object-storage
I can neither spawn NOR delete (when errored) an instance with "Create New Volume" set to "No" (Thus creating an ephemeral backed instance)
I have tested with a standard volume backed instance and it works properly.
No error message provided, but instance will show error in openstack.
After disabling "rbd-client-cache" the instance eventually gives me an error messsage in nova-compute logs.
Error as below:
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [req-6b7a15a8-026a-4234-88e6-3bb1935779b1 003c42b161984b838a0e8e6b56fd0ad7 7795d727c4e443a9a17721cba6182d52 - 59f27b51794340d7aefdb66c2901f8e0 59f27b51794340d7aefdb66c2901f8e0] [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Instance failed to spawn: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: rbd import --pool nova /var/lib/nova/instances/_base/66ea8e506ee442cd9a2f58e8c96667ab2bc3673d 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce_disk --image-format=2 --id nova-compute --conf /var/lib/charm/nova-compute/ceph.conf
Exit code: -15
Stdout: ''
Stderr: 'rbd: --pool is deprecated for import, use --dest-pool\n\rImporting image: 1% complete...\rImporting image: 2% complete...\rImporting image: 3% complete...\rImporting image: 4% complete...\rImporting image: 5% complete...\rImporting image: 6% complete...\rImporting image: 7% complete...\rImporting image: 8% complete...\rImporting image: 9% complete...\rImporting image: 10% complete...\rImporting image: 11% complete...\rImporting image: 12% complete...\rImporting image: 13% complete...\rImporting image: 14% complete...\rImporting image: 15% complete...\rImporting image: 16% complete...\rImporting image: 17% complete...\rImporting image: 18% complete...\rImporting image: 19% complete...\rImporting image: 20% complete...\rImporting image: 21% complete...\rImporting image: 22% complete...\rImporting image: 23% complete...\rImporting image: 24% complete...\rImporting image: 25% complete...\rImporting image: 26% complete...\rImporting image: 27% complete...\rImporting image: 28% complete...\rImporting image: 29% complete...\rImporting image: 30% complete...\rImporting image: 31% complete...\rImporting image: 32% complete...\rImporting image: 33% complete...\rImporting image: 34% complete...\rImporting image: 35% complete...\rImporting image: 36% complete...\rImporting image: 37% complete...\rImporting image: 38% complete...\rImporting image: 39% complete...\rImporting image: 40% complete...\rImporting image: 41% complete...\rImporting image: 42% complete...\rImporting image: 43% complete...\rImporting image: 44% complete...\rImporting image: 45% complete...\rImporting image: 46% complete...\rImporting image: 47% complete...\rImporting image: 48% complete...\rImporting image: 49% complete...\rImporting image: 50% complete...\rImporting image: 51% complete...\rImporting image: 52% complete...\rImporting image: 53% complete...\rImporting image: 54% complete...\rImporting image: 55% complete...\rImporting image: 56% complete...\rImporting image: 57% complete...\rImporting image: 58% complete...\rImporting image: 59% complete...\rImporting image: 60% complete...\rImporting image: 61% complete...\rImporting image: 62% complete...\rImporting image: 63% complete...\rImporting image: 64% complete...\rImporting image: 65% complete...\rImporting image: 66% complete...\rImporting image: 67% complete...\rImporting image: 68% complete...\rImporting image: 69% complete...\rImporting image: 70% complete...\rImporting image: 71% complete...\rImporting image: 72% complete...\rImporting image: 73% complete...\rImporting image: 74% complete...\rImporting image: 75% complete...\rImporting image: 76% complete...\rImporting image: 77% complete...\rImporting image: 78% complete...\rImporting image: 79% complete...\rImporting image: 80% complete...\rImporting image: 81% complete...\rImporting image: 82% complete...\rImporting image: 83% complete...\rImporting image: 84% complete...\rImporting image: 85% complete...\rImporting image: 86% complete...\rImporting image: 87% complete...\rImporting image: 88% complete...\rImporting image: 89% complete...\rImporting image: 90% complete...\rImporting image: 91% complete...\rImporting image: 92% complete...\rImporting image: 93% complete...\rImporting image: 94% complete...\rImporting image: 95% complete...\rImporting image: 96% complete...\rImporting image: 97% complete...\rImporting image: 98% complete...\rImporting image: 99% complete...'
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Traceback (most recent call last):
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2614, in _build_resources
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] yield resources
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2374, in _build_and_run_instance
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self.driver.spawn(context, instance, image_meta,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3561, in spawn
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self._create_image(context, instance, disk_info['mapping'],
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3933, in _create_image
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self._create_and_inject_local_root(context, instance,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4040, in _create_and_inject_local_root
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self._try_fetch_image_cache(backend, fetch_func, context,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9500, in _try_fetch_image_cache
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] image.cache(fetch_func=fetch_func,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 274, in cache
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self.create_image(fetch_func_sync, base, size,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 945, in create_image
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] self.driver.import_image(base, self.rbd_name)
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 335, in import_image
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] processutils.execute('rbd', 'import', *args)
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 421, in execute
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] raise ProcessExecutionError(exit_code=_returncode,
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Command: rbd import --pool nova /var/lib/nova/instances/_base/66ea8e506ee442cd9a2f58e8c96667ab2bc3673d 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce_disk --image-format=2 --id nova-compute --conf /var/lib/charm/nova-compute/ceph.conf
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Exit code: -15
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Stdout: ''
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce] Stderr: 'rbd: --pool is deprecated for import, use --dest-pool\n\rImporting image: 1% complete...\rImporting image: 2% complete...\rImporting image: 3% complete...\rImporting image: 4% complete...\rImporting image: 5% complete...\rImporting image: 6% complete...\rImporting image: 7% complete...\rImporting image: 8% complete...\rImporting image: 9% complete...\rImporting image: 10% complete...\rImporting image: 11% complete...\rImporting image: 12% complete...\rImporting image: 13% complete...\rImporting image: 14% complete...\rImporting image: 15% complete...\rImporting image: 16% complete...\rImporting image: 17% complete...\rImporting image: 18% complete...\rImporting image: 19% complete...\rImporting image: 20% complete...\rImporting image: 21% complete...\rImporting image: 22% complete...\rImporting image: 23% complete...\rImporting image: 24% complete...\rImporting image: 25% complete...\rImporting image: 26% complete...\rImporting image: 27% complete...\rImporting image: 28% complete...\rImporting image: 29% complete...\rImporting image: 30% complete...\rImporting image: 31% complete...\rImporting image: 32% complete...\rImporting image: 33% complete...\rImporting image: 34% complete...\rImporting image: 35% complete...\rImporting image: 36% complete...\rImporting image: 37% complete...\rImporting image: 38% complete...\rImporting image: 39% complete...\rImporting image: 40% complete...\rImporting image: 41% complete...\rImporting image: 42% complete...\rImporting image: 43% complete...\rImporting image: 44% complete...\rImporting image: 45% complete...\rImporting image: 46% complete...\rImporting image: 47% complete...\rImporting image: 48% complete...\rImporting image: 49% complete...\rImporting image: 50% complete...\rImporting image: 51% complete...\rImporting image: 52% complete...\rImporting image: 53% complete...\rImporting image: 54% complete...\rImporting image: 55% complete...\rImporting image: 56% complete...\rImporting image: 57% complete...\rImporting image: 58% complete...\rImporting image: 59% complete...\rImporting image: 60% complete...\rImporting image: 61% complete...\rImporting image: 62% complete...\rImporting image: 63% complete...\rImporting image: 64% complete...\rImporting image: 65% complete...\rImporting image: 66% complete...\rImporting image: 67% complete...\rImporting image: 68% complete...\rImporting image: 69% complete...\rImporting image: 70% complete...\rImporting image: 71% complete...\rImporting image: 72% complete...\rImporting image: 73% complete...\rImporting image: 74% complete...\rImporting image: 75% complete...\rImporting image: 76% complete...\rImporting image: 77% complete...\rImporting image: 78% complete...\rImporting image: 79% complete...\rImporting image: 80% complete...\rImporting image: 81% complete...\rImporting image: 82% complete...\rImporting image: 83% complete...\rImporting image: 84% complete...\rImporting image: 85% complete...\rImporting image: 86% complete...\rImporting image: 87% complete...\rImporting image: 88% complete...\rImporting image: 89% complete...\rImporting image: 90% complete...\rImporting image: 91% complete...\rImporting image: 92% complete...\rImporting image: 93% complete...\rImporting image: 94% complete...\rImporting image: 95% complete...\rImporting image: 96% complete...\rImporting image: 97% complete...\rImporting image: 98% complete...\rImporting image: 99% complete...'
2020-07-03 20:50:06.969 463270 ERROR nova.compute.manager [instance: 98c5b6f8-41c6-44ce-b025-ff127ca8b6ce]
By being unable to use ephemeral (nova) backed images, I am unable to rescue instance, because "Stable device instance rescue" (introduced in 21.0.0) does not work as well.
