Resize to swapless flavor still looking for swap

[Maxence Sartiaux]

Description
===========
Resizing an instance with swap to a flavor without swap, the instance won't run because libvirt will still look after a swap file, for exemple, the default_swap_device in nova/instances table is still set to /dev/vdb after resizing.

The XML passed to libvirt still have the swap device defined.

Steps to reproduce
==================
* Create an instance with Swap
* Resize instance to a flavor without swap
* Try to start instance

Expected result
===============
The instance start without swap

Actual result
=============
Error in nova log
OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/66035f60-6e17-4f09-bb00-885e3cc5750f/disk.swap'

libvirt xml still have the swap defined

Environment
===========

rpm -qa | grep nova
openstack-nova-console-17.0.2-1.el7.noarch
openstack-nova-common-17.0.2-1.el7.noarch
openstack-nova-scheduler-17.0.2-1.el7.noarch
python2-novaclient-9.1.1-1.el7.noarch
python-nova-17.0.2-1.el7.noarch
openstack-nova-novncproxy-17.0.2-1.el7.noarch
openstack-nova-conductor-17.0.2-1.el7.noarch
openstack-nova-api-17.0.2-1.el7.noarch
openstack-nova-placement-api-17.0.2-1.el7.noarch

rpm -qa | grep libvirt
libvirt-daemon-kvm-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.9.x86_64
libvirt-client-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7_4.9.x86_64
libvirt-libs-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.9.x86_64
libvirt-python-3.2.0-3.el7_4.1.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.9.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7_4.9.x86_64

rpm -qa | grep cinder
python-cinder-12.0.0-1.el7.noarch
python2-cinderclient-3.5.0-1.el7.noarch
openstack-cinder-12.0.0-1.el7.noarch
rpm -qa | grep lvm
lvm2-libs-2.02.171-8.el7.x86_64
lvm2-2.02.171-8.el7.x86_64

[jichenjc (jichenjc)]

INFO nova.compute.manager [None req-0516127d-e96f-4a85-a050-70c058f77a44 service nova] [instance: 2ebb3db1-e99f-4b5c-85f4-ae3c9f93ca79] Successfully reverted task state from resize_finish on failure for instance.
ERROR oslo_messaging.rpc.server [None req-0516127d-e96f-4a85-a050-70c058f77a44 service nova] Exception during message handling: libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/2ebb3db1-e99f-4b5c-85f4-ae3c9f93ca79/disk.swap' (as uid:64055, gid:124): No such file or directory
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 76, in wrapped
ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 67, in wrapped
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 186, in decorated_function
ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 156, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 976, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
ERROR oslo_messaging.rpc.server File "/usr/local/lib/pyth...

[jichenjc (jichenjc)]

some debug shows even swap disk is 0, we will still create guest configuration which leads to problem ......

> /opt/stack/nova/nova/virt/libvirt/driver.py(3922)_get_guest_storage_config()
-> devices.append(diskswap)
(Pdb) l
3917 pdb.set_trace()
3918 diskswap = self._get_guest_disk_config(instance,
3919 'disk.swap',
3920 disk_mapping,
3921 inst_type)
3922 -> devices.append(diskswap)
3923 instance.default_swap_device = (
3924 block_device.prepend_dev(diskswap.target_dev))
3925
3926 config_name = 'disk.config.rescue' if rescue else 'disk.config'
3927 if config_name in disk_mapping:
(Pdb) p diskswap
<nova.virt.libvirt.config.LibvirtConfigGuestDisk object at 0x7f34e00565d0>
(Pdb) dir(diskswap)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_new_node', '_text_node', 'auth_secret_type', 'auth_secret_uuid', 'auth_username', 'backing_store', 'boot_order', 'device_addr', 'disk_read_bytes_sec', 'disk_read_iops_sec', 'disk_total_bytes_sec', 'disk_total_iops_sec', 'disk_write_bytes_sec', 'disk_write_iops_sec', 'driver_cache', 'driver_discard', 'driver_format', 'driver_io', 'driver_name', 'encryption', 'format_dom', 'logical_block_size', 'mirror', 'ns_prefix', 'ns_uri', 'parse_dom', 'parse_str', 'physical_block_size', 'readonly', 'root_name', 'serial', 'shareable', 'snapshot', 'source_device', 'source_hosts', 'source_name', 'source_path', 'source_ports', 'source_protocol', 'source_type', 'target_bus', 'target_dev', 'target_path', 'to_xml']
(Pdb) p diskswap.source_name
None
(Pdb) p diskswap.source_path
'/opt/stack/data/nova/instances/114418e2-1d4e-47ba-9b20-bf592f772af5/disk.swap'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/561122

[Wendy Mitchell (wmitchellwr)]

Load: 20190404T013000Z
Node Config: 2+3
Failing resize - original flavor has swap. Instance Power Status Running but Status changes to ERROR

| fault | {"message": "Cannot access storage file '/var/lib/nova/instances/18a2bb19-5044-440b-9064-96b130e85c2a/disk.swap' (as uid:0, gid:0): No such file or directory", "code": 500, "details": " File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py\", line 201, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py\", line 4703, in finish_resize |
| | self._revert_allocation(context, instance, migration) |
| | File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 220, in __exit__ |
| | self.force_reraise() |
| | File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 196, in
| | return function(self, context, *args, **kwargs) |
| | File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py\", line 4703, in finish_resize |
| | self._revert_allocation(context, instance, migration) ...

[Wendy Mitchell (wmitchellwr)]

Behaviour in Load: 20190622T013000Z
 Lab: WP_3_7

The instance is in Error on resize, then rebuilds eventually with the original flavor.
req-f8ecef0e-d627-447e-8d65-cccc502d175a Rebuild June 25, 2019, 7:03 p.m. af0923a7f8a641d1902e3552be9f74f2 -
req-6fa43151-6e55-4d60-9a28-82082f173b42 Reboot June 25, 2019, 7:02 p.m. af0923a7f8a641d1902e3552be9f74f2 Error
req-d15ddce3-69e2-4863-b594-ba16fec43399 Resize June 25, 2019, 7:01 p.m. af0923a7f8a641d1902e3552be9f74f2 Error
req-f2cb3640-5b7d-4514-90f5-f252aebd33b6 Create June 25, 2019, 7 p.m. a1f1bc3dc19442f99f28b9c8a4dc3408 -

    Cannot access storage file '/var/lib/nova/instances/5e289a1c-c6c9-4268-950e-89aa27486b4a/disk.swap' (as uid:0, gid:0): No such file or directory
Code
    500
Details
    File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function return function(self, context, *args, **kwargs) File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4724, in finish_resize self._revert_allocation(context, instance, migration) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4721, in finish_resize migration) File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4740, in _finish_resize_helper disk_info, image_meta, bdms) File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4691, in _finish_resize old_instance_type) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4686, in _finish_resize block_device_info, power_on) File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 9018, in finish_migration post_xml_callback=gen_confdrive) File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5714, in _create_domain_and_network destroy_disks_on_failure) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5683, in _create_domain_and_network post_xml_callback=post_xml_callback) File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5618, in _create_domain guest.launch(pause=pause) File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch self._encoded_xml, errors='ignore') File "/var/lib/openstack/lib/python2.7/site-...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Fried (<email address hidden>) on branch: master
Review: https://review.opendev.org/561122
Reason: This has had no activity for >1y. Please reopen if still needed.