[NFS] Nova raises error on server resize command

[Alexander Malashenko]

[env] devstack
[OS] Ubuntu 20.04.4 LTS
[driver] generic-nfs

Nova raises error when changing the parameters of the instance and connected disk with the type volume generic-nfs.

Execution steps for this case:
1. Create a new volume with image and with type generic-nfs
2. Create a new instance with boot volume.
3. Do something interactive with the guest like get its console output.
4. Modify server disk and RAM to other provided flavor via openstack cli "server resize --flavor FLAVOR INST"

/etc/cinder/cinder.conf
[generic-nfs]
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_qcow2_volumes=True
nfs_snapshot_support=True
nfs_sparsed_volumes=False
nas_host=nas_ip
nas_share_path=nas_path
nas_secure_file_operations=False
nas_secure_file_permissions=False
volume_backend_name=generic-nfs

(openstack) server list
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+------------------------+--------------------------+---------+
| 78355147-7b94-479d-99e4-8821ec9883ff | testGenericInst | ACTIVE | shared=192.168.233.227 | N/A (booted from volume) | m1.nano |

(openstack) volume show genericVolume

| Field | Value |
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2022-07-13T10:03:06.000000 |
| description | |
| encrypted | False |
| id | b2ed4de3-9564-4d85-a030-2d53a2c058d3 |
| migration_status | None |
| multiattach | False |
| name | genericVolume |
| os-vol-host-attr:host | ubuntu@generic-nfs#generic-nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | ca2d2e2e1f6640e899e6f822d18bf11d |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | attaching |
| type | generic-nfs |
| updated_at | 2022-07-13T10:17:00.000000 |
| user_id | b55ef3ba97674255966e278b81aff22f |
| volume_image_metadata | {'signature_verified': 'False', 'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros-0.5.2-x86_64-disk', 'owner_specified.openstack.sha256': '', 'image_id': '83560164-8301-425b-89c5-36fe5b3f222c', 'image_name': 'cirros-0.5.2-x86_64-disk', 'checksum': 'b874c39491a2377b8490f5f1e89761a4', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '16300544'} |

(openstack) server resize --flavor 2 testGenericInst

<email address hidden> output
Jul 13 03:17:08 ubuntu nova-compute[991]: : libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-07-13T10:17:07.119481Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest Traceback (most recent call last):
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193, in doit
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151, in proxy_call
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest six.reraise(c, e, tb)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest raise value
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-07-13T10:17:07.119481Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.guest
Jul 13 03:17:08 ubuntu nova-compute[991]: ERROR nova.virt.libvirt.driver [None req-4c327787-e543-4353-af4d-a98c7a9a906a admin admin] [instance: 78355147-7b94-479d-99e4-8821ec9883ff] Failed to start libvirt guest: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-07-13T10:17:07.119481Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}: Image is not in qcow2 format

qemu-img info ./volume-b2ed4de3-9564-4d85-a030-2d53a2c058d3
image: ./volume-b2ed4de3-9564-4d85-a030-2d53a2c058d3
file format: raw
virtual size: 1 GiB (1073741824 bytes)
disk size: 56.4 MiB

[Sofia Enriquez (lsofia-enriquez)]

Greetings,
As far as I understand, the generic-nfs tend to format raw volumes instead of qcow2, so I do not think this problem is an issue with Cinder.
 I am not sure I understand why it is mandatory to use a qcow2 volume during instance resizing, I think nova could use the raw volume.
Please let me know if this is not correct and if you need my input again.
Sofia

[Jean Pierre Roquesalane (jproque15130)]

Thanks Sofia for your input.
We probably have to file this bug against nova, not cinder

[melanie witt (melwitt)]

I don't have anything super helpful to add, but in trying to learn more about this error, I found the following bug reports [1][2] (emphasis on "reports" as these were closed NOTABUG) which concluded with them being issues in the NFS server itself related to locking.

*If* this issue happens to be the same root cause as [1] and [2], then adding the "nolock" mount option for NFS [3] as a workaround should make the problem disappear:

  [libvirt]
  nfs_mount_options = nolock

(multiple options can be specified with commas between them, see [3])

It might be worth giving that a try and let us know whether it has any effect on your scenario.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1817640
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1891673
[3] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options

[Sylvain Bauza (sylvain-bauza)]

Can the reporter please try to verify whether the nolock mount option works for this ?

Moving to Imcomplete.