Cinder NFS backend with image-volume cache creates volumes with wrong size

[Danila Balagansky]

Cinder version (commit SHA): 83089b92467d8deac5793f4f4d739c936a758ade

backend configuration in cinder.conf:
[vm]
image_volume_cache_enabled = True
image_volume_cache_max_count = 40
image_volume_cache_max_size_gb = 30
nas_secure_file_operations = False
nas_secure_file_permissions = False
nfs_mount_options = rsize=65535,wsize=65535,timeo=1200,actimeo=120
nfs_qcow2_volumes = True
nfs_shares_config = /etc/cinder/vm_nfs_shares
nfs_snapshot_support = True
report_discard_supported = True
volume_backend_name = vm
volume_driver = cinder.volume.drivers.nfs.NfsDriver

Debian 11 image: https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2

Image in Glance:
+------------------+--------------------------------------+
| Field | Value |
+------------------+--------------------------------------+
| checksum | 06e481ddd23682af4326226661c13d8f |
| container_format | bare |
| disk_format | qcow2 |
| id | 7673db9f-6c20-4b21-b817-2932019d5f2b |
| name | debian11 |
| size | 254672896 |
| virtual_size | 2147483648 |
+------------------+--------------------------------------+

Creating volume without image-volume cache, like this:
root@oscp99:~# openstack volume create --image debian11 --size 11 test
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.029899 |
| description | None |
| encrypted | False |
| id | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| migration_status | None |
| multiattach | False |
| name | test |
| properties | |
| replication_status | None |
| size | 11 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | vm |
| updated_at | None |
| user_id | b14242e3bdef45e797a7973bc75d680d |
+---------------------+--------------------------------------+

Produces the log `cinder-volume_create_volume_without_cache.txt` with stacktrace, but in the end, volume does get created successfully:
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.000000 |
| description | None |
| encrypted | False |
| id | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| migration_status | None |
| multiattach | False |
| name | test |
| os-vol-host-attr:host | oscn99.local@vm#vm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 8e0353b219594c819af69294e15f4e23 |
| properties | |
| replication_status | None |
| size | 11 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | vm |
| updated_at | 2022-09-18T07:28:53.000000 |
| user_id | b14242e3bdef45e797a7973bc75d680d |
| volume_image_metadata | {'signature_verified': 'False', 'os_glance_importing_to_stores': '', 'os_glance_failed_import': '', 'image_id': '7673db9f-6c20-4b21-b817-2932019d5f2b', 'image_name': 'debian11', 'checksum': '06e481ddd23682af4326226661c13d8f', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '254672896'} |
+--------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

But the size on the NFS share is wrong:

qemu-img info /vm/cinder/volumes/volume-4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0
image: /vm/cinder/volumes/volume-4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 700 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Image-volume cache volume:
+--------------------------------+--------------------------------------------+
| Field | Value |
+--------------------------------+--------------------------------------------+
| attachments | [] |
| availability_zone | office |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-09-18T07:25:54.000000 |
| description | None |
| encrypted | False |
| id | 4a85b808-2ccf-442e-875e-1c71e65f21e1 |
| migration_status | None |
| multiattach | False |
| name | image-7673db9f-6c20-4b21-b817-2932019d5f2b |
| os-vol-host-attr:host | oscn99.local@vm#vm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 09d81658b99d464aa3b4d747389e38e1 |
| properties | readonly='True' |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 4eb3d4e8-043d-4a2f-9fb8-98f4c91d01c0 |
| status | available |
| type | vm |
| updated_at | 2022-09-18T07:27:58.000000 |
| user_id | b14242e3bdef45e797a7973bc75d680d |
+--------------------------------+--------------------------------------------+

Creating volume for the second time (with image-volume cache populated):

openstack volume create --image debian11 --size 11 tttt

produces `cinder-volume_create_volume_with_cache.txt` log, where `qemu-img create` is called with the original volume size (2G in this example):

qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/0825b3db343399a47ca1773445a9278f/volume-4a85b808-2ccf-442e-875e-1c71e65f21e1,backing_fmt=qcow2 /var/lib/cinder/mnt/0825b3db343399a47ca1773445a9278f/volume-4a85b808-2ccf-442e-875e-1c71e65f21e1.337354d2-07eb-4b92-9ab7-6f300d8a7440 2G

and after that, volume never gets resized to the requested size.

Created volume has expected size in DB:
+--------------------------------------+------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+-----------+------+-------------+
| 21be2d3a-1788-4234-934c-de4a0e2e12f4 | tttt | available | 11 | |
+--------------------------------------+------+-----------+------+-------------+

But not on the NFS share:
root@oscn99:~# qemu-img info /vm/cinder/volumes/volume-21be2d3a-1788-4234-934c-de4a0e2e12f4
image: /vm/cinder/volumes/volume-21be2d3a-1788-4234-934c-de4a0e2e12f4
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 700 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Supposed fix: https://review.opendev.org/c/openstack/cinder/+/855964
Related bug: https://bugs.launchpad.net/cinder/+bug/1988803 (without information about used backend), but it's about creating volumes with size, less that image-volume cache volume.

When using LVM backend, logical volume is created with expected size.