nfs: failed to extend volume after blockRebase

[Rajat Dhasmana]

Followings are the steps to reproduce the issue:

1) create a bootable volume
2) launch an instance from the volume
3) create snapshot of instance (using nova image-create)
4) delete the snapshot (blockRebase operation)
5) delete the VM
6) try to perform extend operation on volume

it fails with the following error in cinder-volume:

ERROR cinder.volume.manager [req-5f939fc7-f4a5-4f6a-9b70-6dcb52ee7f2e req-a2702c72-fc40-41df-a600-903a5293b898 admin None] Extend volume failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe 2G
Exit code: 1
Stdout: ''
Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': No such file or directory\n"
ERROR cinder.volume.manager Traceback (most recent call last):
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2767, in extend_volume
ERROR cinder.volume.manager self.driver.extend_volume(volume, new_size)
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 373, in extend_volume
ERROR cinder.volume.manager extend_by = int(new_size) - volume.size
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/image/image_utils.py", line 413, in resize_image
ERROR cinder.volume.manager utils.execute(*cmd, run_as_root=run_as_root)
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 126, in execute
ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
ERROR cinder.volume.manager File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 431, in execute
ERROR cinder.volume.manager raise ProcessExecutionError(exit_code=_returncode,
ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe 2G
ERROR cinder.volume.manager Exit code: 1
ERROR cinder.volume.manager Stdout: ''
ERROR cinder.volume.manager Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': No such file or directory\n"
ERROR cinder.volume.manager

stack@ubuntu2004:~/cinder$ cinder create --image 4921910a-d230-4933-b773-4af90544024f 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-06T15:18:43.000000 |
| description | None |
| encrypted | False |
| id | 2a351c27-abb0-4965-a16e-5378bd2c2afe |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | ubuntu2004@nfs#nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 90d7287d062c4ef99c0624af9023fd13 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2020-11-06T15:18:44.000000 |
| user_id | 0cd3910ea1ca40528da47cf255320a76 |
| volume_type | nfs |
+--------------------------------+--------------------------------------+
stack@ubuntu2004:~/cinder$ nova boot --flavor 42 --block-device source=volume,id=2a351c27-abb0-4965-a16e-5378bd2c2afe,size=1,shutdown=preserve,bootindex=0 ints1 --nic net-id=d6f03ab9-4849-4c4a-8d84-9d508b9c8919
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | ints1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-gfhqe76n |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | zxeQ5n3n2RaE |
| config_drive | |
| created | 2020-11-06T15:19:07Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw_rng:allowed": "True"} |
| flavor:original_name | m1.nano |
| flavor:ram | 128 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | 3a922fe9-bae1-4d80-aa96-654ed0b8b110 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | ints1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 90d7287d062c4ef99c0624af9023fd13 |
| trusted_image_certificates | - |
| updated | 2020-11-06T15:19:06Z |
| user_id | 0cd3910ea1ca40528da47cf255320a76 |
+--------------------------------------+-------------------------------------------------+
stack@ubuntu2004:~/cinder$ nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
| 3a922fe9-bae1-4d80-aa96-654ed0b8b110 | ints1 | ACTIVE | - | Running | private=fdc8:7eef:b74b:0:f816:3eff:fe75:196d, 10.0.0.48 |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
stack@ubuntu2004:~/cinder$ nova image-create 3a922fe9-bae1-4d80-aa96-654ed0b8b110 snap1
cinder snapshstack@ubuntu2004:~/cinder$ cinder snapshot-list
cinder snapsho+--------------------------------------+--------------------------------------+-----------+--------------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+--------------------+------+
| 546e3016-8f6f-4e13-9099-53c26d454890 | 2a351c27-abb0-4965-a16e-5378bd2c2afe | available | snapshot for snap1 | 1 |
+--------------------------------------+--------------------------------------+-----------+--------------------+------+
tstack@ubuntu2004:~/cinder$ cinder snapshot-delete 546e3016-8f6f-4e13-9099-53c26d454890
stack@ubuntu2004:~/cinder$ cinder snapshot-list
+----+-----------+--------+------+------+
| ID | Volume ID | Status | Name | Size |
+----+-----------+--------+------+------+
+----+-----------+--------+------+------+
stack@ubuntu2004:~/cinder$ nova delete 3a922fe9-bae1-4d80-aa96-654ed0b8b110
Request to delete server 3a922fe9-bae1-4d80-aa96-654ed0b8b110 has been accepted.
stack@ubuntu2004:~/cinder$ nova list
cinder +----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
lstack@ubuntu2004:~/cinder$ cinder list
cinde+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 2a351c27-abb0-4965-a16e-5378bd2c2afe | available | - | 1 | nfs | true | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@ubuntu2004:~/cinder$ cinder extend 2a351c27-abb0-4965-a16e-5378bd2c2afe 2
stack@ubuntu2004:~/cinder$ cinder list
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+
| 2a351c27-abb0-4965-a16e-5378bd2c2afe | error_extending | - | 1 | nfs | true | |
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762462

[Lucio Seki (lseki)]

Hi Rajat, I reproduced the issue with NetApp ONTAP NFS driver [0].

The error message is slightly different, though [1]:

---
Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/83c3167af2464949d83cddf33977804e/volume-ac6a5bbd-ae96-490c-a1f9-04f744d5e4ba': Could not open '/
opt/stack/data/cinder/mnt/83c3167af2464949d83cddf33977804e/volume-ac6a5bbd-ae96-490c-a1f9-04f744d5e4ba': Permission denied\n".
---

[0] http://paste.openstack.org/show/799996/
[1] http://paste.openstack.org/show/799997/

[Rajat Dhasmana (whoami-rajat)]

Hi Lucio,

Looks pretty much the same issue.
Can you manually check if the volume file it's trying to access exists in the directory /opt/stack/data/cinder/mnt/<share-hash>/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/763181

[Lucio Seki (lseki)]

Hi Rajat, the file does exist [0].

The failing qemu-img info command succeeds when manually executed as root [1].

[0] http://paste.openstack.org/show/800387/
[1] http://paste.openstack.org/show/800388/

[Rajat Dhasmana (whoami-rajat)]

The code seems strange to me for deleting snapshot, It doesn't check if the volume is attached, doesn't perform any rebase of data in the base file to the snapshot file while deleting. It is just locating the file and deleting it.[1]
Looks very wrong to me or I'm just missing something here.

[1] https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/netapp/dataontap/nfs_base.py#L266-L280

[Sofia Enriquez (lsofia-enriquez)]

Should we move this to won't fix?

[Rajat Dhasmana (whoami-rajat)]

Hi Sofia,

We already have a fix proposed[1] and IMO this is an important issue since we don't keep track of nfs file changing names. So if we attach, perform snapshot operations, detach, then cinder won't be able to locate the volume file.

[1] https://review.opendev.org/c/openstack/cinder/+/763181/