NetApp ONTAP enhanced instance creation using file copy API should allow use of DNS name for NFS share

[Michael Arndt]

Looking at the NetApp ONTAP changes from the https://review.opendev.org/c/openstack/cinder/+/847732 commit to enable enhanced instance provisioning and it has issues when using DNS names for cinder shares.

This is because driver is hardcoding the IP address to determine destination share path. https://review.opendev.org/c/openstack/cinder/+/847732/5/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py#762

Whenever we use IP address for cinder mounts, we don’t see any errors during image-cache file creation. To fix this, we changed above line to below. Please check and confirm if below code change is good and if this can be added to upstream code.
        #dst_share = dst_ip + ':' + dest_share_path
        dst_share = volume_utils.extract_host(volume['host'], level='pool')

[Sofia Enriquez (lsofia-enriquez)]

Is this related to https://bugs.launchpad.net/cinder/+bug/2020733?

[Saravanan Manickam (msaravan)]

@Michael Arndt :

Thank you for raising this. Yes, its ideal to use DNS name which is present in nfs_shares_config file.

I tried mounting the share via DNS name, something like this:

root@linux-box:~# cat /etc/cinder/nfs_shares
msaravan-openstack-vm.netapp.com:/cinder1
root@linux-box:~#

Without code change, the destination share looks like this :
10.1.1.1:/msaravan_cinder1.

I am able to hit the code flow you suggested. With the code change you have proposed, the destination
share would look like this, and this is exactly matching with what is configured in NFS shares file:

msaravan-openstack-vm.netapp.com:/cinder1

When the copy file is invoked with the existing code, the variable dst_share is never used. You can see that below.

self._copy_file(img_file, img_file, src_volume, src_vserver,
                                dest_share_path, dest_vserver,
                                dest_backend_name=self.backend_name,
                                dest_file_name=tmp_img_file)

Also the copy takes place using below parameters via NetApp ZAPI call, and its always successful:

job_uuid = self.zapi_client.start_file_copy(
            file_name, dest_ontap_volume, src_ontap_volume=src_ontap_volume,
            dest_file_name=dest_file_name)

The actual values passed in the above method are :

file_name: d4ed4356-cfeb-474d-b254-90a0501352c9
dest_ontap_volume : msaravan_cinder1
src_ontap_volume: glance_flexvol
dest_file_name: b16dbe27-f47b-4449-935d-f084b26a5c8c

I also see that the img-cache is created successfully.

root@scspo2853621001:/opt/stack/data/cinder/mnt/99f446539c9f7c9374044cec5d11981a# ls
8107ad2a-89ec-4241-8ce9-cb51340c51b3 volume-42efa05a-6416-4ef4-ac67-ac37ca875efc volume-7f4e04c3-5c50-4f31-a16c-0c1be8e6d5fd
a8d760cd-8ba0-4969-ae2b-1a8c566223eb volume-44d52221-8d25-4735-ac4e-d6a58f418295 volume-8100d537-92c4-4e9d-81f0-e16ee85b7e0f
img-cache-d4ed4356-cfeb-474d-b254-90a0501352c9 volume-4f308896-11a4-4748-b2ce-080bf9bc2578 volume-a60a63a0-578b-48ba-90ab-789d24b4a3c7
volume-092e044d-ce2c-4f66-a225-ec2b1df89fd9 volume-4fefa610-b056-4ba5-a999-7453f269275a volume-af801dd8-43f4-44bb-88b2-8b1b325f0720
volume-11f3baf7-d11d-4d4a-a583-1e230c66f01f volume-535d5bf1-9322-4a7d-af32-91a363ab8cdc volume-bf342b6b-9175-4238-8cc6-242f562cab78
volume-12bf8500-1d77-439c-a0d7-9d3dc3031770 volume-53661b03-07f2-408f-9b16-61dfb8e99e2d volume-c75b91c9-3f19-4ac9-b7b2-9177aec83b0e
volume-2243360e-c29a-4036-9ec1-6d9fcc966519 volume-597176a1-be2f-41f3-9aa2-b14b5ffbbffe volume-e469a1e5-1420-485d-b47e-b351b90d6e80
volume-34b23b72-b039-4523-bdd0-598b5a9d0c18 volume-5a9f7d9a-3a4a-4cca-9a72-82d804988608 volume-ebb9f74c-7d17-49dd-afe7-68c1dcba9339
volume-3842a0e6-5590-4306-a477-cd32405c861a volume-6c7f7382-54f9-4214-971d-e8606e1ac6ab volume-f98bc97b-1a82-4cd9-8811-6b3ece3d76aa
volume-3c77da2d-510e-47a2-91a8-4b87e2d601e9 volume-6f4440a4-41cf-4abf-9850-2f35f9c6d8e8
volume-3ce2f35b-4112-460a-a75b-ec5772ec4a24 volume-7863bb84-f898-4695-97c9-2f600eb6c01a
root@scspo2853621001:/opt/stack/data/cinder/mnt/99f446539c9f7c9374044cec5d11981a#

Logs:
DEBUG cinder.volume.drivers.netapp.dataontap.nfs_...

[Dilip kumar (dilipkm)]

@saravanan

cinder mounts the nfs shares on mgmt node under /var/lib/cinder/mnt. mount point is a hash of nfs_share defined under cinder configuration. whenever we use DNS name on cinder configuration we noticed that optimized copy fails in final stage when it moves the copied file to image-cache.

Error message seen with debug log:
5d21488280e46a323ffbbccc bb6aa38d5d21488280e46a323ffbbccc] Exception moving file /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/9dc432e5-1194-4557-9d68-1e4df61f85bb. Message - Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mv /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/9dc432e5-1194-4557-9d68-1e4df61f85bb /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/img-cache-2d82f1f3-39d8-4987-bc6b-e05761a2b0bf
Exit code: 1
Stdout: ''
Stderr: "/bin/mv: failed to access '/var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/img-cache-2d82f1f3-39d8-4987-bc6b-e05761a2b0bf': Not a directory\n": oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2023-05-10 12:54:36.818 89854 WARNING cinder.volume.drivers.netapp.dataontap.nfs_base [req-3b6e8aac-2812-4a36-9ddf-948461233ddc ce168bccd20cfa5415ab4783cfa08b0e7650f425e377586ae3cd619136619bb8 0bffbc5387194999916d755c0548e212 - bb6aa38d5d21488280e46a323ffbbccc bb6aa38d5d21488280e46a323ffbbccc] Exception moving file /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/037c9ec0-d18d-4b19-b9cc-127ac3ad1b10. Message - Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mv /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/037c9ec0-d18d-4b19-b9cc-127ac3ad1b10 /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/img-cache-2d82f1f3-39d8-4987-bc6b-e05761a2b0bf
Exit code: 1
Stdout: ''
Stderr: "/bin/mv: failed to access '/var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db/img-cache-2d82f1f3-39d8-4987-bc6b-e05761a2b0bf': Not a directory\n": oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.

[root@svl-fab7-lab1-a-mgmt-001 mnt]# ls -l /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db
-rw-rw-rw-. 1 root root 0 May 10 12:54 /var/lib/cinder/mnt/f838b85018cf425234b4fa41df09d6db
[root@svl-fab7-lab1-a-mgmt-001 mnt]#

above error is from below line:
675 self._move_nfs_file(dst_img_local, dst_img_cache_local)

dst_share is used for determining the dir_path for the nfs mount and below function uses dst_share to determine the hash. Since the nfs_share is different (ip vs name) hash generated is different and hence move operation fails.

640 dst_dir = self._get_mount_point_for_share(dst_share)
641 dst_img_local = os.path.join(dst_dir, tmp_img_file)