cinder can not create volume on NetApp NFS

[[e-mail]]

Using latest Mirantis FUEL 7.0 (kilo) - date 16-06-30
Using fuel-plugin-cinder-netapp, branch 7.0, V3.2.1, commit 41e161f2b42caf0c11cb4c1f20cf0791c421a0fe

3x Controller Node, Ceilomenter
1x Cinder Node
1x Compute Node

Plugin parameters:
Cinder and NetApp integration
Multibackend enabled
OnTap 7mode
Storage protocol NFS
Username: root
Password: correct one:-)
NetApp IP: 172.16.11.31 (Controller)
NetApp Server Port: 80

Transport Type: http
NFS Server: 172.21.22.252
Shares: 1
Share: /vol/vol1

After successful deployment of this environment, the "Health Check" function from FUEL fails at "Create volume and boot instance from it"

::::
Looking into /var/log/cinder-all.log on cinder storage node indicates a "permission denied":

2016-07-01 08:04:02.660 4473 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume volume-5431fcb2-efff-4788-9a27-0fd878b625d3 could not be created on shares.

::::
Creating a file on NFS mount as "root" on cinder node is successful:

# touch /var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf/foobar
# ls /var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf/foobar
/var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf/foobar

::
Creating a file on NFS mount as user "cinder" fails:

# sudo -u cinder touch /var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf/barfoo
touch: cannot touch '/var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf/barfoo': Permission denied

[Daniel Roesen (daniel-roesen)]

The problem is posix file ownership problem:

root@node-30:/var/lib/cinder/mnt# mount | grep nfs
172.21.22.252:/vol/vol1 on /var/lib/cinder/mnt/bfe0522da8ec08bea42affcc3f7831bf type nfs (rw,vers=4,addr=172.21.22.252,clientaddr=172.21.43.4)

root@node-30:/var/lib/cinder/mnt# ll
total 16
drwxr-xr-x 3 cinder cinder 4096 Jun 20 17:56 ./
drwxr-xr-x 4 cinder cinder 4096 Jun 20 17:56 ../
drwxrwxr-x 59 nobody 4294967294 8192 Jul 5 20:43 bfe0522da8ec08bea42affcc3f7831bf/

I guess earlier, cinder was running as root, so permissions on the mounted NFS share didn't matter.
Now that cinder runs as user "cinder", creation of files on an NFS share with ownership user "nobody"
and group "$maxint32" doesn't work when the mount point has permissions 0775, as user "cinder" is part of "other" and thus not permitted to write (create a file).

We have no older working installations and can only guess that in earlier versions (FUEL 6.1, Juno) cinder was running as root... thus the weird permissions on the NFS share didn't matter.

So - unclear what the proper fix is. Beating the NetApp share into "fitting" permissions for the share prolly isn't a sensible path. A better way would be for the cinder node to mount the NFS share with NFS mount options forcing the NFS share's permissions to be UID=cinder (and prolly GID=cinder as well). This way, it doesn't matter what the NFS filesystem's original ownership are, but locally on the cinder node all NFS share files/folders are "owned" by the local UNIX "cinder" user under which cinder as a service operates.

Best regards,
Daniel

[Daniel Roesen (daniel-roesen)]

Might be related to:
https://blueprints.launchpad.net/cinder/+spec/secure-nfs