[openstack-kolla][cinder][netapp] With snaphot visibility cinder-volume doesn't start

[andrea gargano]

hi all,
I want to ask you about our openstack-kolla version is 12.2.0, we have netapp as backend, if we enable the visibility of the .snapshot on Netapp side, cinder_volume doesn't start correctly,it remains in Restarting.
Here you can find a piece of docker logs cinder_volume:

ERROR:__main__:Failed to change ownership of /var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot to 42407:42407
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 360, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot'
....
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-f6bdbf8b-4186-4229-b9d9-356760471ac3': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/img-cache-083c7e2b-ec22-4e47-a77c-df3e41f22ef6': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-825acbe4-8e0e-463d-9141-7ad03fcc429e': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-eeea7cb3-7bf5-463f-ad25-96cb8d6d567b': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/img-cache-27cabece-236c-44d2-a3e9-2f7f141d41df': Read-only file system

Thank you in advance.

Andrea

[ignazio (cassano)]

Hello,
I would like to add that the only workaround found is to disable the visibility of snapshots on netapp.
We have also talked to netapp support and they told that in a devstack environment this problem does not occur .
Ignazio

[ignazio (cassano)]

PS
the kolla-ansible version is 12.2.0
Ignazio

[ignazio (cassano)]

Hello, as you can see in the logs, probably the issue is caused by /usr/local/bin/kolla_set_configs.
It is not a component of netapp cinder driver.
It tries to change the owner of .snapshots directories and it is not possible because on netapp ONTAP they are read only.
Ignazio

[ignazio (cassano)]

This is our globals.yml:

---
# You can use this file to override _any_ variable throughout Kolla.
# Additional options can be found in the
# 'kolla-ansible/ansible/group_vars/all.yml' file. Default value of all the
# commented parameters are shown here, To override the default value uncomment
# the parameter and change its value.

###############
# Kolla options
###############
# Valid options are [ COPY_ONCE, COPY_ALWAYS ]
config_strategy: "COPY_ALWAYS"

# Valid options are ['centos', 'debian', 'rhel', 'ubuntu']
kolla_base_distro: "ubuntu"

# Valid options are [ binary, source ]
kolla_install_type: "source"

# Do not override this unless you know what you are doing.
openstack_release: "wallaby"

# Docker image tag used by default.
#openstack_tag: "{{ openstack_release ~ openstack_tag_suffix }}"

# Suffix applied to openstack_release to generate openstack_tag.
openstack_tag_suffix: "13.09.2021"

# Location of configuration overrides
node_custom_config: "/etc/kolla/config"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. If you want to run an
# All-In-One without haproxy and keepalived, you can set enable_haproxy to no
# in "OpenStack options" section, and set this value to the IP of your
# 'network_interface' as set in the Networking section below.
kolla_internal_vip_address: "10.102.119.194"

# This is the DNS name that maps to the kolla_internal_vip_address VIP. By
# default it is the same as kolla_internal_vip_address.
#kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. It defaults to the
# kolla_internal_vip_address, allowing internal and external communication to
# share the same address. Specify a kolla_external_vip_address to separate
# internal and external requests between two VIPs.
kolla_external_vip_address: "10.102.184.190"

# The Public address used to communicate with OpenStack as set in the public_url
# for the endpoints that will be created. This DNS name should map to
# kolla_external_vip_address.
kolla_external_fqdn: "{{ kolla_external_vip_address }}"

# Optionally change the path to sysctl.conf modified by Kolla Ansible plays.
#kolla_sysctl_conf_path: /etc/sysctl.conf

################
# Docker options
################

# Custom docker registry settings:
docker_registry: "10.102.184.160:4000"
docker_registry_insecure: "yes"
#docker_registry:
#docker_registry_insecure: "{{ 'yes' if docker_registry else 'no' }}"
#docker_registry_username:
# docker_registry_password is set in the passwords.yml file.

# Namespace of images:
#docker_namespace: "kolla"

# Docker client timeout in seconds.
#docker_client_timeout: 120

#docker_configure_for_zun: "no"
#containerd_configure_for_zun: "no"
#containerd_grpc_gid: 42463

###################
# Messaging options
###################
# Below is an example of an separate backend that provides brokerless
# messaging for oslo.messaging RPC communications

#om_rpc_transport: "amqp"
#om_rpc_user: "{{ qdrouterd_user }}"
#om_rpc_password: "{{ qdrouterd_password }}"
#om_rpc_port: "...

[ignazio (cassano)]

This is our /etc/kolla/config/cinder.conf:

DEFAULT]
host = cinder-cluster1
enabled_backends=nfsnetapp, rbd-1
#backup_mount_point_base = /backup_mount

default_volume_type = nfsnetapp
#backup_mount_point_base = /var/lib/cinder/ppp

[nfsnetapp]
netapp_enable_multiattach = True
nas_secure_file_operations = false
nas_secure_file_permissions = false
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_storage_family = ontap_cluster
netapp_storage_protocol = nfs
netapp_vserver = netapp-openstack-tst2
netapp_server_hostname = clus-fas.csi.it
netapp_server_port = 80
netapp_login = admin
netapp_password = *****
nfs_shares_config = /etc/cinder/nfs_shares
volume_backend_name = nfsnetapp
#nfs_mount_options = lookupcache=pos
nfs_mount_options = nfsvers=3,lookupcache=pos
image_upload_use_internal_tenant = True
image_volume_cache_enabled = True

[ignazio (cassano)]

This is our /etc/kolla/config/nfs_shares:

10.102.189.182:/netappopenstacktst2_cinder2

[Michal Nasiadka (mnasiadka)]

Seems we shouldn't be changing ownership recursively on whole /var/lib/cinder - and it seems it was just a fix for this bug 5 years ago: https://bugs.launchpad.net/kolla/+bug/1631503
I'll remove the recursive chown on container startup.

[andrea gargano (andrea-ga)]

Hi,
we had tried to delete recursive option on cinder json file template and it works well.

Andrea

[andrea gargano (andrea-ga)]

Hi,
when will be released the patch?which vesion of kolla-ansible?
Thank you.

Andrea

[ignazio (cassano)]

Please, is it possible to know in which kolla-ansible version the patch will be applied?
Thanks
Ignazio