[openstack-kolla][cinder][netapp] With snaphot visibility cinder-volume doesn't start

Bug #1943532 reported by andrea gargano
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Confirmed
Undecided
Michal Nasiadka

Bug Description

hi all,
I want to ask you about our openstack-kolla version is 12.2.0, we have netapp as backend, if we enable the visibility of the .snapshot on Netapp side, cinder_volume doesn't start correctly,it remains in Restarting.
Here you can find a piece of docker logs cinder_volume:

ERROR:__main__:Failed to change ownership of /var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot to 42407:42407
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 360, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot'
....
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-f6bdbf8b-4186-4229-b9d9-356760471ac3': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/img-cache-083c7e2b-ec22-4e47-a77c-df3e41f22ef6': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-825acbe4-8e0e-463d-9141-7ad03fcc429e': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/volume-eeea7cb3-7bf5-463f-ad25-96cb8d6d567b': Read-only file system
chown: changing ownership of '/var/lib/cinder/mnt/55fcb53b884ca983ed3d6fa8aac57810/.snapshot/hourly.2021-09-14_0405/img-cache-27cabece-236c-44d2-a3e9-2f7f141d41df': Read-only file system

Thank you in advance.

Andrea

Revision history for this message
ignazio (cassano) wrote :

Hello,
I would like to add that the only workaround found is to disable the visibility of snapshots on netapp.
We have also talked to netapp support and they told that in a devstack environment this problem does not occur .
Ignazio

Revision history for this message
ignazio (cassano) wrote :

PS
the kolla-ansible version is 12.2.0
Ignazio

description: updated
Revision history for this message
ignazio (cassano) wrote :

Hello, as you can see in the logs, probably the issue is caused by /usr/local/bin/kolla_set_configs.
It is not a component of netapp cinder driver.
It tries to change the owner of .snapshots directories and it is not possible because on netapp ONTAP they are read only.
Ignazio

Revision history for this message
ignazio (cassano) wrote :
Download full text (29.5 KiB)

This is our globals.yml:

---
# You can use this file to override _any_ variable throughout Kolla.
# Additional options can be found in the
# 'kolla-ansible/ansible/group_vars/all.yml' file. Default value of all the
# commented parameters are shown here, To override the default value uncomment
# the parameter and change its value.

###############
# Kolla options
###############
# Valid options are [ COPY_ONCE, COPY_ALWAYS ]
config_strategy: "COPY_ALWAYS"

# Valid options are ['centos', 'debian', 'rhel', 'ubuntu']
kolla_base_distro: "ubuntu"

# Valid options are [ binary, source ]
kolla_install_type: "source"

# Do not override this unless you know what you are doing.
openstack_release: "wallaby"

# Docker image tag used by default.
#openstack_tag: "{{ openstack_release ~ openstack_tag_suffix }}"

# Suffix applied to openstack_release to generate openstack_tag.
openstack_tag_suffix: "13.09.2021"

# Location of configuration overrides
node_custom_config: "/etc/kolla/config"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. If you want to run an
# All-In-One without haproxy and keepalived, you can set enable_haproxy to no
# in "OpenStack options" section, and set this value to the IP of your
# 'network_interface' as set in the Networking section below.
kolla_internal_vip_address: "10.102.119.194"

# This is the DNS name that maps to the kolla_internal_vip_address VIP. By
# default it is the same as kolla_internal_vip_address.
#kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. It defaults to the
# kolla_internal_vip_address, allowing internal and external communication to
# share the same address. Specify a kolla_external_vip_address to separate
# internal and external requests between two VIPs.
kolla_external_vip_address: "10.102.184.190"

# The Public address used to communicate with OpenStack as set in the public_url
# for the endpoints that will be created. This DNS name should map to
# kolla_external_vip_address.
kolla_external_fqdn: "{{ kolla_external_vip_address }}"

# Optionally change the path to sysctl.conf modified by Kolla Ansible plays.
#kolla_sysctl_conf_path: /etc/sysctl.conf

################
# Docker options
################

# Custom docker registry settings:
docker_registry: "10.102.184.160:4000"
docker_registry_insecure: "yes"
#docker_registry:
#docker_registry_insecure: "{{ 'yes' if docker_registry else 'no' }}"
#docker_registry_username:
# docker_registry_password is set in the passwords.yml file.

# Namespace of images:
#docker_namespace: "kolla"

# Docker client timeout in seconds.
#docker_client_timeout: 120

#docker_configure_for_zun: "no"
#containerd_configure_for_zun: "no"
#containerd_grpc_gid: 42463

###################
# Messaging options
###################
# Below is an example of an separate backend that provides brokerless
# messaging for oslo.messaging RPC communications

#om_rpc_transport: "amqp"
#om_rpc_user: "{{ qdrouterd_user }}"
#om_rpc_password: "{{ qdrouterd_password }}"
#om_rpc_port: "...

Revision history for this message
ignazio (cassano) wrote :

This is our /etc/kolla/config/cinder.conf:

DEFAULT]
host = cinder-cluster1
enabled_backends=nfsnetapp, rbd-1
#backup_mount_point_base = /backup_mount

default_volume_type = nfsnetapp
#backup_mount_point_base = /var/lib/cinder/ppp

[nfsnetapp]
netapp_enable_multiattach = True
nas_secure_file_operations = false
nas_secure_file_permissions = false
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_storage_family = ontap_cluster
netapp_storage_protocol = nfs
netapp_vserver = netapp-openstack-tst2
netapp_server_hostname = clus-fas.csi.it
netapp_server_port = 80
netapp_login = admin
netapp_password = *****
nfs_shares_config = /etc/cinder/nfs_shares
volume_backend_name = nfsnetapp
#nfs_mount_options = lookupcache=pos
nfs_mount_options = nfsvers=3,lookupcache=pos
image_upload_use_internal_tenant = True
image_volume_cache_enabled = True

Revision history for this message
ignazio (cassano) wrote :

This is our /etc/kolla/config/nfs_shares:

10.102.189.182:/netappopenstacktst2_cinder2

Changed in kolla-ansible:
assignee: nobody → Michal Nasiadka (mnasiadka)
Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

Seems we shouldn't be changing ownership recursively on whole /var/lib/cinder - and it seems it was just a fix for this bug 5 years ago: https://bugs.launchpad.net/kolla/+bug/1631503
I'll remove the recursive chown on container startup.

Changed in kolla-ansible:
status: New → Confirmed
Revision history for this message
andrea gargano (andrea-ga) wrote :

Hi,
we had tried to delete recursive option on cinder json file template and it works well.

Andrea

Revision history for this message
andrea gargano (andrea-ga) wrote :

Hi,
when will be released the patch?which vesion of kolla-ansible?
Thank you.

Andrea

Revision history for this message
ignazio (cassano) wrote :

Please, is it possible to know in which kolla-ansible version the patch will be applied?
Thanks
Ignazio

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers