OpenStack Shared File Systems Service (Manila)

[NetApp driver] Race condition between setup_server and teardown

Bug #1816640 reported by Carlos Eduardo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Carlos Eduardo
OpenStack Shared File Systems Service (Manila) stein-3

Bug Description

NetApp driver is leaving leftover VLAN ports in the backend after a CI run. There are also errors when trying to delete an IPspace. It seems there are race conditions between the setup_server and teardown_server in the driver.

Steps:
Unable to reproduce this issue locally, but this issue happens a lot in the CI.

Stacktrace:
Feb 19 15:06:30.181067 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR manila.share.manager [None req-66472ca9-3edb-4c0e-9932-8be2bdabad9a None None] Creation of share instance d2c5e394-75c4-4016-ae9e-8840d70e000f failed: failed to get share server.: NaApiError: NetApp API failed. Reason - 18603:Specified IPspace not found
Feb 19 15:06:30.209663 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server [None req-66472ca9-3edb-4c0e-9932-8be2bdabad9a None None] Exception during message handling: NaApiError: NetApp API failed. Reason - 18603:Specified IPspace not found
Feb 19 15:06:30.209796 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Feb 19 15:06:30.209943 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Feb 19 15:06:30.210068 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Feb 19 15:06:30.210178 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Feb 19 15:06:30.210283 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Feb 19 15:06:30.210398 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Feb 19 15:06:30.210514 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Feb 19 15:06:30.210623 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 187, in wrapped
Feb 19 15:06:30.210725 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Feb 19 15:06:30.210847 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Feb 19 15:06:30.210989 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Feb 19 15:06:30.211096 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 1724, in create_share_instance
Feb 19 15:06:30.211207 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server detail=message_field.Detail.NO_SHARE_SERVER)
Feb 19 15:06:30.211303 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 19 15:06:30.211430 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self.force_reraise()
Feb 19 15:06:30.211533 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 19 15:06:30.211642 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Feb 19 15:06:30.211741 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 1706, in create_share_instance
Feb 19 15:06:30.211852 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server share_group=share_group_ref,
Feb 19 15:06:30.211962 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 626, in _provide_share_server_for_share
Feb 19 15:06:30.212062 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return _wrapped_provide_share_server_for_share()
Feb 19 15:06:30.212175 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Feb 19 15:06:30.212272 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Feb 19 15:06:30.212372 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 622, in _wrapped_provide_share_server_for_share
Feb 19 15:06:30.212492 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server metadata=metadata))
Feb 19 15:06:30.212592 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 638, in _create_share_server_in_backend
Feb 19 15:06:30.212686 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server metadata=metadata)
Feb 19 15:06:30.212788 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 3519, in _setup_server
Feb 19 15:06:30.212895 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self.driver.deallocate_network(context, share_server['id'])
Feb 19 15:06:30.212995 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 19 15:06:30.213108 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self.force_reraise()
Feb 19 15:06:30.213204 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 19 15:06:30.213318 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Feb 19 15:06:30.213415 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 3470, in _setup_server
Feb 19 15:06:30.213523 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server network_info, metadata=metadata)
Feb 19 15:06:30.213636 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/driver.py", line 903, in setup_server
Feb 19 15:06:30.213732 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return self._setup_server(*args, **kwargs)
Feb 19 15:06:30.213873 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 115, in _setup_server
Feb 19 15:06:30.213976 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return self.library.setup_server(network_info, metadata)
Feb 19 15:06:30.214093 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.214211 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)

Feb 19 15:06:30.214308 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 139, in setup_server
Feb 19 15:06:30.214403 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return setup_server_with_lock()
Feb 19 15:06:30.214519 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Feb 19 15:06:30.214614 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Feb 19 15:06:30.214709 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 132, in setup_server_with_lock
Feb 19 15:06:30.214831 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self._create_vserver(vserver_name, network_info)
Feb 19 15:06:30.215047 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.215167 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.215267 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 209, in _create_vserver
Feb 19 15:06:30.215376 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server security_services=security_services)
Feb 19 15:06:30.215502 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 19 15:06:30.215608 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self.force_reraise()
Feb 19 15:06:30.215710 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 19 15:06:30.215821 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Feb 19 15:06:30.215926 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 187, in _create_vserver
Feb 19 15:06:30.216033 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server ipspace_name)
Feb 19 15:06:30.216133 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.216245 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.216356 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 250, in _create_vserver_lifs
Feb 19 15:06:30.216480 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server node_name, lif_name, network_allocation)
Feb 19 15:06:30.216590 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.216690 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.216792 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 330, in _create_lif
Feb 19 15:06:30.216936 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server lif_name, ipspace_name, mtu)
Feb 19 15:06:30.217048 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.217150 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.217251 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 607, in create_network_interface
Feb 19 15:06:30.217378 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server node, home_port_name, mtu, ipspace=ipspace_name)
Feb 19 15:06:30.217500 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.217617 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.217720 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 753, in _ensure_broadcast_domain_for_port
Feb 19 15:06:30.217854 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self._create_broadcast_domain(domain, ipspace, mtu)
Feb 19 15:06:30.217962 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 96, in trace_wrapper
Feb 19 15:06:30.218062 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Feb 19 15:06:30.218162 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 822, in _create_broadcast_domain
Feb 19 15:06:30.218267 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server self.send_request('net-port-broadcast-domain-create', api_args)
Feb 19 15:06:30.218368 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_base.py", line 88, in send_request
Feb 19 15:06:30.218551 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server return self.connection.invoke_successfully(request, enable_tunneling)
Feb 19 15:06:30.218658 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py", line 287, in invoke_successfully
Feb 19 15:06:30.218759 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server raise NaApiError(code, msg)
Feb 19 15:06:30.218870 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server NaApiError: NetApp API failed. Reason - 18603:Specified IPspace not found
Feb 19 15:06:30.219000 devstack-xenial-cdot-msvm-manila-provider-70-404176 manila-share[21030]: ERROR oslo_messaging.rpc.server

See original description
Tom Barron (tpb)
tags: added: driver netapp tests
Rodrigo Barbieri (rodrigo-barbieri2010)
Changed in manila:
status: New → Triaged
assignee: nobody → Rodrigo Barbieri (rodrigo-barbieri2010)
milestone: none → stein-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/638674

Changed in manila:
assignee: Rodrigo Barbieri (rodrigo-barbieri2010) → Carlos Eduardo (silvacarlose)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/638674
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=96d6869dd60c90231ad32426c7cd165cf670e61b
Submitter: Zuul
Branch: master

commit 96d6869dd60c90231ad32426c7cd165cf670e61b
Author: silvacarloss <email address hidden>
Date: Tue Feb 19 15:20:48 2019 -0300

    [NetApp] Fix race condition issue in NetApp driver

    This patch fixes the concurrency in the NetApp driver between the
    method setup_server and teardown_server by doing the
    teardown_server method lock the needed resources correctly.

    Closes-bug: #1816640
    Change-Id: I7d56384e77b7e18b3c95f182c0a01af82e5fa5a5

Changed in manila:
status: In Progress → Fix Released
riteaidpharmacy (ordertramadolonline)
summary: - [NetApp driver] Race condition between setup_server and teardown
+ Buy tramadol online at riteaidpharmacy[NetApp driver] Race condition
+ between setup_server and teardown
description: updated
summary: - Buy tramadol online at riteaidpharmacy[NetApp driver] Race condition
- between setup_server and teardown
+ [NetApp driver] Race condition between setup_server and teardown
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 8.0.0.0rc1

This issue was fixed in the openstack/manila 8.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.