OpenStack Shared File Systems Service (Manila)

Can not delete share server cause of soft-deleted

Bug #1844596 reported by Eric Xie on 2019-09-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Low	Eric Xie	OpenStack Shared File Systems Service (Manila) train-rc1

Bug Description

When executing CLI 'manila share-server-delete', the status of share-server became 'error' after 300s.

Reproduce steps
1. Configure reclaim_instance_interval = 3600 in nova.conf
2. Create one share with generic driver
3. After share server is OK, execute manila delete and manila share-server-delete

Log:
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server [req-03b16a33-3827-4073-bcd6-c8ff5042170f 4740f066d77b4bcab8f7b4fe5a1c61d1 1fc145617d5e4e97bad32c98b5d9b7c0 - - -] Exception during message handling: ServiceInstanceException: Instance 'b0d4cef8-f0c6-4a2a-a62b-0c03dfd9884f' has not been deleted in 300s. Giving up.
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/manager.py", line 187, in wrapped
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/utils.py", line 589, in wrapper
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/manager.py", line 3593, in delete_share_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server _wrapped_delete_share_server()
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 277, in inner
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/manager.py", line 3589, in _wrapped_delete_share_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server context, server_id, {'status': constants.STATUS_ERROR})
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/manager.py", line 3582, in _wrapped_delete_share_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server security_services=security_services)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/driver.py", line 1048, in teardown_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server return self._teardown_server(*args, **kwargs)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/drivers/generic.py", line 903, in _teardown_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server self.admin_context, server_details)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py", line 624, in delete_service_instance
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server self._delete_server(context, instance_id)
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py", line 394, in _delete_server
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server 'id': server_id, 's': self.max_time_to_build_instance})
2019-09-19 08:59:46.473 22963 ERROR oslo_messaging.rpc.server ServiceInstanceException: Instance 'b0d4cef8-f0c6-4a2a-a62b-0c03dfd9884f' has not been deleted in 300s. Giving up.

Versions:
$ apt list --installed |grep manila

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

manila-common/xenial,xenial,now 1:6.0.2-2~u16.04 all [installed,automatic]
manila-share/xenial,xenial,now 1:6.0.2-2~u16.04 all [installed]
python-manila/xenial,xenial,now 1:6.0.2-2~u16.04 all [installed,automatic

Tags:

Eric Xie (eric-xie) on 2019-09-19

Changed in manila:
assignee:	nobody → Eric Xie (eric-xie)
status:	New → Confirmed

Tom Barron (tpb) on 2019-09-19

tags:

added: driver generic

Eric Xie (eric-xie) on 2019-09-20

Changed in manila:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-21: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/683767

Goutham Pacha Ravi (gouthamr) on 2019-09-25

Changed in manila:
importance:	Undecided → Low
milestone:	none → train-rc1
tags:	added: queens-backport-potential rc1-potential rocky-backport-potential stein-backport-potential

Goutham Pacha Ravi (gouthamr) on 2019-09-26

tags:

added: train-rc1-potential
removed: rc1-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-26: Fix merged to manila (master)

Reviewed: https://review.opendev.org/683767
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=6b1a5e0119115e78c98d9cfb4ec238ac20280225
Submitter: Zuul
Branch: master

commit 6b1a5e0119115e78c98d9cfb4ec238ac20280225
Author: ericxiett <eric_xiett@163.com>
Date: Sat Sep 21 15:40:28 2019 +0800

Fix timeout when compute server was soft-deleted

    When set 'reclaim_instance_interval' larger than 0, share server
    was deleted cause that the server exists until
    reclaim_instance_interval. This patch checks status 'soft_deleted'
    of instance to avoid the timeout.

Change-Id: Ib481c1699b19ee8c37473081f1aec780ed384e9d
Closes-Bug: #1844596

Changed in manila:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-26: Fix included in openstack/manila 9.0.0.0rc1

This issue was fixed in the openstack/manila 9.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.