cinder should not fail to delete a volume already deleted on zfssa
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Undecided
|
John Griffith |
Bug Description
After running some rally scenarios on my main openstack (Solaris, Juno, mix of x86 & sparc, zfssa for storage) I noticed in the admin panel in horizon that several volumes created by the test were still present but in an ERROR_DELETING state.
Setting these to AVAILABLE and trying to delete them manually failed and they changed back to the ERROR_DELETING state.
The volumes were created on an attached ZFSSA (7320) however the volumes were no longer present on that device.
Looking in the cinder-volume log on the host that has the zfssa cinder driver we find a trace of the error:
2015-07-07 22:35:48.599 19035 INFO cinder.
2015-07-07 22:35:50.231 19035 ERROR cinder.
2015-07-07 22:35:50.262 19035 ERROR cinder.
2015-07-07 22:35:50.315 19035 ERROR oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:50.315 19035 TRACE oslo.messaging.
2015-07-07 22:35:55.551 19035 INFO cinder.
So there may be a race here somewhere where the volume got deleted but cinder still has it.
Anyway, it seems that we should handle this a bit better. I'd suggest in this particular case where we have established that connection to the zfssa is working correctly that instead of throwing an exception that will cause a stack trace and an ERROR_DELETING state and have the user have to hunt through logs that zfssarest detects that the volume is not present (assumed already deleted) and allows cinder to delete it from its database of current volumes.
tags: | added: zfssa |
tags: | added: drivers |
Changed in cinder: | |
assignee: | nobody → John Griffith (john-griffith) |
status: | New → In Progress |
description: | updated |
Changed in cinder: | |
milestone: | none → liberty-2 |
status: | Fix Committed → Fix Released |
Changed in cinder: | |
milestone: | liberty-2 → 7.0.0 |
Reviewed: https:/ /review. openstack. org/199330 /git.openstack. org/cgit/ openstack/ cinder/ commit/ ?id=78e25a577fa 4c5e47c871c4358 98c847423d7ac6
Committed: https:/
Submitter: Jenkins
Branch: master
commit 78e25a577fa4c5e 47c871c435898c8 47423d7ac6
Author: John Griffith <email address hidden>
Date: Tue Jul 7 16:26:21 2015 -0600
Handle volume not found on zfssa volume delete
The zfssa driver doesn't handle cases where a volume may not
exist on the backend when a delete call is sent. The result
is a stack-trace in the logs, sometimes a crash of the service
and an inability to clean up volumes from Cinder's perspective.
This patch doesn't address the root cause of the issue, but
it does handle the case more gracefully by catching the
exception, logging an error and returning.
This way the volume can be cleaned up on the Cinder side and
the operator still has an indication that something went wrong.
This is a common pattern for most of the drivers in Cinder.
Change-Id: I09725b29effb79 450d010949527bd 54329919f52
Closes-Bug: #1472412