Concurrency issues on rtstool (LIO)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Undecided
|
Gorka Eguileor | ||
Kilo |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Since rtslib has no synchronized access to configfs [1] it can happen that rtstool or rtslib try to access something that is no longer there because other Cinder request has removed the target.
These errors can happen on creating, attaching, detaching and deleting volumes, and resulting log messages are weird/misleading because the exception refers to a different volume-id from the one we were working on.
The reason why the exception is from another volume is because library was looping through all elements, and when it tried to access one to get its info to do a dump (for example) the info no longer exists and that's unexpected and raises an Exception.
I have located different errors that are originated from this same issue:
1)
TRACE oslo.messaging.
File "/bin/cinder-
sys.
File "/bin/cinder-
get_targets()
File "/bin/cinder-
for t, dump in _get_targets(
File "/bin/cinder-
return [(target, target.dump()) for target in rtsroot.targets]
File "/usr/lib/
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/
d[\'luns\'] = [lun.dump() for lun in self.luns]
File "/usr/lib/
(self.
File "/usr/lib/
return tcm.StorageObje
File "/usr/lib/
return so_mapping[
KeyError: \'lun\'
2)
Stderr: u'Traceback (most recent call last):
File "/bin/cinder-
sys.
File "/bin/cinder-
get_targets()
File "/bin/cinder-
print(
File "/usr/lib/
d[\'tpgs\'] = [tpg.dump() for tpg in self.tpgs]
File "/usr/lib/
d = super(TPG, self).dump()
File "/usr/lib/
params[item] = self.get_
File "/usr/lib/
return fread(path)
File "/usr/lib/
with open(path, \'r\') as file_fd:
IOError: [Errno 2] No such file or directory: \'/sys/
3)
2015-05-17 20:51:49.829 62162 ERROR cinder.
File "/bin/cinder-
sys.exit(main())
File "/bin/cinder-
delete(iqn)
File "/bin/cinder-
for x in rtsroot.
File "/usr/lib/
for so in StorageObject.
File "/usr/lib/
yield cls.so_
File "/usr/lib/
return mapping[
File "/usr/lib/
super(
File "/usr/lib/
self._backstore = _Backstore(name, type(self), mode)
File "/usr/lib/
(self._plugin, name))
rtslib.
The cleanest solution is synchronizing access in Cinder.
Changed in cinder: | |
assignee: | nobody → Gorka Eguileor (gorka) |
description: | updated |
Changed in cinder: | |
milestone: | none → liberty-1 |
status: | Fix Committed → Fix Released |
Changed in cinder: | |
milestone: | liberty-1 → 7.0.0 |
Fix proposed to branch: master /review. openstack. org/187260
Review: https:/