For a variety of operations, the VMAX cinder driver invokes a CIM method, and then must periodically poll for completion of the backend submitted job. To accomplish this polling, the FixedIntervalLoopingCall cinder utility construct is used by both emc_vmax_utils.py-->_wait_for_job_complete() and emc_vmax_utils.py-->wait_for_sync().
However, the logic uses global class variables in order to track a retry count to a configured maximum, rather than using thread local variables. Because of the way the EMCVMAXUtils class objects get constructed, there exists 4 of these objects hanging off of common, masking, provision, and masking.provision, so there is some isolation between driver areas, but not within an area (masking, for example).
Here is one of the problematic methods:
def wait_for_sync(self, conn, syncName):
"""Given the sync name wait for it to fully synchronize.
:param conn: connection to the ecom server
:param syncName: the syncName
"""
def _wait_for_sync():
"""Called at an interval until the synchronization is finished"""
if self._is_sync_complete(conn, syncName):
raise loopingcall.LoopingCallDone()
if self.retries > JOB_RETRIES: <----- **************
LOG.error(_("_wait_for_sync failed after %(retries)d tries")
% {'retries': self.retries})
raise loopingcall.LoopingCallDone()
try:
self.retries += 1 <----- **************
if not self.wait_for_sync_called:
if self._is_sync_complete(conn, syncName):
self.wait_for_sync_called = True <----- **************
except Exception as e:
LOG.error(_("Exception: %s") % six.text_type(e))
exceptionMessage = (_("Issue encountered waiting for "
"synchronization."))
LOG.error(exceptionMessage)
raise exception.VolumeBackendAPIException(exceptionMessage)
self.retries = 0 <----- **************
self.wait_for_sync_called = False <----- **************
timer = loopingcall.FixedIntervalLoopingCall(_wait_for_sync)
timer.start(interval=INTERVAL_10_SEC).wait()
The intent may have been to wrapper the nested method in an outer class, but instead it is just a parent method. Concurrent operations can cause self.retries to be bumped up or zeroed out incorrectly, resulting in hard-to-debug failures or timeouts.
Fix proposed to branch: master
Review: https:/