commit e3feb0297dfece1fc198b231d9aa39ae2920d877
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 16 08:39:07 2014 -0700
Log a warning when getting lvs and vgs takes longer than 60 seconds
We know something is causing lvs/vgs commands to block while deleting a
volume and this is causing Tempest to timeout while waiting for the
volume to be deleted. What we don't have right now is very good
(specific) logging when this happens, unless we get messages in syslog
for lvm tasks taking more than 120 seconds, but that doesn't always
happen when we see the volume delete timeout in Tempest.
This patch adds a check for when getting logical volumes and volume
groups takes longer than 60 seconds and logs a warning if that happens.
This is helpful in production also because the default interval for
periodic tasks is 60 seconds so having these take longer than that time
could cause periodic tasks to block up on each other and you'll get
warnings from the FixedIntervalLoopingCall in oslo which is controlling
the task runs.
Reviewed: https:/ /review. openstack. org/128962 /git.openstack. org/cgit/ openstack/ cinder/ commit/ ?id=e3feb0297df ece1fc198b231d9 aa39ae2920d877
Committed: https:/
Submitter: Jenkins
Branch: master
commit e3feb0297dfece1 fc198b231d9aa39 ae2920d877
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 16 08:39:07 2014 -0700
Log a warning when getting lvs and vgs takes longer than 60 seconds
We know something is causing lvs/vgs commands to block while deleting a
volume and this is causing Tempest to timeout while waiting for the
volume to be deleted. What we don't have right now is very good
(specific) logging when this happens, unless we get messages in syslog
for lvm tasks taking more than 120 seconds, but that doesn't always
happen when we see the volume delete timeout in Tempest.
This patch adds a check for when getting logical volumes and volume opingCall in oslo which is controlling
groups takes longer than 60 seconds and logs a warning if that happens.
This is helpful in production also because the default interval for
periodic tasks is 60 seconds so having these take longer than that time
could cause periodic tasks to block up on each other and you'll get
warnings from the FixedIntervalLo
the task runs.
Related-Bug: #1373513
Change-Id: I7ac5ddbbd9bda7 744db33f0bb8d56 879301f5538