Operations frequently timeout waiting for iDRAC to become ready

Bug #1718277 reported by Christopher Dearborn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-dracclient
Fix Released
High
Christopher Dearborn

Bug Description

When executing any operation against an iDRAC, python-dracclient will frequently timeout waiting for the iDRAC to become ready and error out.

A typical error looks like this:

2017-09-18 19:24:25.359 875 ERROR ironic.drivers.modules.drac.job [req-ee14d494-692e-4834-ae8a-90456a0b8f34 - - - - -] DRAC driver failed to get the list of unfinished jobs for node d409c4bb-9cf9-4715-9a5a-d56e0c4e4f86. Reason: DRAC operation failed. Messages: Timed out waiting for the iDRAC to become ready.
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base [req-ee14d494-692e-4834-ae8a-90456a0b8f34 - - - - -] vendor_passthru failed with method list_unfinished_jobs
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base Traceback (most recent call last):
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base File "/usr/lib/python2.7/site-packages/ironic/drivers/base.py", line 690, in passthru_handler
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base return func(*args, **kwargs)
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/drac/vendor_passthru.py", line 128, in list_unfinished_jobs
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base jobs = drac_job.list_unfinished_jobs(task.node)
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/drac/job.py", line 82, in list_unfinished_jobs
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base raise exception.DracOperationError(error=exc)
2017-09-18 19:24:25.360 875 ERROR ironic.drivers.base DracOperationError: DRAC operation failed. Reason: DRAC operation failed. Messages: Timed out waiting for the iDRAC to become ready

Changed in python-dracclient:
assignee: nobody → Christopher Dearborn (cdearbor)
status: New → In Progress
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-dracclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/505398

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-dracclient (master)

Reviewed: https://review.openstack.org/505398
Committed: https://git.openstack.org/cgit/openstack/python-dracclient/commit/?id=8f368ed3e1f94003e9faa5c647dd9d568a6fea60
Submitter: Jenkins
Branch: master

commit 8f368ed3e1f94003e9faa5c647dd9d568a6fea60
Author: Christopher Dearborn <email address hidden>
Date: Tue Sep 19 16:23:48 2017 -0400

    Increased is-iDRAC-ready timeout

    The current is-iDRAC-ready timeout is set to 24 iterations times a
    delay of 10 seconds, or 4 minutes. This timeout is too short for 12G
    and 13G hardware as operations against the iDRAC will frequently
    timeout if the iDRAC is not in a ready state.

    This patch increases the number of iterations to 48, which bumps the
    timeout up to 8 minutes. This increased timeout is sufficient for the
    iDRAC on 14G, 13G, and 12G hardware to become ready and therefore avoid
    a timeout exception.

    Change-Id: I9dc476d3c28475919404077875f04fd006d4464a
    Closes-Bug: 1718277

Changed in python-dracclient:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.