Ironic gate jobs are randomly timing out

Bug #1576100 reported by Vasyl Saienko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Vasyl Saienko

Bug Description

From time to time Ironic gate jobs are timed out with TEMPEST_OS_TEST_TIMEOUT which is set to 900.
At the same time build and terminate server timeouts are set to 600. So in fact we cutting legitimate allowed time by TEMPEST_OS_TEST_TIMEOUT.

https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L339

2016-04-28 05:59:51.508 | Captured traceback:
2016-04-28 05:59:51.508 | ~~~~~~~~~~~~~~~~~~~
2016-04-28 05:59:51.508 | Traceback (most recent call last):
2016-04-28 05:59:51.508 | File "tempest/test.py", line 113, in wrapper
2016-04-28 05:59:51.508 | return f(self, *func_args, **func_kwargs)
2016-04-28 05:59:51.508 | File "/opt/stack/new/ironic/ironic_tempest_plugin/tests/scenario/test_baremetal_basic_ops.py", line 128, in test_baremetal_server_ops
2016-04-28 05:59:51.508 | self.terminate_instance()
2016-04-28 05:59:51.508 | File "/opt/stack/new/ironic/ironic_tempest_plugin/tests/scenario/baremetal_manager.py", line 180, in terminate_instance
2016-04-28 05:59:51.508 | interval=30)
2016-04-28 05:59:51.508 | File "/opt/stack/new/ironic/ironic_tempest_plugin/tests/scenario/baremetal_manager.py", line 94, in wait_provisioning_state
2016-04-28 05:59:51.509 | target_states=state, timeout=timeout, interval=interval)
2016-04-28 05:59:51.509 | File "/opt/stack/new/ironic/ironic_tempest_plugin/tests/scenario/baremetal_manager.py", line 86, in _node_state_timeout
2016-04-28 05:59:51.509 | if not tempest.test.call_until_true(check_state, timeout, interval):
2016-04-28 05:59:51.509 | File "tempest/test.py", line 888, in call_until_true
2016-04-28 05:59:51.509 | time.sleep(sleep_for)
2016-04-28 05:59:51.509 | File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 34, in sleep
2016-04-28 05:59:51.509 | hub.switch()
2016-04-28 05:59:51.509 | File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2016-04-28 05:59:51.509 | return self.greenlet.switch()
2016-04-28 05:59:51.509 | File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in run
2016-04-28 05:59:51.509 | self.wait(sleep_time)
2016-04-28 05:59:51.509 | File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 82, in wait
2016-04-28 05:59:51.510 | sleep(seconds)
2016-04-28 05:59:51.510 | File "/usr/local/lib/python2.7/dist-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
2016-04-28 05:59:51.510 | raise TimeoutException()
2016-04-28 05:59:51.510 | fixtures._fixtures.timeout.TimeoutException

Tags: gate
Revision history for this message
Dmitry Tantsur (divius) wrote :
summary: - Iornic gate jobs are timed out
+ Ironic gate jobs are randomly timing out
Changed in ironic:
status: New → Confirmed
importance: Undecided → High
tags: added: gate
Changed in ironic:
status: Confirmed → In Progress
assignee: nobody → Vasyl Saienko (vsaienko)
Revision history for this message
Vasyl Saienko (vsaienko) wrote :

The patch has been merged: https://review.openstack.org/#/c/305270/

Changed in ironic:
status: In Progress → Fix Committed
Dmitry Tantsur (divius)
Changed in ironic:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.