Kilo dsvm gate is mostly broken (timeout when waiting for active state)

[Dmitry Tantsur]

Most jobs finish with build timeout, see https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:stable/kilo. After quick look at logs I've figured out that ironic does manage to deploy an instance (http://logs.openstack.org/60/253460/4/check/gate-tempest-dsvm-ironic-pxe_ipa/5a7b303/logs/screen-ir-cond.txt.gz?level=INFO#_2016-01-06_08_42_04_396) but then the node seemingly fails to boot into instance (no signs of TFTP boot in http://logs.openstack.org/60/253460/4/check/gate-tempest-dsvm-ironic-pxe_ipa/5a7b303/logs/syslog.txt.gz).

[Matt Riedemann (mriedem)]

jroll recently bumped the ironic test timeout to 900 in devstack-gate:

https://github.com/openstack-infra/devstack-gate/commit/729ce7986050f377f3e6ff697725057dd525c5a9

This was an earlier change to increase the job timeout (which is actually less than the test timeout):

https://github.com/openstack-infra/devstack-gate/commit/4d9b178a87cc86569f60d271ad2b9a68ac50bdfd

[Matt Riedemann (mriedem)]

I don't see TEMPEST_OS_TEST_TIMEOUT getting into the stable/kilo job localrc here:

https://github.com/openstack-infra/devstack-gate/commit/729ce7986050f377f3e6ff697725057dd525c5a9

http://logs.openstack.org/65/248365/3/check/gate-tempest-dsvm-ironic-pxe_ssh/7070ac5/logs/localrc.txt.gz

[Matt Riedemann (mriedem)]

I guess that's not a localrc var, it's used when running tempest here:

https://github.com/openstack-infra/devstack-gate/blob/729ce7986050f377f3e6ff697725057dd525c5a9/devstack-vm-gate.sh#L751

And we know we're getting here:

https://github.com/openstack-infra/devstack-gate/blob/729ce7986050f377f3e6ff697725057dd525c5a9/devstack-vm-gate.sh#L761

Because of the logs:

http://logs.openstack.org/65/248365/3/check/gate-tempest-dsvm-ironic-pxe_ssh/7070ac5/console.html#_2016-01-06_12_55_46_033

But it's running with the default timeout of 500:

http://logs.openstack.org/65/248365/3/check/gate-tempest-dsvm-ironic-pxe_ssh/7070ac5/console.html#_2016-01-06_12_57_01_267

https://github.com/openstack/tempest/blob/master/.testr.conf#L4

So for some reason that variable isn't being used.

[Matt Riedemann (mriedem)]

Adding tempest since it looks like the devstack-gate code is doing the right thing, tempest just isn't running with the OS_TEST_TIMEOUT value passed in on the command line for some reason.

[Matt Riedemann (mriedem)]

Removing tempest since http://logs.openstack.org/65/248365/3/check/gate-tempest-dsvm-ironic-pxe_ssh/7070ac5/console.html#_2016-01-06_12_57_01_267 is just bash dumping .testr.conf which has the default 500 in it.

And the build_timeout is correctly set to 600 in tempest.conf, so it looks like ironic is just super duper slow on some nodes (this was an OVH node which are the slowest in infra).

[aeva black (tenbrae)]

As of May 2016, Kilo is no longer supported by openstack infrastructure (*), so I think we can close this bug.

(*) http://releases.openstack.org/