Gate failure: Timed out waiting for Nova hypervisor-stats count >= 1

[Dmitry Tantsur]

See for example http://logs.openstack.org/39/170439/2/gate/gate-tempest-dsvm-ironic-agent_ssh/66849d2/logs/devstacklog.txt.gz

Not sure how often it happens, just opening this bug to track the issue.

[Vladyslav Drok (vdrok)]

Looking at the logs, ir-api started at 2015-04-06 16:48:10.051, while last attempt to do node.list from n-cpu happened on 2015-04-06 16:47:53.497. Also just happened to me on fresh devstack locally.

[Vladyslav Drok (vdrok)]

There are some nova x509 requests which fail, and it takes a minute to fail each request. That seems to cause this late start of ironic.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/171313

[Adam Gandelman (gandelman-a)]

It looks like openstackclient calls are eating up the window of time between n-cpu starting and ir-api coming up. I've opened https://bugs.launchpad.net/python-openstackclient/+bug/1441294 to look at that separately, but there is no harm in adjusting Ironic's api max retries here to cope with this and allow devstack more time to do whatever it may need to do (171313)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/173493

[Thiago Paiva Brito (outbrito)]

n-cpu tries to communicate with Ironic by 60 seconds before giving up. As vdrok said, the nova x509 requests were failing and the timeout was causing n-cpu to exceed the maximum attempts to reach ir-api.

This was caused by a patch that removed n-crt from the default devstack setup. This patch corrects it.

P.S.: The requests of n-cpu were failing on devstack without Ironic setup, but it isn't noted because the libvirt driver doesn't need a post-installed service.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/173493
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=73af846ca064f214828c9833ab83561be53a1be4
Submitter: Jenkins
Branch: master

commit 73af846ca064f214828c9833ab83561be53a1be4
Author: Thiago Paiva <email address hidden>
Date: Tue Apr 14 16:57:22 2015 -0300

    Fixing n-crt removal from stackrc

    The commit 279cfe75198c723519f1fb361b2bff3c641c6cef removed the n-crt
    service from the default devstack setup. As such, the stack.sh script
    begun to thrown the following error when trying to "nova x509-create-cert":

      ERROR (ClientException): The server has either erred or is incapable of
      performing the requested operation. (HTTP 500)

    This patches reintroduces the n-crt as a default service.

    Change-Id: Id9695a37e1c6df567f2c86baa4475225adcfb0ee
    Closes-bug: #1441007

[Zhenzan Zhou (zhenzan-zhou)]

I opened a bug before for almost the same issue: https://launchpad.net/bugs/1430616. The proposed patch is under review https://review.openstack.org/#/c/173681/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adam Gandelman (<email address hidden>) on branch: master
Review: https://review.openstack.org/171313

[Matt Riedemann (mriedem)]

This is still happening:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZGV2c3RhY2svbGliL2lyb25pY1wiIEFORCBtZXNzYWdlOlwiVGltZWQgb3V0IHdhaXRpbmcgZm9yIE5vdmEgaHlwZXJ2aXNvci1zdGF0cyBjb3VudCA+PSAxXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzcxNzUwODM0MjZ9

[Dmitry Tantsur (divius)]

I see it again now: http://logs.openstack.org/61/246161/12/check/gate-tempest-dsvm-ironic-pxe_ipa-ipxe/169b905/logs/devstacklog.txt.gz