Error contacting Ironic server for 'node.list' during check-tempest-dsvm-ironic-pxe_ssh

[Joe Gordon]

http://logs.openstack.org/33/165233/11/check/check-tempest-dsvm-ironic-pxe_ssh/842e655/logs/screen-n-cpu.txt.gz#_2015-04-03_18_41_12_757

2015-04-03 18:41:00.738 WARNING nova.virt.ironic.client_wrapper [req-fee6ff6b-e7e9-4df1-8e82-1a2cb2c6a696 None None] Error contacting Ironic server for 'node.list'. Attempt 54 of 60
2015-04-03 18:41:02.739 DEBUG ironicclient.common.http [req-fee6ff6b-e7e9-4df1-8e82-1a2cb2c6a696 None None] curl -i -X GET -H 'X-Auth-Token: ccd1b4fd1b1442baa5e15c6dd5e7ca9c' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-ironicclient' http://127.0.0.1:6385/v1/nodes/?associated=True log_curl_request /usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py:123

Seen during an unrelated patch.

[Joe Gordon (jogo)]

It looks like this stacktrace happens a lot:

message:"Error contacting Ironic server for 'node.list'" AND tags:"screen-n-cpu.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3IgY29udGFjdGluZyBJcm9uaWMgc2VydmVyIGZvciAnbm9kZS5saXN0J1wiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI4MDkxODgxMDg0LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

Even on successful jobs.

[Adam Gandelman (gandelman-a)]

This is expected on successful jobs. devstack starts n-cpu and the ironic driver starts to ping the ironic API server for nodes. devstack starts up ironic a bit later in the deployment, so there's a window where it's unreachable. The fact that its hitting 60 attempts and failing means something new is blocking and widening that window to the point where nova ends up giving up.