ovb fails with CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: Failed to provision instance ... Timeout reached while waiting for callback"

[Alex Schultz]

The deployment fails while trying to spawn instances for the nodes.
http://logs.openstack.org/85/431185/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/05312d4/console.html#_2017-02-14_18_39_33_651121

The ironic conductor has some errors in it...

2017-02-14 18:06:58.625 28229 ERROR ironic.drivers.modules.ipmitool [-] IPMI power on timed out after 4 retries on node 60831fcb-be9b-4d8e-9c92-558b96f74891.
2017-02-14 18:06:58.650 28229 ERROR ironic.drivers.modules.agent_base_vendor [req-f28bd4cd-f75f-4178-b7ed-c7bdfe28e974 - - - - -] Error rebooting node 60831fcb-be9b-4d8e-9c92-558b96f74891 after deploy. Error: Failed to set node power state to power on.

http://logs.openstack.org/85/431185/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/05312d4/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2017-02-14_18_06_58_625

2017-02-14 18:07:16.691 28229 ERROR ironic.drivers.modules.agent_client [req-f28bd4cd-f75f-4178-b7ed-c7bdfe28e974 - - - - -] Error invoking agent command log.collect_system_logs for node 60831fcb-be9b-4d8e-9c92-558b96f74891. Error: HTTPConnectionPool(host='192.168.24.18', port=9999): Max retries exceeded with url: /v1/commands?wait=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x54c8710>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))

http://logs.openstack.org/85/431185/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/05312d4/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2017-02-14_18_07_16_691

2017-02-14 18:07:17.979 28229 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Error rebooting node 60831fcb-be9b-4d8e-9c92-558b96f74891 after deploy. Error: Failed to set node power state to power on.
2017-02-14 18:07:17.979 28229 ERROR ironic.drivers.modules.agent_base_vendor

http://logs.openstack.org/85/431185/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/05312d4/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2017-02-14_18_07_17_979

[Emilien Macchi (emilienm)]

logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22Message%3A%20Failed%20to%20provision%20instance%5C%22

[Emilien Macchi (emilienm)]

Even better: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Error%3A%20Failed%20to%20set%20node%20power%20state%20to%20power%20on%5C%22

[Emilien Macchi (emilienm)]

Hitting the issue again:
http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/78acd34/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2017-05-31_08_15_41_348

It happens with latest code of Ironic/Neutron/Nova.

[Emilien Macchi (emilienm)]

Failures in Neutron:

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/78acd34/logs/undercloud/var/log/neutron/server.txt.gz#_2017-05-31_08_16_19_966
http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/78acd34/logs/undercloud/var/log/neutron/openvswitch-agent.txt.gz#_2017-05-31_06_24_10_496