[ironic] job master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy failed: TimeoutError: Failed to update hypervisor details

[Alexander Gordeev]

The CI job is continuing to failing with:

======================================================================
ERROR: Revert snapshot, update fuel-agent, bootstrap from review
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/gates_tests/tests/test_review_in_fuel_agent.py", line 126, in gate_patch_fuel_agent
    self._create_os_resources(ironic_conn)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/tests/test_ironic_base.py", line 139, in _create_os_resources
    ironic_conn.wait_for_ironic_hypervisors(ironic_conn, ironic_slaves)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/helpers/ironic_actions.py", line 94, in wait_for_ironic_hypervisors
    timeout_msg='Failed to update hypervisor details')
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 111, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Failed to update hypervisor details

----------------------------------------------------------------------

Eg.: https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/660/console

[Alexander Gordeev (a-gordeev)]

Set to Crit just because it blocks the incoming CR queue for fuel-agent from merges.

[Vasyl Saienko (vsaienko)]

nova-compute logs contain the following:

2016-08-31T15:20:54.427570+00:00 debug: 2016-08-31 15:20:54.425 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] REQ: curl -g -i -X GET http://10.109.25.5:6385/v1/nodes/detail -H "X-OpenStack-Ironic-API-Version: 1.21" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}8044ce8b8891579ba962dd2155492783c054e963" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:337
2016-08-31T15:20:54.513042+00:00 debug: 2016-08-31 15:20:54.505 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] RESP: [406] X-Openstack-Ironic-Api-Minimum-Version: 1.1 X-Openstack-Ironic-Api-Maximum-Version: 1.20 Content-Type: application/json Content-Length: 217 Date: Wed, 31 Aug 2016 15:20:54 GMT Connection: close
RESP BODY: {"error_message": "{\"code\": 406, \"description\": \"Version 1.21 was requested but the minor version is not supported by this service. The supported version range is: [1.1, 1.20].\", \"title\": \"Not Acceptable\"}"}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:366
2016-08-31T15:20:54.513042+00:00 err: 2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] Error during ClusteredComputeManager.update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task task(self, context)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6429, in update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task nodenames = set(self.driver.get_available_nodes())
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 577, in get_available_nodes
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task self._refresh_cache()
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 547, in _refresh_cache
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task for node in self._get_node_list(detail=True, limit=0):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 467, in _get_node_list
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task node_list = self.ironicclient.call("node.list", **kwargs)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/client_wrapper.py", line 139, in call
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task return self._multi_getattr(client, method)(*args, **kwargs)
2016-08-31 15:2...