[Ironic] [Tempest] All Ironic-related Tempest tests failed

[Timur Nurlygayanov]

Environment:
MOS 9.1, Tempest CI environments.

Steps To Reproduce:
1) Deploy MOS 9.1 environment with Ironic and 3 controllers (we reproduced the issue on 4 different configurations)
2) Run full Tempest test suite

Expected Result:
All Tempest tests passed

Observed Result:
All Ironic-related tests failed with the following error:

Traceback (most recent call last):

  File "tempest/api/baremetal/admin/test_ports_negative.py", line 26, in setUp

    _, self.node = self.create_node(self.chassis['uuid'])

  File "tempest/api/baremetal/admin/base.py", line 41, in wrapper

    resp, body = f(cls, *args, **kwargs)

  File "tempest/api/baremetal/admin/base.py", line 128, in create_node

    driver=cls.driver)

  File "tempest/services/baremetal/base.py", line 34, in wrapper

    return f(*args, **kwargs)

  File "tempest/services/baremetal/v1/json/baremetal_client.py", line 156, in create_node

    return self._create_request('nodes', node)

  File "tempest/services/baremetal/base.py", line 147, in _create_request

    resp, body = self.post(uri, body=body)

  File "tempest/lib/common/rest_client.py", line 276, in post

    return self.request('POST', url, extra_headers, headers, body, chunked)

  File "tempest/lib/common/rest_client.py", line 665, in request

    resp, resp_body)

  File "tempest/lib/common/rest_client.py", line 768, in _error_checker

    raise exceptions.BadRequest(resp_body, resp=resp)

tempest.lib.exceptions.BadRequest: Bad request

Details: {u'error_message': u'{"debuginfo": null, "faultcode": "Client", "faultstring": "No valid host was found. Reason: No conductor service registered which supports driver fake."}'}

[Pavlo Shchelokovskyy (pshchelo)]

The nature of the failure makes me strongly suspect that ironic api failed to connect to RPC (rabbit) bus. Ironic api communicates with ironic-conductors to find which conductor supports the requested driver (the set of supported drivers can be configured per-conductor). It seems that due to rabbit connectivity loss no valid conductor was found (none at all most probably) and thus this error.

Where can we see the logs from ironic-api?

Also - you mention 3 controllers. Just to be sure - how many Ironic conductors were there (Fuel role "ironic") and where were they deployed, on controllers or separate nodes? We'd like to see the logs from them as well.

[Timur Nurlygayanov (tnurlygayanov)]

Ironic QA team will provide the logs and environment for you.

I've updated the description of the issue, the issue reproduced in 100% of Tempest tests suite execution.

[Timur Nurlygayanov (tnurlygayanov)]

We are going to try the workaround in QA scripts, we will have an update tomorrow.

[Timur Nurlygayanov (tnurlygayanov)]

The fixes were merged and verified on CI:

https://github.com/Mirantis/mos-ci-deployment-scripts/commit/774cb7bc0048149cfb4baf7c54d7ed6ed4a87e8c
https://github.com/Mirantis/mos-ci-deployment-scripts/commit/abfcd7d4262e75e35b475a4445ede2f86e9877b1
https://github.com/Mirantis/mos-ci-deployment-scripts/commit/610c445a043c510ba069fed4de39f526a1b2cfa9

The root of the issue was in sync of the time of cluster nodes.
We have added additional sync and now it works fine.