Tempest test test_list_get_baremetal_nodes fails with 'No module named generic' Error

[Anna Babich]

The bug has been reproduced against the following env:
Snapshot:
ababich@srv129-bud:~$ grep SNAPSHOT /home/jenkins/env_inject.properties
SNAPSHOT_ID=#268
Environment:
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+--------+---------------------------+---------+-------------+-------------------+-------------------+---------------+--------+---------
 1 | ready | slave-05_compute_cinder | 1 | 10.109.20.7 | 64:e1:ac:6f:44:3f | cinder, compute | | 1 | 1
 6 | ready | slave-04_compute_cinder | 1 | 10.109.20.6 | 64:cf:f7:7c:6b:a8 | cinder, compute | | 1 | 1
 4 | ready | slave-03_controller_mongo | 1 | 10.109.20.5 | 64:6f:79:ea:ce:ad | controller, mongo | | 1 | 1
 3 | ready | slave-01_controller_mongo | 1 | 10.109.20.3 | 64:95:c1:35:ef:dd | controller, mongo | | 1 | 1
 2 | ready | slave-06_ironic | 1 | 10.109.20.8 | 64:e4:00:b1:f1:ee | ironic | | 1 | 1
 5 | ready | slave-02_controller_mongo | 1 | 10.109.20.4 | 64:54:55:2b:b0:83 | controller, mongo | | 1 | 1
[root@nailgun ~]#

Tempest test tempest.api.compute.admin.test_baremetal_nodes.BaremetalNodesAdminTestJSON.test_list_get_baremetal_nodes fails with the " Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible." Error

request ID for logs debugging - req-009a2ab6-a0be-4c43-be70-ae28888d70e4 cf90808a84fa4d85bb8d09b8b23cadcb 40a1711a67064b74bc66ddcc207db32d

nova-all.log trace related to it: http://paste.openstack.org/show/579986/
nova-api log related to it: http://paste.openstack.org/show/579987/

The diagnostic snapshot will be attached below.

[Anna Babich (ababich)]

Diagnostic snapshot - https://drive.google.com/a/mirantis.com/file/d/0B-mMDT56Q1EGUk81djR4Y1YtTWc/view?usp=sharing

[Roman Podoliaka (rpodolyaka)]

This was reproduced on CI in the job running Tempest tests *only* on one of three controllers. After nova-api is restarted, everything starts to work correctly. The error is really weird, but I assume module "generic" is

https://github.com/openstack/keystoneauth/blob/stable/mitaka/keystoneauth1/loading/_plugins/identity/generic.py

not sure why it can't be loaded, though. Maybe pkg_resources caches some info about entrypoints and for some reason the cache is broken after update of keystoneuath. We'll need to look closer at this one.