The Tempest test fails and there is an appropriate error in nova-api.log:

2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/baremetal_nodes.py", line 105, in index
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions     icli = _get_ironic_client()
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/baremetal_nodes.py", line 76, in _get_ironic_client
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions     icli = ironic_client.get_client(CONF.ironic.api_version, **kwargs)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/ironicclient/client.py", line 137, in get_client
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions     raise exc.AmbiguousAuthSystem(exception_msg)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions 
2016-05-12 12:10:20.333 31692 INFO nova.api.openstack.wsgi [req-14504d54-3f40-41ac-9dbb-34e1dab3473f 2a91e632bd98451d92038cf000ba4a52 0fc3f4af3c9c4590ba06aa3467c80be2 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'>

which indicates that ironic credentials in nova.conf were not specified correctly.

At the same time nova.conf says:

[ironic]
admin_tenant_name=services
admin_username=ironic
api_endpoint=http://10.109.1.3:6385/v1
admin_password=wStYWdzAtHsXYZxmV4RJJiRt
admin_url=http://10.109.1.3:35357/v2.0


but in Guru Meditation Report we can see that these settings were ignored:

root@node-2:~# ps -ef | grep nova-api
root     18511 15637  0 12:15 pts/33   00:00:00 grep --color=auto nova-api
nova     30240     1  1 05:03 ?        00:04:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova     31692 30240  0 05:03 ?        00:02:12 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova     31693 30240  0 05:03 ?        00:02:12 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova     31720 30240  0 05:03 ?        00:00:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova     31721 30240  0 05:03 ?        00:00:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
root@node-2:~# kill -SIGUSR2 30240


root@node-2:~# grep -i 'ironic:' /var/log/upstart/nova-api.log -A 10
ironic: 
  admin_auth_token = ***
  admin_password = ***
  admin_tenant_name = None
  admin_url = None
  admin_username = None
  api_endpoint = None
  api_max_retries = 60
  api_retry_interval = 2
  api_version = 1
  client_log_level = None
  
  
Checking of nova.conf last modification date and process start time allows to see why: 


root@node-2:~# ls -al /etc/nova/nova.conf 
-rw-r--r-- 1 root root 40963 May 11 22:39 /etc/nova/nova.conf

root@node-2:~# pgrep nova-api
30240
31692
31693
31720
31721

root@node-2:~# ls -ld /proc/30240
dr-xr-xr-x 9 nova nova 0 May 11 22:32 /proc/30240
root@node-2:~# ls -ld /proc/31692
dr-xr-xr-x 9 nova nova 0 May 11 22:33 /proc/31692
root@node-2:~# ls -ld /proc/31693
dr-xr-xr-x 9 nova nova 0 May 11 22:33 /proc/31693
root@node-2:~# ls -ld /proc/31720
dr-xr-xr-x 9 nova nova 0 May 11 22:33 /proc/31720
root@node-2:~# ls -ld /proc/31721
dr-xr-xr-x 9 nova nova 0 May 11 22:33 /proc/31721


nova-api had been started before /etc/nova/nova.conf configuration was completed, i.e. the latest modifications were ignored.


We should check our Ironic Puppet manifests to understand why this happens: either nova-api restart is not triggered after configuring nova.conf or it's some kind of a race condition.