nova baremetal-node-list unexpected API error 500

[mark lewis]

$ nova baremetal-node-list

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'> (HTTP 500) (Request-ID: req-78138575-cdcc-408e-a232-dc717bfe96d9)

Nova API log.
2016-02-19 15:38:04.507 DEBUG nova.osapi_compute.wsgi.server [req-48a19e8f-00dd-4fdf-a079-5cd17a030f7e None None] (31654) accepted ('10.0.0.150', 37341) from (pid=31654) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2016-02-19 15:38:04.508 DEBUG keystoneauth.session [-] REQ: curl -g -i --cacert "/opt/stack/data/ca-bundle.pem" -X GET http://10.0.0.150:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}2548654be6a35b05a064f0bd15e4da50119ebeda" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}05b09de98dd6820f5766b21e415cbc27190125e6" from (pid=31654) _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:248
2016-02-19 15:38:04.524 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 4227 X-Subject-Token: {SHA1}2548654be6a35b05a064f0bd15e4da50119ebeda Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5 Connection: Keep-Alive Date: Fri, 19 Feb 2016 15:38:04 GMT Content-Type: application/json x-openstack-request-id: req-4dd14b09-bc0e-4dfb-ae6c-76d77cbc0bd4
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "16109acaf0c640e4b96abeb2eca388e1", "name": "admin"}], "expires_at": "2016-02-19T16:38:04.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "0b130fa37b9a44a696b64559e713e032", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "4e6697fd84604775b097bb53bd290367", "name": "admin"}, "audit_ids": ["kEE5cM-hRHyH50QppYnuOw"], "issued_at": "2016-02-19T15:38:04.499020"}}
 from (pid=31654) _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:277
2016-02-19 15:38:04.525 INFO nova.osapi_compute.wsgi.server [req-82377dd1-15fb-439c-959b-c1afec33091b admin admin] 10.0.0.150 "GET /v2.1/0b130fa37b9a44a696b64559e713e032 HTTP/1.1" status: 404 len: 264 time: 0.0177109
2016-02-19 15:38:04.529 DEBUG nova.api.openstack.wsgi [req-24e38a23-d0ba-46aa-84ba-f6dfc27961e9 admin admin] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7df5a10>>' from (pid=31654) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:699
2016-02-19 15:38:04.530 INFO nova.osapi_compute.wsgi.server [req-24e38a23-d0ba-46aa-84ba-f6dfc27961e9 admin admin] 10.0.0.150 "GET /v2.1/ HTTP/1.1" status: 200 len: 652 time: 0.0027330
2016-02-19 15:38:04.600 DEBUG nova.api.openstack.wsgi [req-78138575-cdcc-408e-a232-dc717bfe96d9 admin admin] Calling method '<bound method BareMetalNodeController.index of <nova.api.openstack.compute.baremetal_nodes.BareMetalNodeController object at 0x76b68d0>>' from (pid=31654) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:699
2016-02-19 15:38:04.601 ERROR nova.api.openstack.extensions [req-78138575-cdcc-408e-a232-dc717bfe96d9 admin admin] Unexpected exception in API method
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/baremetal_nodes.py", line 105, in index
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions icli = _get_ironic_client()
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/baremetal_nodes.py", line 76, in _get_ironic_client
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions icli = ironic_client.get_client(CONF.ironic.api_version, **kwargs)
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/ironicclient/client.py", line 106, in get_client
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions raise exc.AmbiguousAuthSystem(e)
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions
2016-02-19 15:38:04.601 INFO nova.api.openstack.wsgi [req-78138575-cdcc-408e-a232-dc717bfe96d9 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'>
2016-02-19 15:38:04.601 DEBUG nova.api.openstack.wsgi [req-78138575-cdcc-408e-a232-dc717bfe96d9 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'> from (pid=31654) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1070
2016-02-19 15:38:04.602 INFO nova.osapi_compute.wsgi.server [req-78138575-cdcc-408e-a232-dc717bfe96d9 admin admin] 10.0.0.150 "GET /v2.1/0b130fa37b9a44a696b64559e713e032/os-baremetal-nodes HTTP/1.1" status: 500 len: 513 time: 0.0036349

* bug guideline stuff
> git log -1
commit 4bcf6529d5096c16fd1b9c418d5c40d2bb8a2831
Merge: 772e914 edc11c2
Author: Jenkins <email address hidden>
Date: Thu Feb 18 06:50:18 2016 +0000

    Merge "Keystone support deploy in uwsgi"

> CentOS Linux release 7.2.1511 (Core) fresh install minimal
> yum -y install wget telnet git
> git clone https://git.openstack.org/openstack-dev/devstack
> and have pretty much done nothing else but read the manuals.

Trying to launch instance in horizon and got:
Please try again later [Error: No valid host was found. There are not enough hosts available.].

Googled error and got https://ask.openstack.org/en/question/6445/no-valid-host-was-found-when-launching-an-instance/
read that, and so thought to try the bare-metal-node list and it said to report it.

Expected result: VM instance starts on default cirros image

Sorry too inexperienced with devstakc to produce script.

Good luck

[Sean Dague (sdague)]

It does look like this no longer works. The recommended approach is to access ironic directly.

[Matt Riedemann (mriedem)]

The error is when nova is trying to talk to ironic via the ironic client, there is an auth error:

TRACE nova.api.openstack.extensions raise exc.AmbiguousAuthSystem(e)
2016-02-19 15:38:04.601 TRACE nova.api.openstack.extensions AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token

So you need to check your configuration in nova.conf to make sure it has the proper credentials to be able to talk to ironic.

[Matt Riedemann (mriedem)]

Make sure you have the [ironic] section of nova.conf configured properly:

[ironic]

#
# From nova.conf
#

# Version of Ironic API service endpoint. DEPRECATED: Setting the API version
# is not possible anymore. (integer value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#api_version = 1

# URL for Ironic API endpoint. (string value)
#api_endpoint = <None>

# Ironic keystone admin name (string value)
#admin_username = <None>

# Ironic keystone admin password. (string value)
#admin_password = <None>

# Ironic keystone auth token.DEPRECATED: use admin_username, admin_password,
# and admin_tenant_name instead (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#admin_auth_token = <None>

# Keystone public API endpoint. (string value)
#admin_url = <None>

# PEM encoded Certificate Authority to use when verifying HTTPs connections.
# (string value)
#cafile = <None>

# Log level override for ironicclient. Set this in order to override the global
# "default_log_levels", "verbose", and "debug" settings. DEPRECATED: use
# standard logging configuration. (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#client_log_level = <None>

# Ironic keystone tenant name. (string value)
#admin_tenant_name = <None>

# How many retries when a request does conflict. If <= 0, only try once, no
# retries. (integer value)
#api_max_retries = 60

# How often to retry in seconds when a request does conflict (integer value)
#api_retry_interval = 2

[Matt Riedemann (mriedem)]

If you're using devstack and have just setup a basic environment, ironic is not enabled by default so that's probably your issue.

[kamalhussain (hussain-kamal)]

Thanks Matt for the input. I did verify that this is indeed a problem with the devstack configuration. I modified the local.conf based on the instructions from this link - http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html

After that, the command showed the right output:

ubuntu@devstack-kamal:~$ nova baremetal-node-list
+--------------------------------------+----------------+------------+------+-----------+---------+-------------+------------+-------------+-------------+---------------+
| ID | Host | Task State | CPUs | Memory_MB | Disk_GB | MAC Address | PM Address | PM Username | PM Password | Terminal Port |
+--------------------------------------+----------------+------------+------+-----------+---------+-------------+------------+-------------+-------------+---------------+
| fcb9de55-71d1-4a93-91fd-7771bcf5b676 | IRONIC MANAGED | available | 1 | 1024 | 10 | | | | | |
| 09e70ef4-2e04-427d-af0c-1bcb4b792d0d | IRONIC MANAGED | available | 1 | 1024 | 10 | | | | | |
| 465cadc8-99ce-42c4-8e97-a3693d34d9b4 | IRONIC MANAGED | available | 1 | 1024 | 10 | | | | | |
+--------------------------------------+----------------+------------+------+-----------+---------+-------------+------------+-------------+-------------+---------------+

This issue can be closed.