ironicclient "Not Acceptable" error

[Steven Hardy]

I'm seeing this with recent versions of ironicclient, dtantsur says there's been a backwards incompatible change there (fail!):

Traceback (most recent call last):
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/os_cloud_config/cmd/register_nodes.py", line 74, in main
    blocking=True, keystone_client=keystone_client)
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/os_cloud_config/nodes.py", line 281, in register_all_nodes
    node_map = _populate_node_mapping(ironic_in_use, client)
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/os_cloud_config/nodes.py", line 142, in _populate_node_mapping
    nodes = [n.to_dict() for n in client.node.list()]
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/ironicclient/v1/node.py", line 88, in list
    return self._list(self._path(path), "nodes")
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/ironicclient/common/base.py", line 121, in _list
    resp, body = self.api.json_request('GET', url)
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/ironicclient/common/http.py", line 206, in json_request
    resp, body_iter = self._http_request(url, method, **kwargs)
  File "/home/shardy/tripleo/tripleo-incubator/openstack-tools/lib/python2.7/site-packages/ironicclient/common/http.py", line 189, in _http_request
    error_json.get('debuginfo'), method, url)
NotAcceptable: Not Acceptable (HTTP 406)

I got this running devtest.sh --trash-my-machine --no-mergepy --no-undercloud

http://paste.fedoraproject.org/207983/84140721 is the environment I'm using for devtest

[Steven Hardy (shardy)]

Ok, seems this is caused by the ironicclient requesting v1.6 and my seed ironic only supports up to v1.4

Looking into how that happened, but it looks like this is not a bug (other than the backwards incompatible change to the client) so setting incomplete while I confirm.

[Giulio Fidente (gfidente)]

I'm marking as critical, at least for tripleo project, as this is currently blocking puppet CI job

[Giulio Fidente (gfidente)]

Dan Prince submitted https://review.openstack.org/#/c/161277/ which should unblock CI by using "current" delorean; it remains to be investigated if the backward incompatible ironic change is a bug or not

[Steven Hardy (shardy)]

I can confirm that removing the pin as in https://review.openstack.org/#/c/161277/ does work around this problem - I was using the same delorean pin as CI, which is why I had an "old" ironic in my seed.

[Ruby Loo (rloo)]

This patch is a partial fix for this bug: https://review.openstack.org/#/c/171334/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/171334
Committed: https://git.openstack.org/cgit/openstack/python-ironicclient/commit/?id=8b453ff88a04da994bfd97e49d9fb3822c275eed
Submitter: Jenkins
Branch: master

commit 8b453ff88a04da994bfd97e49d9fb3822c275eed
Author: Devananda van der Veen <email address hidden>
Date: Tue Apr 7 11:43:54 2015 -0700

    Client should fall back to the lower versions if necessary

    The python client should catch 406 Not Acceptable errors when talking to
    older versions of the server. If the user did not request a specific
    version, the client should negotiate (fall back to) a mutually supported
    API version with the server.

    If there is no mutually supported version, or if the user specified a
    version in which the requested feature is unavailable, the client should
    raise an error.

    This patch is a partial implementation of the client side of
    auto-negotiation. It catches the 406 error and attempts to determine
    what version the server is based on the returned headers. It does NOT
    actually change what values are sent or returned by the client, aside
    from the version headers.

    Without this fix, the client fails to connect to any server version
    between commit 41595327 (when v1.1 was introduced) and commit 6ecee368
    (when v1.6 was introduced), because it is requesting API version 1.6
    and not handling the 406 Not Acceptable response.

    Note that API server versions prior to commit 32fb6e99 did not return
    any version information in the header of a 406 Not Acceptable response.
    To work around this, when the client receives a 406 response that does
    not contain any version header, the client performs an extra GET request
    to the root /v1/ URL to retrieve the supported versions. This allows the
    client to work with API services that might be running unreleased
    mid-kilo code as well as both stable/juno and the latest kilo code.

    Co-Authored-By: Dmitry Tantsur <email address hidden>
    Co-Authored-By: John L. Villalovos <email address hidden>
    Change-Id: Iab6a0814f0db6dd6e7bfc00da28baf8403a6b1db
    Blueprint: api-microversions
    Partial-bug: #1441170

[Steven Hardy (shardy)]

Fixed in ironicclient so removing TripleO