Only Ironic public endpoint is supported

[Guang Yee]

Currently, there are number of places in Ironic that does endpoint lookup from the Keystone service catalog. By default, keystoneauth set it to 'public' if not specified.
Description
===========
We are supposed to be able to select the endpoint type by specify either the 'interface' or 'valid_interfaces' option in the [keystone_authtoken] section in nova.conf. But that parameter is not being conveyed in ironicclient.

Consequently, this makes it impossible to using Ironic without having to expose the public endpoint in the service catalog. Furthermore, for security reasons, our controller nodes (subnet) have no route to the public network and therefore will not be able to access the public endpoint. This is a rather significant limitation in deploying Ironic. Also, we seem to have broken backward compatibility as well as Ironic use to work in Pike without having to configure a public endpoint.

Steps to reproduce
==================
1) enable Ironic in devstack
2) delete the Ironic public endpoint in Keystone
3) set 'valid_interfaces = internal' in the [ironic] section in nova.conf and restart nova-compute service
4) try to provision a server and it will fail with errors similar to these in nova-compute logs

2019-02-28 18:00:28.136 48891 ERROR nova.virt.ironic.driver [req-4bace607-0ab6-45b5-911b-1df5fbcc0e01 None None] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Must provide Keystone credentials or user-defined endpoint, error was: publicURL endpoint for baremetal service not found: AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint, error was: publicURL endpoint for baremetal service not found

Expected result
===============
Server created without error.

Actual result
=============
Server failed to create, with errors similar to these in nova-compute logs

2019-02-28 18:00:28.136 48891 ERROR nova.virt.ironic.driver [req-4bace607-0ab6-45b5-911b-1df5fbcc0e01 None None] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: Must provide Keystone credentials or user-defined endpoint, error was: publicURL endpoint for baremetal service not found: AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint, error was: publicURL endpoint for baremetal service not found

Environment
===========
This bug is reproducible in devstack with Ironic plugin enabled.

Related bugs:

Ironic: https://storyboard.openstack.org/#!/story/2005118
Nova: https://bugs.launchpad.net/nova/+bug/1707860

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/640879

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/642863

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640879
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e082bdc166cb8215576801e0c89ef1fe771681ed
Submitter: Zuul
Branch: master

commit e082bdc166cb8215576801e0c89ef1fe771681ed
Author: Guang Yee <email address hidden>
Date: Mon Mar 4 13:30:09 2019 -0800

    pass endpoint interface to Ironic client

    Via change [1], ironicclient began to use endpoint_filter in the
    version negotiation code path, whereas it was previously unused if a
    fully-qualified endpoint had already been determined. Suddenly it was
    important that the `interface` part of this endpoint_filter be correct.

    Prior to ironicclient change [2], there was no way to pass an
    appropriate `interface` value through ironicclient's initialization, so
    the ironicclient used from nova would always end up with the default
    value, `public`, in the endpoint_filter. This would break in clouds
    lacking a public ironic API endpoint (see the referenced bug).

    With this change, we pass the value of the (standard, per ksa)
    `valid_interfaces` ironic config option into the ironicclient
    initialization, where (if and only if the ironicclient fix [2] is also
    present) it eventually gets passed through to the ksa Adapter
    initialization (which is set up to accept values from exactly that conf
    option) to wind up in the endpoint_filter.

    The effect is that nova's ironicclient will actually be using the
    interface from nova.conf throughout. (Because `valid_interfaces` is also
    used in recommended configuration setups - i.e. those that use the
    service catalog to determine API endpoints - to construct the
    endpoint_override used to initialize the ironicclient, the value used
    during version negotiation should be in sync with that used for regular
    API calls.)

    [1] I42b66daea1f4397273a3f4eb1638abafb3bb28ce
    [2] I610836e5038774621690aca88b2aee25670f0262

    Change-Id: I5f78d21c39ed2fd58d2a0f3649116e39883d5a2c
    closes-bug: 1818295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/643098

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/643098
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4cea970c2d56ef8ff813473cd5ba0e38e3f04fd
Submitter: Zuul
Branch: stable/rocky

commit d4cea970c2d56ef8ff813473cd5ba0e38e3f04fd
Author: Guang Yee <email address hidden>
Date: Mon Mar 4 13:30:09 2019 -0800

    pass endpoint interface to Ironic client

    Via change [1], ironicclient began to use endpoint_filter in the
    version negotiation code path, whereas it was previously unused if a
    fully-qualified endpoint had already been determined. Suddenly it was
    important that the `interface` part of this endpoint_filter be correct.

    Prior to ironicclient change [2], there was no way to pass an
    appropriate `interface` value through ironicclient's initialization, so
    the ironicclient used from nova would always end up with the default
    value, `public`, in the endpoint_filter. This would break in clouds
    lacking a public ironic API endpoint (see the referenced bug).

    With this change, we pass the value of the (standard, per ksa)
    `valid_interfaces` ironic config option into the ironicclient
    initialization, where (if and only if the ironicclient fix [2] is also
    present) it eventually gets passed through to the ksa Adapter
    initialization (which is set up to accept values from exactly that conf
    option) to wind up in the endpoint_filter.

    The effect is that nova's ironicclient will actually be using the
    interface from nova.conf throughout. (Because `valid_interfaces` is also
    used in recommended configuration setups - i.e. those that use the
    service catalog to determine API endpoints - to construct the
    endpoint_override used to initialize the ironicclient, the value used
    during version negotiation should be in sync with that used for regular
    API calls.)

    [1] I42b66daea1f4397273a3f4eb1638abafb3bb28ce
    [2] I610836e5038774621690aca88b2aee25670f0262

    Change-Id: I5f78d21c39ed2fd58d2a0f3649116e39883d5a2c
    closes-bug: 1818295
    (cherry picked from commit e082bdc166cb8215576801e0c89ef1fe771681ed)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642863
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=229101e6af234a8f051544d07fbf01964ec39c34
Submitter: Zuul
Branch: master

commit 229101e6af234a8f051544d07fbf01964ec39c34
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 12 15:10:11 2019 -0400

    Require python-ironicclient>=2.7.0

    This is related to change I5f78d21c39ed2fd58d2a0f3649116e39883d5a2c
    which depends on python-ironicclient change
    I610836e5038774621690aca88b2aee25670f0262 which was released in
    2.7.0. The former did not have a direct dependency or version
    bump since it is being backported to stable branches where we
    don't bump minimum required versions of dependencies.

    Change-Id: I5edf350b29ef3bd1f103cdd4b0b997cb5ca32117
    Related-Bug: #1818295

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.0 release.