Neutron Metadata Agent broken on Keystone v3 in Kilo (and probably Liberty)

[Ian Cordasco]

We have a customer that launches roughly 700 (usually more) instances at once for burst workloads on OpenStack Ansible Kilo. When they do, Neutron Metadata Agent starts returning 500 errors.

We spent many hours tracking the source of the 500 errors and we have found *one part* of the root cause. Here's what happens:

In Kilo Neutron Metadata Agent (henceforth known as NMA), attempts to talk to Neutron API over RPC (in OSA's case Rabbit MQ). If that fails, it then attempts to use Neutronclient to talk to it over HTTP.

In /etc/neutron/metadata-agent.ini we have auth_url = https://<internalvip>:35357/v3 for Keystone, but if you follow the code path through Neutronclient, it is using internal code that *only* works on Keystone v2. This means that the HTTP fallback ends up receiving a 404 from Keystone which causes an unhandled exception in Neutron.

The mitigation for this is to fix the auth_url in metadata-agent.ini to use /v2.0 instead of /v3.

I looked at Neutron and Neutronclient on stable/liberty and they both have the *same* problem and roughly the same code path.

Naturally, this doesn't solve whatever is causing Rabbit connection problems from NMA *but* this does fix the 500 errors because NMA can successfully fallback to HTTP using neutronclient.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/327955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/327960

[Jesse Pretorius (jesse-pretorius)]

@Ian is this issue present in Mitaka onwards too?

[Ian Cordasco (icordasc)]

I haven't looked that far into the present yet Jesse. ;) That's on my list of things to do today.

[Ian Cordasco (icordasc)]

So a quick look at stable/mitaka's neutron/agent/metadata/agent.py file shows that it *only* uses RPC for the calls that start to fail in this case. In other words, there is no HTTP fallback which makes this configuration in mitaka + unnecessary potentially.

[Bjoern (bjoern-t)]

See https://bugs.launchpad.net/neutron/+bug/1590816

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ian Cordasco (<email address hidden>) on branch: kilo
Review: https://review.openstack.org/327955
Reason: I've merged this into the other metadata_agent.ini fixes for Keystone auth via Neutronclient v2_0

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ian Cordasco (<email address hidden>) on branch: liberty
Review: https://review.openstack.org/327960
Reason: I've merged this fix into a separate review for simplicity.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328430
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=b271778e16a83aefb0e704a9aca0ef09f405084c
Submitter: Jenkins
Branch: liberty

commit b271778e16a83aefb0e704a9aca0ef09f405084c
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 12:35:41 2016 -0500

    Use correct keystone auth parameters

    Neutron Metadata Agent uses the authentication parameters as a fallback
    method in the event that communication with Neutron API over RPC fails.
    In the fallback case, it looks for Keystone v2.0 authentication
    credentials but looks for them under the names:

    - admin_user
    - admin_password
    - admin_tenant_name

    Which can be the service user information. We were previously
    configuring Keystone v3 authentication parameters in this space and this
    causes 400 Bad Request responses from Keystone (since the Agent has null
    values for those config options).

    Further, the metadata agent does not need to use the admin URL to
    authenticate. Instead, it can use the internal URL to retrieve its token
    and then authenticate to Neutron API over HTTP.

    Change-Id: Ib413d3f3f3351bef29b0e68a2cfb96b7f3dff3c3
    Closes-bug: 1591282
    Closes-bug: 1590957

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328432
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=75deb166c8922f7ceb120d3892ae1ac4fa4702bb
Submitter: Jenkins
Branch: kilo

commit 75deb166c8922f7ceb120d3892ae1ac4fa4702bb
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 12:35:41 2016 -0500

    Use correct keystone auth parameters

    Neutron Metadata Agent uses the authentication parameters as a fallback
    method in the event that communication with Neutron API over RPC fails.
    In the fallback case, it looks for Keystone v2.0 authentication
    credentials but looks for them under the names:

    - admin_user
    - admin_password
    - admin_tenant_name

    Which can be the service user information. We were previously
    configuring Keystone v3 authentication parameters in this space and this
    causes 400 Bad Request responses from Keystone (since the Agent has null
    values for those config options).

    Further, the metadata agent does not need to use the admin URL to
    authenticate. Instead, it can use the internal URL to retrieve its token
    and then authenticate to Neutron API over HTTP.

    Change-Id: Ib413d3f3f3351bef29b0e68a2cfb96b7f3dff3c3
    Closes-bug: 1591282
    Closes-bug: 1590957
    (cherry picked from commit b271778e16a83aefb0e704a9aca0ef09f405084c)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.15 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.17 release.