auth_url contains wrong configuration for metadata_agent.ini and other neutron config

[Bjoern]

The current configuration

auth_url = {{ keystone_service_adminuri }}

will lead to a incomplete URL like http://1.2.3.4:35357 and will cause the neutron-metadata-agent to make bad token requests like :

POST /tokens HTTP/1.1
Host: 1.2.3.4:35357
Content-Length: 91
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient

and the response is

HTTP/1.1 404 Not Found
Date: Tue, 01 Mar 2016 22:14:58 GMT
Server: Apache
Vary: X-Auth-Token
Content-Length: 93
Content-Type: application/json

and the agent will stop responding with

2016-02-26 13:34:46.478 33371 INFO eventlet.wsgi.server [-] (33371) accepted ''
2016-02-26 13:34:46.486 33371 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 109, in __call__
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 204, in _get_instance_and_tenant_id
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 197, in _get_ports
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in __call__
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_from_cache(target_self, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in _get_from_cache
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent item = self.func(target_self, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 166, in _get_ports_for_remote_address
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ip_address=remote_address)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 135, in _get_ports_from_server
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_using_client(filters)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 177, in _get_ports_using_client
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = client.list_ports(**filters)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ret = self.function(instance, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent **_params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent for r in self._pagination(collection, path, **params):
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent res = self.get(path, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200, in do_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent content_type=self.content_type())
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in do_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate_and_fetch_endpoint_url()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in authenticate
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self._authenticate_keystone()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in _authenticate_keystone
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent raise exceptions.Unauthorized(message=resp_body)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent

I seems we need to change the template to use actually keystone_service_adminurl like

auth_url = {{ keystone_service_adminurl }}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287411

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288642

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288655

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288693

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288730

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288749

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288780

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/288781

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288749
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_glance/commit/?id=c728f17d078ddfedfd49cf4097482617f28df6b5
Submitter: Jenkins
Branch: master

commit c728f17d078ddfedfd49cf4097482617f28df6b5
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 14:55:30 2016 -0600

    Fix auth_url for glance configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: If7ca6ea13db317e37fb6453366c348752b84e3d0
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288780
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/commit/?id=58e1520d93188ac2ed73e7afc569f1ed677e6388
Submitter: Jenkins
Branch: master

commit 58e1520d93188ac2ed73e7afc569f1ed677e6388
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 16:44:57 2016 -0600

    Fix auth_url for heat configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: I5d5b32dc9e91f9c127660d0b7fb96765f55b7b32
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288781
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_swift/commit/?id=ec6439a54e45756035512d2737efccf2924903ba
Submitter: Jenkins
Branch: master

commit ec6439a54e45756035512d2737efccf2924903ba
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 16:46:02 2016 -0600

    Fix auth_url for swift configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: I46f2ab7cbdb579dda5d019c29950af7e8c974bea
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288730
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=cfdfa385d624c98e84e6c0a7fb7fef0152c8c076
Submitter: Jenkins
Branch: master

commit cfdfa385d624c98e84e6c0a7fb7fef0152c8c076
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 14:54:00 2016 -0600

    Fix auth_url for cinder configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: Ib2f47cb62bd9098fa2e7335f7f776b525da71ac3
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287411
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=eda93fea238e07c083e5425a54b43dc7829bf9d8
Submitter: Jenkins
Branch: master

commit eda93fea238e07c083e5425a54b43dc7829bf9d8
Author: Bjoern Teipel <email address hidden>
Date: Wed Mar 2 13:54:19 2016 -0600

    Fix auth_url for neutron configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: Ic8b353b6d41b48cb9d7d825d26dc932d940994e8
    Closes-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288642
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=58e8d870bc1eea3211ed9cb7a2601bb27b6653c2
Submitter: Jenkins
Branch: master

commit 58e8d870bc1eea3211ed9cb7a2601bb27b6653c2
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 12:25:17 2016 -0600

    Fix auth_url for nova configuration

    This fix removes the invalid configuration for auth_url which was
    configured to use a URI (keystone_service_adminuri) rather than
    then a valid Keystone URL. Additionally the options is not
    necessary according

    http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html

    Change-Id: I5ad996a39263d0b66ab923c38d8b6cf656929210
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288655
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_aodh/commit/?id=935029bb633332f28f53f523b6bad2b6f88e6262
Submitter: Jenkins
Branch: master

commit 935029bb633332f28f53f523b6bad2b6f88e6262
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 13:13:20 2016 -0600

    Fix auth_url for aodh configuration

    This fix configures the auth_url parameter to use keystone_service_adminurl
    over the existing keystone_service_adminuri parameter which actually leads
    to a incomplete URL lacking the API version like /v3/tokens

    Change-Id: If40962423f72259b8d7a6203890d6923be02c273
    Related-Bug: #1552394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288693
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=9083159e96b37acf622aee3d9d75ee94b494be98
Submitter: Jenkins
Branch: master

commit 9083159e96b37acf622aee3d9d75ee94b494be98
Author: Bjoern Teipel <email address hidden>
Date: Fri Mar 4 13:27:14 2016 -0600

    Fix auth_url for ceilometer configuration

    This fix removes the invalid configuration for auth_url which was
    configured to use a URI (keystone_service_adminuri) rather than
    then a valid Keystone URL. Additionally the option inside
    keystone_authtoken is not necessary according

    http://docs.openstack.org/liberty/config-reference/content/section_ceilometer.conf.html

    Change-Id: Ibb1b565eda005bcfc99e611afff723ca24176c19
    Related-Bug: #1552394

[Bjoern (bjoern-t)]

Reassigning so I can back port the patches

[Bjoern (bjoern-t)]

Marking neutron as affected since correcting the auth_url did not seem to fix this reliable enough.
We observed still issues, especially growing metadata response with a 404 wrapped into a 401. The interesting part is that this error goes aways after the neutron-metadata-agent restart but ultimately comes back. We think it is triggered with increasing volume but could not locate when it happens, certainly not if the service token is expired.

[Miguel Angel Ajo (mangelajo)]

@boejern-teipel, The bug description doesn't seem to match anymore with what you're describing in #18, could you open a separate bug for neutron with the details?

Thank you.

[Bjoern (bjoern-t)]

Opened Neutron bug https://bugs.launchpad.net/neutron/+bug/1590816

[Bjoern (bjoern-t)]

Back ports to liberty not necessary anymore due to the v2 fallback issues we found at https://review.openstack.org/#/c/327960/

Setting liberty to invalid state.