nova-api-metadata does not use keystone internal endpoint

[Nobuto Murata]

xenial + pike

I believe this is a duplicate of:
https://bugs.launchpad.net/charms/+source/neutron-gateway/+bug/1657131
But let me open a new bug not to hijack the original one.

nova-api-metadata tries to access a public keystone endpoint and fails. Thus instance metadata won't work.

In my case, the host of neutron-gateway does not have any interface binding to OpenStack public endpoints network or a route to the network. use-internal-endpoints is essential to achieve a secure and confined network design.

172.18.246.39 - keystone public endpoint
172.18.248.39 - keystone internal endpoint
172.18.247.39 - keystone admin endpoint

[/var/log/nova/nova-api-metadata.log]

2018-03-15 05:39:33.708 505314 ERROR nova.api.metadata.handler ConnectFailure: Unable to establish connection to http://172.18.246.39:5000/v3/auth/tokens: HTTPConnectionPool(host='172.18.246.39', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f673286d290>: Failed to establish a new connection: [Errno 101] ENETUNREACH',))
2018-03-15 05:39:33.708 505314 ERROR nova.api.metadata.handler
2018-03-15 05:39:33.710 505314 INFO nova.metadata.wsgi.server [req-326e357e-2ffc-4cbe-9c3a-adf9a4dadb93 - - - - -] 192.168.0.13,172.18.248.69 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 229 time: 0.1714139

[Nobuto Murata (nobuto)]

"Canonical Field High" has been subscribed since this bug affects a delivery.

[Nobuto Murata (nobuto)]

nova-api-metadata process under strace.

[Nobuto Murata (nobuto)]

connect(9, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("172.18.247.39")}, 16) = -1 EINPROGRESS (Operation now in progress)

^^ here nova-api-metadata connects to admin endpoint (.247.)

getsockopt(9, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
connect(9, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("172.18.247.39")}, 16) = 0
sendto(9, "GET / HTTP/1.1\r\nHost: 172.18.247.39:5000\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nUser-Agent: nova-api-metadata keystoneauth1/3.1.0 python-requests/2.18.1 CPython/2.7.12\r\n\r\n", 215, 0, NULL, 0) = 215
recvfrom(9, 0x3619304, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(5, 0x7f4c2f9a0264, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(9, "HTTP/1.1 300 Multiple Choices\r\nDate: Thu, 15 Mar 2018 14:56:22 GMT\r\nServer: Apache/2.4.18 (Ubuntu)\r\nVary: X-Auth-Token\r\nX-Distribution: Ubuntu\r\nContent-Length: 600\r\nKeep-Alive: timeout=5, max=100\r\nConnection: Keep-Alive\r\nContent-Type: application/json\r\n\r\n{\"versions\": {\"values\": [{\"status\": \"stable\", \"updated\": \"2017-02-22T00:00:00Z\", \"media-types\": [{\"base\": \"application/json\", \"type\": \"application/vnd.openstack.identity-v3+json\"}], \"id\": \"v3.8\", \"links\": [{\"href\": \"http://172.18.246.39:5000/v3/\", \"rel\": \"self\"}]}, {\"status\": \"deprecated\", \"updated\": \"2016-08-04T00:00:00Z\", \"media-types\": [{\"base\": \"application/json\", \"type\": \"application/vnd.openstack.identity-v2.0+json\"}], \"id\": \"v2.0\", \"links\": [{\"href\": \"http://172.18.246.39:5000/v2.0/\", \"rel\": \"self\"}, {\"href\": \"https://docs.openstack.org/\", \"type\": \"text/html\", \"rel\": \"describedby\"}]}]}}", 8192, 0, NULL, NULL) = 855

^^^ multiple choices with v2 and v3, but all with public endpoint (.246.)

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 10
setsockopt(10, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(10, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(10, SOL_TCP, TCP_KEEPCNT, [4], 4) = 0
setsockopt(10, SOL_TCP, TCP_KEEPINTVL, [15], 4) = 0
connect(10, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("172.18.246.39")}, 16) = -1 ENETUNREACH (Network is unreachable)

^^^ then tries to connect to the public one (.246.) and unreachable.

sendto(8, "HTTP/1.1 500 Internal Server Error\r\nContent-Length: 207\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Thu, 15 Mar 2018 14:56:22 GMT\r\n\r\n<html>\n <head>\n <title>500 Internal Server Error</title>\n </head>\n <body>\n <h1>500 Internal Server Error</h1>\n An unknown error has occurred. Please try your request again.<br /><br />\n\n\n\n </body>\n</html>", 343, 0, NULL, 0) = 343

[Nobuto Murata (nobuto)]

nova.conf https://pastebin.canonical.com/p/q54WTtdqdD/
debug nova-api-metadata.log https://pastebin.canonical.com/p/GsfgT2jrqf/

Those pastebin are Canonical only (sorry, i didn't have time to mask info).

[James Page (james-page)]

OK so this is a little more complex that I first thought; the keystone credentials and URL are provided via the nova-cloud-controller charm, which uses a legacy hack to retrieve information from the api-paste.ini file (which is no longer used by nova itself).

That data always points at the public endpoint afaict; the neutron information will always use the internal URL for the neutron api (so that's ok).

[Nobuto Murata (nobuto)]

Change the following block in nova.conf in neutron-gateway makes the nova-api-metadata use admin port and admin endpoint.

[neutron]
auth_url=http://172.18.247.39:5000

[neutron]
auth_url=http://172.18.247.39:35357

Then metadata works. If I understand it correctly, non-admin port(5000) does not allow to return admin/internal endpoints?

[James Page (james-page)]

Nobuto that's quite possible - I think that the right fix is to switch the neutron-gateway charm to use the auth_protocol, auth_port data set rather than the service_protocol and port that its using atm.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/554240

[Corey Bryant (corey.bryant)]

Backport to stable/18.02?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554240
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=802f607b8c0f9dc20703b0eac9706431fc2f08bc
Submitter: Zuul
Branch: master

commit 802f607b8c0f9dc20703b0eac9706431fc2f08bc
Author: James Page <email address hidden>
Date: Mon Mar 19 14:24:05 2018 +0000

    Switch keystone authentication calls to admin ep

    Ensure that the keystone admin endpoint is used for calls
    to keystone, resolving issues when the public ep is not
    network accessible from the neutron-gateway units.

    Change-Id: I79a1183e7eddd4981367baf4a22fe2ec6374b0b9
    Closes-Bug: 1756111

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/554732

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554732
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=82bb7166ef25562d8fcd93f946151eb5c487747e
Submitter: Zuul
Branch: stable/18.02

commit 82bb7166ef25562d8fcd93f946151eb5c487747e
Author: James Page <email address hidden>
Date: Mon Mar 19 14:24:05 2018 +0000

    Switch keystone authentication calls to admin ep

    Ensure that the keystone admin endpoint is used for calls
    to keystone, resolving issues when the public ep is not
    network accessible from the neutron-gateway units.

    Change-Id: I79a1183e7eddd4981367baf4a22fe2ec6374b0b9
    Closes-Bug: 1756111
    (cherry picked from commit 802f607b8c0f9dc20703b0eac9706431fc2f08bc)