[RFE] Support metadata service with IPv6-only tenant network

[Baodong (Robert) Li]

EC2 metatdata service is supported by nova metadata service that is running in the management network. Cloud-init running in the instance normally accesses the service at 169.254.169.254. Cloud-init can be configured with metadata_urls other than the default http://169.254.169.254 to access the service. But such configuration is not currently supported by openstack. In order for the instance to access the nova metadata service, neutron provides proxy service that terminates http://169.254.169.254 and forwards the request to the nova metadata service, and responds back to the instance. Apparently, this works only when IPv4 is available in the tenant network. For an IPv6-only tenant work, to continue the support of this service, the instance has to access it at an IPv6 address. This requires enhancement in Neutron to support it.

A few options have been discussed so far:
   -- define a well-known ipv6 link-local address to access the metadata service.
   -- enhance IPv6 RA to advertise the metadata service endpoint to instances. This would require standards work and enhance cloud-init to support it.
   -- define a well-known name for the metadata service and configure metadata_urls to use the name. The name will be resolved to a datacenter specific IP address. The corresponding DNS record should be pre-provisioned in the datacenter DNS server for the instance to resolve the name.

[Fred Baker (b-fred)]

I can't really recommend the use of a link-local address in a network without NATs; the use of 169.254.169.254 (an IPv4 link-local address) depends on a NAT or a proxy. I can recommend a DNS SRV record (a well-known name), an RA option that advertises the relevant metadata server address, or an anycast address (http://www.iana.org/assignments/ipv6-anycast-addresses/ipv6-anycast-addresses.xhtml).

[Fred Baker (b-fred)]

However, I'll support the need to deploy a metadata service. Anyone that is using metadata for IPv4 is likely to want to implement IPv6 the same way, and we already have people turning IPv4 off.

[Ian Wells (ijw-ubuntu)]

In the case of the DNS record, we can implement this in Openstack by ensuring that the record is in the local Openstack-administrated DNS server. The DNS server is already provided to the VMs by SLAAC. If the user overrides or turns off DNS they would be on their own recognisance (which is what happens today with DHCPv4).

In fact, if we use a SRV we require two records, the SRV and the AAAA for the hostname referred to by the SRV. Given that cloud-init takes a list of URLs and that I can't find anything that uses SRV in the resolution of URLs (*), we might instead just want to go with a well known hostname that will have a specific address assigned to it when used from within an Openstack cloud.

Unstated above is the aim that we don't just choose a well-known address for a v6 metadata server. Firstly, it's hard to choose one (the v4 metadata server is using an inappropriate address per v4 standards to get around the problem of accidentally stealing someone else's address away) and secondly there's no really appropriate address category we fall into. We could get a v6 anycast address assigned by IANA, but the service is not technically anycast.

(*) It seems that Chrome won't fix this and the Firefox bug for it is 15 years old. I admit I've not checked wget and curl, mind, but that's still quite a precedent.

[Kyle Mestery (mestery)]

I think we agree this feature is needed. Lets bike shed on the solution once Robert posts some devref documentation for it.

[Kyle Mestery (mestery)]

Moving to Triaged, we discussed this in the neutron-drivers meeting and work can now proceed on this.

[Sean M. Collins (scollins)]

Before we get too far into this - let's remember that the metadata API is an Amazon EC2 compatibility feature.

We cannot make changes to this API - we do not own this API contract. Amazon is the owner of this API and their documentation and cloud is where this API is controlled. If we start making changes to this API, that are incompatible, or will be incompatible if/when Amazon decides to implement IPv6 on their cloud, we will have broken this contract with our users that OpenStack has a compatibility layer for apps that rely on AWS APIs - if we end up implementing it in a way that Amazon later chooses to implement differently.

In the case where OpenStack has innovated in ways that has beaten Amazon to the market (like, for instance, IPv6 functionality) we should not try and "fix" someone's API contract for them. Instead, we should provide compelling alternatives (such as Config-Drive) as the way to use metadata in IPv6 clouds.

/rant

[Ian Wells (ijw-ubuntu)]

Firstly, we have made changes to this API and we pass data in both the Amazon and now also Openstack formats (at the same time) - the cloud-init script in the VM can try to use either and gets the data in the form it expects.

Secondly, adding a new endpoint address for IPv6 does not stop us from keeping compatibility with EC2 in the future by adding whatever address they come up with to the endpoints in the future. But they don't have one so there's nothing here we can do right now other than invent for ourselves or do nothing at all, and I don't think either is wise.

We are fine with API compatibility as long as what we offer is a superset of EC2. Listening on another (IPv6) address is most definitely a superset.

[Sean M. Collins (scollins)]

Big mailing list thread:

http://lists.openstack.org/pipermail/openstack-dev/2015-September/073571.html

[Sean M. Collins (scollins)]

Frankly I'd vote this as WONTFIX due to issues discussed on the mailing list

[Armando Migliaccio (armando-migliaccio)]

I am not biased one way or the other and I can see both sides of the arguments. Based on the lack of consensus and in the absence of someone volunteering to resolve/define the course of action, I can't see how this can be targeted/approved.

[Ian Wells (ijw-ubuntu)]

james-page: could you explain why?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/315604

[Henry Gessau (gessau)]

More discussion is happening in the spec, https://review.openstack.org/315604

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/315604
Reason: Comments on bug report seems to hint this effort has been abandoned. Please resume, if you intend to proceed.

[YAMAMOTO Takashi (yamamoto)]

has the situation changed since the last time this has been discussed on the ML a few years ago?
in AWS, or cloud-init.

[Brian Haley (brian-haley)]

And I have another question. Say neutron and cloud-init can be upgraded to support an IPv6-only metadata request. Are there additional changes required to the API? For example, there is a local-ipv4 value today, is a local-ipv6 needed? I'm confused by the wording of the 'network/interfaces/macs/mac/ipv6s' field - "The IPv6 addresses associated with the interface. Returned only for instances launched into a VPC." - local-ipv4 doesn't mention VPC.

[YAMAMOTO Takashi (yamamoto)]

@Brian

just because non VPC (EC2-Classic?) doesn't support ipv6?
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html

[Romain Meillon (rmeillon)]

With all my respect, don't you think it is time to go on this RFE ?
I cannot imagine how many cases are openned in openstack vendors because of this.
Config drive is not supported in major OS distributions.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/315604
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=20088d8d65d53ba1f13b554326478531ce14e941
Submitter: Zuul
Branch: master

commit 20088d8d65d53ba1f13b554326478531ce14e941
Author: Sergey Nechaev <email address hidden>
Date: Fri Oct 14 15:39:33 2016 +0300

    Spec for IPv6 support in metadata service

    Co-Authored-By: Slawek Kaplonski <email address hidden>

    Change-Id: I724050c3ca5815a7bde367987f6e86bb9b4e52b5
    Related-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/718729

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738205

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738205
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=60eab578fc74bf1350539f7d4729fb05abf5e0c8
Submitter: Zuul
Branch: master

commit 60eab578fc74bf1350539f7d4729fb05abf5e0c8
Author: Bence Romsics <email address hidden>
Date: Fri Jun 26 15:30:06 2020 +0200

    Metadata addresses

    Define well-known metadata addresses in neutron-lib so they are easily
    shared between various implementations like dhcp-agent and ovn.

    Change-Id: Ic382feb7fca036b3c7aae5f21399b7639520a92b
    Partial-Bug: #1460177
    Related-Change: https://review.opendev.org/738247

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/745705

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715482
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a0b18d553d75977fa48c6a64904b19c8fcd1b6de
Submitter: Zuul
Branch: master

commit a0b18d553d75977fa48c6a64904b19c8fcd1b6de
Author: Bence Romsics <email address hidden>
Date: Fri Mar 27 16:37:34 2020 +0100

    metadata-ipv6: DHCP namespace

    Send IPv6 metadata traffic (dst=fe80::a9fe:a9fe) to the metadata-agent.

    When running on IPv6 enabled system bind haproxy (i.e. the
    metadata-proxy) to 169.254.169.254 and to fe80::a9fe:a9fe also.

    We do not introduce new config options. The usual config options
    (enable_isolated_metadata, force_metadata, enable_metadata_proxy)
    now control the metadata service over both IPv4 and IPv6.

    This change series only affects the guests' access to the metadata
    service (over tenant networks). They change nothing about how the
    metadata-agent talks to Nova's metadata service.

    Metadata access over IPv6 is supposed to work both on dual-stack and
    v6-only networks.

    In order to enable the metadata service on pre-existing isolated
    networks during an upgrade, this change makes each dhcp-agent restart
    trigger a quick restart of dhcp-agent-controlled metadata-proxies,
    so they can pick up their new config making them also bind to
    fe80::a9fe:a9fe.

    Change-Id: If35f00d1fc9e4ab7e232660362410ce7320c45ba
    Partial-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715483
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a1f4ee3ade0e694b6b9c1de5389924467b7fd943
Submitter: Zuul
Branch: master

commit a1f4ee3ade0e694b6b9c1de5389924467b7fd943
Author: Bence Romsics <email address hidden>
Date: Fri Mar 27 16:37:26 2020 +0100

    metadata-ipv6: Router namespace

    We push a v6 host route to make the guest send its metadata requests
    in the direction of our router. We redirect it to haproxy which
    mangles the headers and sends the request along to metadata-agent.

    Apparently the supported list of dhcp options for dhcpv6 is quite
    short in dnsmasq (cf. dnsmasq --help dhcp6) - not including anything
    like classless-static-route for dhcpv4. So we must rely solely on
    radvd to push host routes to the guest.

    Metadata access over IPv6 is supposed to work both on dual-stack and
    v6-only networks.

    The following v6 subnet modes are supposed to work:

    --ipv6-ra-mode slaac --ipv6-address-mode slaac
    --ipv6-ra-mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless
    --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful

    Change-Id: I28f2914b1b67659af2db7240eae730ac43daccd2
    Partial-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718729
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a818c41c25c33a3491254c74f33b7d273fb16f1b
Submitter: Zuul
Branch: master

commit a818c41c25c33a3491254c74f33b7d273fb16f1b
Author: Bence Romsics <email address hidden>
Date: Thu Apr 9 16:49:00 2020 +0200

    metadata-ipv6: Accept link local address in X-Forwarded-For

    In the spec we said:
    """
    When the metadata proxy processes a request, it gathers the L2 addresses
    of a VM, and the source interface, and passes it to the metadata service.

    The Metadata service, instead of using the VM IP, uses the "VM MAC" and
    "Gateway MAC" to identify the instance.
    """

    But since we switched from the home-grown metadata-ns-proxy to haproxy
    we no longer control some of the headers included, like X-Forwarded-For.
    haproxy allows us to turn X-Forwarded-For on or off, but it cannot
    give us an X-Forwarded-For-MAC header.

    Instead it seems we have to rely on the source address being the IPv6
    link local address generated from the NIC's MAC address as specified
    in RFC 4291:
    https://tools.ietf.org/html/rfc4291#section-2.5.6
    https://tools.ietf.org/html/rfc4291#appendix-A

    Note that means you cannot use IPv6 Privacy Extensions:
    https://tools.ietf.org/html/rfc4941

    Change-Id: Ife592fcfc69e26f61ec1f45c06821cb025cc7cf2
    Closes-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745705
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3f30727c0a8ef792b0c9e6a4739c9e0af61d9bd8
Submitter: Zuul
Branch: master

commit 3f30727c0a8ef792b0c9e6a4739c9e0af61d9bd8
Author: Bence Romsics <email address hidden>
Date: Tue Aug 11 17:52:19 2020 +0200

    Use metadata constants from n-lib

    Change-Id: I8045d1670045a68c955b0bf95077137ea07fd72e
    Partial-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/750325

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/750326

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/750355

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bence Romsics (<email address hidden>) on branch: master
Review: https://review.opendev.org/750326
Reason: Bump to 18.04 looks smoother.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750325
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=67621f75bcaa2515bf2de5370805c7a2c9562fb0
Submitter: Zuul
Branch: master

commit 67621f75bcaa2515bf2de5370805c7a2c9562fb0
Author: Bence Romsics <email address hidden>
Date: Tue Sep 8 13:43:45 2020 +0200

    Bump advanced image to ubuntu 18.04

    Ubuntu 16.04 seems insufficient for tempest testing the metadata
    over ipv6 feature. IIRC it ignores routes offered in router
    advertisements.

    Change-Id: If32483458963939c89fae2850b8477f0614d139f
    Related-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750355
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=6158965423f3f3315b21eae2ea80403a12864511
Submitter: Zuul
Branch: master

commit 6158965423f3f3315b21eae2ea80403a12864511
Author: Bence Romsics <email address hidden>
Date: Fri Sep 4 14:49:58 2020 +0200

    Test metadata over IPv6

    To ssh into the instance we use an IPv4 floating IP,
    but then we test accessing metadata over IPv6.

    tempest.conf:

    [neutron_plugin_options]
    advanced_image_ref = ubuntu 16.04 does not but 18.04 and 20.04 work
    advanced_image_flavor_ref = d1 # ds512M
    advanced_image_ssh_user = ubuntu
    default_image_is_advanced = False

    Introduce new tempest(-plugin) boolean config option:
        neutron_plugin_options.ipv6_metadata
        Defaults to True.
    This controls whether the test should run since API extension based
    auto-discovery is naturally unavailable for the feature.

    Disable executing the test in ovn and older-than-victoria gate jobs.

    Change-Id: I7a1d2057489067e555cb247e473ad11e63dbdd86
    Related-Bug: #1460177

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/865036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865036
Committed: https://opendev.org/openstack/neutron/commit/bb2d706107c7be353aa5ebcd1fb735d13d6f6c49
Submitter: "Zuul (22348)"
Branch: master

commit bb2d706107c7be353aa5ebcd1fb735d13d6f6c49
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Nov 14 08:26:10 2022 +0100

    [doc] Metadata service supports IPv6 networks

    Metadata service now supports IPv6 networks. This patch removes
    this limitation in the documentation.

    Related-Bug: #1460177
    Change-Id: I1c50aef5e4aa60705f923f399e8d36c37a9d6289