[DHCP] Metadata IPv6 duplicated address when setting the interface

[Rodolfo Alonso]

When setting the IPv6 address for the metadata interface [2], it could happen that the IPv6 address can be considered as duplicated [2]. The IPv6 address in the DHCP namespace is set as "dadfailed tentative".

This is happening when using VLAN networks and HA DHCP agents and "enable_isolated_metadata=True". When the first DHCP agent boots, it configures the IPv6 address on the DHCP namespace interface (the metadata IPv6 address is METADATA_V6_IP='fe80::a9fe:a9fe'). Then the second DHCP agent tries to configure the same IPv6 address on the interface, but this IPv6 address is duplicated.

[2]https://github.com/openstack/neutron/blob/8455edda46b4a465e2f184b59ad31476ce79c6c4/neutron/agent/metadata/driver.py#L244-L246
[1]https://paste.opendev.org/show/bXDAHSHpvEAIHwdPjxrB/

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Bence:

Qq, when you implemented this feature, did you consider the HA case (maybe I'm doing something wrong)? Please ping me in IRC if you have some minutes.

Thanks!

[Bence Romsics (bence-romsics)]

I was able to reproduce this bug. Not just fe80::a9fe:a9fe, but 169.254.169.254 is also configured in multiple dhcp-agents.

devstack0 $ sudo ip netns exec qdhcp-$( openstack net show net1 -f value -c id ) ip a show dev tap$( openstack port list --device-owner network:dhcp --host $( hostname ) --network net1 -f value -c id | cut -b1-11 )
35: tap062c74c8-f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:2e:2e:43 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.2/24 brd 10.0.4.255 scope global tap062c74c8-f1
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/32 brd 169.254.169.254 scope global tap062c74c8-f1
       valid_lft forever preferred_lft forever
    inet6 2001:db8::2/32 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a9fe:a9fe/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2e:2e43/64 scope link
       valid_lft forever preferred_lft forever

devstack0a $ sudo ip netns exec qdhcp-$( openstack net show net1 -f value -c id ) ip a show dev tap$( openstack port list --device-owner network:dhcp --host $( hostname ) --network net1 -f value -c id | cut -b1-11 )
27: tapffa83554-c7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:84:7b:27 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.3/24 brd 10.0.4.255 scope global tapffa83554-c7
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/32 brd 169.254.169.254 scope global tapffa83554-c7
       valid_lft forever preferred_lft forever
    inet6 2001:db8::1/32 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe84:7b27/64 scope link
       valid_lft forever preferred_lft forever

Obviously we don't have mandatory DAD in ipv4, but if I try intentionally, there's still no duplication with v4:

$ sudo ip netns exec qdhcp-$( openstack net show net1 -f value -c id ) ping -c3 169.254.169.254
PING 169.254.169.254 (169.254.169.254) 56(84) bytes of data.
64 bytes from 169.254.169.254: icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from 169.254.169.254: icmp_seq=2 ttl=64 time=0.088 ms
64 bytes from 169.254.169.254: icmp_seq=3 ttl=64 time=0.121 ms

--- 169.254.169.254 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2038ms
rtt min/avg/max/mdev = 0.035/0.081/0.121/0.035 ms

I'm looking to see how we avoid the duplication with v4...

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

So this is also a problem with IPv4, but this protocol doesn't have a mechanism to detect duplicated IP addresses. Let's keep for now the focus on the IPv6 problem.

If we detect a duplicated IPv6 address on this network, that means there is other metadata agent spawned (on other DHCP agent). In that case, we should skip the configuration of the metadata server. That means we won't have HA for metadata in this network. If the DHCP agent with the active metadata server does down, we need to be able to, in the other DHCP agents, configure the IPv6 address and spawn a metadata server.

This mechanism (I don't know now how to implement it) will periodically check the existence of this IPv6 metadata address on other interface. If that is not present, then configure it and spawn the metadata server.

Do you have other alternative? Because having multiple metadata IPs is not an option, I think. And having something like VRRP is, IMO, over-engineering.

[Bence Romsics (bence-romsics)]

Sorry for the slow followup, but it took me some time until I started undestanding what's happening here. Let me still focus first on IPv4.

Basically the IPv4 link local RFC *does* prescribe duplicate address detection, but its behavior is all silent (at least in Linux). For details please see https://www.rfc-editor.org/rfc/rfc3927#section-2 (particularly subsections 2.2, 2.4 and 2.5).

When we have two dhcp-agents, both serving metadata, then we seemingly have the address 169.254.169.254 successfully configured in both dhcp namespaces. But in reality to conform to the RFC the kernel must track internally the configuration success (/state) of this address. This is analogous to IPv6's dadfailed state, but I believe there's no place in the v4 stack to expose this state information. I did not yet confirm this in kernel sources, but the behavior I'm seeing is consistent with the kernel tracking the dadfailed state and because of that not responding from the second (or further) namespaces.

If we need further confirmation, instead of reading kernel sources we could also just observe the ARP traffic during the 1st and 2nd attempts to configure the same link-local address. From which we could clearly infer the kernel's internal state.

This behavior means that when we have multiple dhcp-agents serving the same IPv4 network and serving IPv4 metadata too, then the metadata service itself is not redundant (and never was). While this could be considered a bug, I also believe it would be overengineering to introduce clustering or VRRP. The DHCP protocol was designed to easily allow active-active HA, but the metadata service was not. The very fact that we never received a bug report about metadata not being HA, despite this being the case for years, tells me that people are happy with the current state.

I believe that the only difference for us in IPv6 link-local address handling is the DAD failure being loud.

For IPv4 I don't have an idea how to improve the situation since the DAD state is private to the kernel and because of that we cannot react from any agent. I believe we should at least document the dhcp-agent-managed metadata not being redundant as a known limitation.

For IPv6 I see two options:
* Silence the DAD failure and by that behave the same way as we do in IPv4.
* Handle the DAD failure by periodically (but slowly) trying to re-configure the link-local address, and by that re-starting the duplicate address detection. Then the kernel's DAD logic would take care of the nitty-gritty details.

By the way I hope (but did not test yet) that when the l3-agent is deployed in an HA fashion, then the metadata service managed by it is also redundant.

What do you think?

[Pierre Riteau (priteau)]

See https://bugs.launchpad.net/neutron/+bug/1953165 for additional discussion of the same issue.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This is the same issue as described in LP#1953165. Let's continue the discussion there.