Incorrect IPv6 lease entries cause DHCPNAKs from Dnsmasq in dual stack DHCPv6 stateful network configuration
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
Medium
|
Rodolfo Alonso |
Bug Description
In a dual stack network with DHCPv6 stateful network configuration, Neutron DHCP Agent
uses the IPv4 lease entry format to write IPv6 lease entries in the fake lease file
used to bootstrap Dnsmasq after agent is restarted or migrated from one node to the
other. As a result, the lease file gets corrupted and overwritten by Dnsmasq after encountering the
invalid IPv6 lease entries and this causes a DHCPNAK when IPv4 clients try to renew
their leases with the Dnsmasq process.
From the Dnsmasq mailing list, a lease entry for DHCPv4 consists of these fields
seperated by spaces:
- The expiration time (seconds since unix epoch) or duration
(if dnsmasq is compiled with HAVE_BROKEN_RTC) of the lease.
0 means infinite.
- The link address, in format XX-YY:YY:YY[...], where XX is the ARP
hardware type. "XX-" may be omitted for Ethernet.
- The IPv4 address
- The hostname (sent by the client or assigned by dnsmasq)
or '*' for none.
- The client identifier (colon-separated hex bytes)
or '*' for none.
While a DHCPv6 lease entry has these fields:
- The expiration time or duration
- The IAID as a Big Endian decimal number, prefixed by T for
IA_TAs (temporary addresses).
- The IPv6 address
- The hostname or '*'
- The client DUID (colon-separated hex bytes) or '*' if unknown.[1]
For DHCPv6, there must also be exactly one special entry indicating
the DUID of the server. This line contains two fields:
- The string "duid".
- The DUID of the server.
See http://
more info about the discussion.
Symptoms:
---------
Currently, the _output_
writes lease entries for both IPv4 and IPv6 like this:
1506979604 fa:16:3e:07:b4:26 10.0.1.128 * *
1506979604 fa:16:3e:07:b4:26 [2404:130:
1506979604 fa:16:3e:20:20:a9 10.0.1.83 * *
1506979604 fa:16:3e:26:cf:8a 10.0.1.135 * *
Changed in neutron: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Brian Haley (brian-haley) wrote : | #1 |
Oladimeji Fayomi (fayomidimeji) wrote : | #2 |
Hi Brian,
It's the v6 entry using a MAC instead of IAID and also v6 addresses being placed inside square brackets. The server DUID is included just fine, I omitted it in my paste, my apologies.
According to the DHCPv6 RFC 3315, The DHCP client creates the IAID and includes it in a solicit message to the DHCP server when requesting an address, so there is no way for use to determine the IAID before hand like we do for MAC addresses. There are three possible solutions that I see:
1.) Generate the IAID for the ports attached to instances like it's currently done for MAC addresses and find a way to make the instance OS aware of the IAIDs
2.) Extract the IAID from the initial DHCP Solicit message sent by the instances when negotiating an IPv6 address, store it and subsequently use it to generate IPv6 lease entries for the instances later.
3.) Filter out IPv6 addresses from being written to the lease file when a DHCP agent is restarted or moved from one node to the other.
Brian Haley (brian-haley) wrote : | #3 |
So here's my take on the options you listed:
# 1 - there is not really a way for us to inject a value into the instance like this from neutron
# 2 - is the IAID known to dnsmasq after the solicit such that it could write it somewhere? although it seems difficult to get right.
# 3 - i guess this looks like the best option - don't write any IPv6 leases to the file on bootstrap
Do you want to work on this change?
Oladimeji Fayomi (fayomidimeji) wrote : | #4 |
Hi Brian,
I agree, option #1 seems impossible and #2 is complicated. #3 is the easiest, I'll have to confirm the effect of doing that on IPv6 and/or DHCPv6 and let you. But yes, I'm happy to work on this change.
Thanks
Dimeji
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #6 |
Fix proposed to branch: master
Review: https:/
Changed in neutron: | |
assignee: | nobody → Rodolfo Alonso (rodolfo-alonso-hernandez) |
status: | Confirmed → In Progress |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master) | #7 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 4747de23d80ab32
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000
Remove IPv6 addresses in dnsmasq leases file
IPv6 address format in dnsmasq leases file is incorrect (correct format
is described in bug description). This bad formatting generates the
following error when initializing dnsmasq:
dnsmasq[
1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:
This patch removes the IPv6 addresses from the leases file, as proposed
in the bug, because the DHCP agent does not have the IAID (identity
association identifier) of each IPv6 address assigned.
In case of agent restart, dnsmasq won't have any IPv6 address in the
leases file, but the hosts file and the additional hosts file will
contain all MAC/IPv6 previous assignations. When the IPv6 client sends
a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
At the same time, the client will request to the server the same address:
DHCPDISCO
DHCPOFFER
DHCPREQUE
DHCPACK(
Once dnsmasq updates the leases database, rewrites the leases file with the
new IPv6 address (including the IAID) and the server DUID (if not present).
Change-Id: Ib1b2f284ab81f1
Closes-Bug: #1722126
Changed in neutron: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky) | #8 |
Fix proposed to branch: stable/rocky
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens) | #9 |
Fix proposed to branch: stable/queens
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens) | #10 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 828daf9f133c51f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000
Remove IPv6 addresses in dnsmasq leases file
IPv6 address format in dnsmasq leases file is incorrect (correct format
is described in bug description). This bad formatting generates the
following error when initializing dnsmasq:
dnsmasq[
1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:
This patch removes the IPv6 addresses from the leases file, as proposed
in the bug, because the DHCP agent does not have the IAID (identity
association identifier) of each IPv6 address assigned.
In case of agent restart, dnsmasq won't have any IPv6 address in the
leases file, but the hosts file and the additional hosts file will
contain all MAC/IPv6 previous assignations. When the IPv6 client sends
a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
At the same time, the client will request to the server the same address:
DHCPDISCO
DHCPOFFER
DHCPREQUE
DHCPACK(
Once dnsmasq updates the leases database, rewrites the leases file with the
new IPv6 address (including the IAID) and the server DUID (if not present).
Change-Id: Ib1b2f284ab81f1
Closes-Bug: #1722126
(cherry picked from commit 4747de23d80ab32
tags: | added: in-stable-queens |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky) | #11 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit 18f2cea730ae1dc
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000
Remove IPv6 addresses in dnsmasq leases file
IPv6 address format in dnsmasq leases file is incorrect (correct format
is described in bug description). This bad formatting generates the
following error when initializing dnsmasq:
dnsmasq[
1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:
This patch removes the IPv6 addresses from the leases file, as proposed
in the bug, because the DHCP agent does not have the IAID (identity
association identifier) of each IPv6 address assigned.
In case of agent restart, dnsmasq won't have any IPv6 address in the
leases file, but the hosts file and the additional hosts file will
contain all MAC/IPv6 previous assignations. When the IPv6 client sends
a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
At the same time, the client will request to the server the same address:
DHCPDISCO
DHCPOFFER
DHCPREQUE
DHCPACK(
Once dnsmasq updates the leases database, rewrites the leases file with the
new IPv6 address (including the IAID) and the server DUID (if not present).
Change-Id: Ib1b2f284ab81f1
Closes-Bug: #1722126
(cherry picked from commit 4747de23d80ab32
tags: | added: in-stable-rocky |
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.0.0b2 | #12 |
This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.3 | #13 |
This issue was fixed in the openstack/neutron 13.0.3 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.6 | #14 |
This issue was fixed in the openstack/neutron 12.0.6 release.
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #15 |
Fix proposed to branch: master
Review: https:/
Slawek Kaplonski (slaweq) wrote : auto-abandon-script | #16 |
This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.
Changed in neutron: | |
assignee: | Rodolfo Alonso (rodolfo-alonso-hernandez) → nobody |
tags: | added: timeout-abandon |
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master) | #17 |
Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https:/
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.
Changed in neutron: | |
assignee: | nobody → Rodolfo Alonso (rodolfo-alonso-hernandez) |
OpenStack Infra (hudson-openstack) wrote : | #18 |
Change abandoned by "Gaudenz Steinlin <email address hidden>" on branch: master
Review: https:/
Reason: Merged into https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master) | #19 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 6bc1c00d66ced82
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
Change-Id: I40761b30563749
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby) | #20 |
Fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria) | #21 |
Fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri) | #22 |
Fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train) | #23 |
Fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein) | #24 |
Fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby) | #25 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit 2ede99559b32d60
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
Change-Id: I40761b30563749
(cherry picked from commit 6bc1c00d66ced82
tags: | added: in-stable-wallaby |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria) | #26 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit f9b1f05def90f75
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
Change-Id: I40761b30563749
(cherry picked from commit 6bc1c00d66ced82
tags: | added: in-stable-victoria |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri) | #27 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit e0748a58efe6fdb
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
Change-Id: I40761b30563749
(cherry picked from commit 6bc1c00d66ced82
tags: | added: in-stable-ussuri |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train) | #28 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/train
commit 229ecdd0083c399
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
This backport contains some test fixes to make the tests work with
Python 2.7 and the PEP8 N322 check.
Changed file: neutron/
Change-Id: I40761b30563749
(cherry picked from commit 6bc1c00d66ced82
tags: | added: in-stable-train |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein) | #29 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/stein
commit 7f63139424adcbb
Author: Gaudenz Steinlin <email address hidden>
Date: Mon Nov 16 17:41:18 2020 +0100
Copy existing IPv6 leases to generated lease file
Because the DHCP agent does not know the IAID (identity association
identifier) of assigned IPv6 addresses it's not possible to generate the
lease file including IPv6 leases. Because of this IPv6 addresses are
excluded when generating the lease file in case of DHCP agent restarts.
This causes DHCPv6 clients to fail to RENEW their lease and to go
through a full address discovery cycle with possible short connectivity
disruption.
This commit copies the existing IPv6 leaes from an already existing
lease file if present. While this does not allow for DHCP agent
failover, this is still better than just skipping the IPv6 addresses.
A lease file without the IPv6 addresses is still generated if an agent
is migrated to a different host.
This commit complements the fix implemented in
Ib1b2f284ab
leases as otherwise the lease file would be invalid and all leases would
be lost. It does not change the behavior for still valid IPv4 leases.
With this issue fixed an additional fix is required to not loose DHCPv6
leases when the agent restarts dnsmasq. Currently the DHCP agent
regenerates all configuration files on restart. This means that DHCPv6
leases are lost as they can't be regenerated. This changes the agent to
only delete the config files if the agent's ports are also removed.
Closes-Bug: #1722126
Related-Change: Ib1b2f284ab81f1
This backport contains some test fixes to make the tests work with
Python 2.7 and the PEP8 N322 check.
Changed file:
Conflicts:
Change-Id: I40761b30563749
(cherry picked from commit 6bc1c00d66ced82
tags: | added: in-stable-stein |
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.0 | #30 |
This issue was fixed in the openstack/neutron 16.4.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.2.0 | #31 |
This issue was fixed in the openstack/neutron 17.2.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.1.0 | #32 |
This issue was fixed in the openstack/neutron 18.1.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1 | #33 |
This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron stein-eol | #34 |
This issue was fixed in the openstack/neutron stein-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron train-eol | #35 |
This issue was fixed in the openstack/neutron train-eol release.
Hi, can you give more info on what is causing the failure? Is it the v6 entry using a MAC instead of IAID, or missing the server DUID, or something else? Just trying to understand this better, but it does look like a problem. Thanks.