Incorrect IPv6 lease entries cause DHCPNAKs from Dnsmasq in dual stack DHCPv6 stateful network configuration

Bug #1722126 reported by Oladimeji Fayomi on 2017-10-08
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Rodolfo Alonso

Bug Description

In a dual stack network with DHCPv6 stateful network configuration, Neutron DHCP Agent
uses the IPv4 lease entry format to write IPv6 lease entries in the fake lease file
used to bootstrap Dnsmasq after agent is restarted or migrated from one node to the
other. As a result, the lease file gets corrupted and overwritten by Dnsmasq after encountering the
invalid IPv6 lease entries and this causes a DHCPNAK when IPv4 clients try to renew
their leases with the Dnsmasq process.

From the Dnsmasq mailing list, a lease entry for DHCPv4 consists of these fields
seperated by spaces:

- The expiration time (seconds since unix epoch) or duration
  (if dnsmasq is compiled with HAVE_BROKEN_RTC) of the lease.
  0 means infinite.

- The link address, in format XX-YY:YY:YY[...], where XX is the ARP
  hardware type. "XX-" may be omitted for Ethernet.

- The IPv4 address

- The hostname (sent by the client or assigned by dnsmasq)
  or '*' for none.

- The client identifier (colon-separated hex bytes)
  or '*' for none.

While a DHCPv6 lease entry has these fields:

- The expiration time or duration

- The IAID as a Big Endian decimal number, prefixed by T for
  IA_TAs (temporary addresses).

- The IPv6 address

- The hostname or '*'

- The client DUID (colon-separated hex bytes) or '*' if unknown.[1]

For DHCPv6, there must also be exactly one special entry indicating
the DUID of the server. This line contains two fields:

- The string "duid".

- The DUID of the server.

See http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2016q2/010595.html for
more info about the discussion.

Symptoms:
---------
Currently, the _output_init_lease_file function in neutron/agent/linux/dhcp.py
writes lease entries for both IPv4 and IPv6 like this:

1506979604 fa:16:3e:07:b4:26 10.0.1.128 * *
1506979604 fa:16:3e:07:b4:26 [2404:130:21:4000::24c] * *
1506979604 fa:16:3e:20:20:a9 10.0.1.83 * *
1506979604 fa:16:3e:26:cf:8a 10.0.1.135 * *

Miguel Lavalle (minsel) on 2017-10-10
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Brian Haley (brian-haley) wrote :

Hi, can you give more info on what is causing the failure? Is it the v6 entry using a MAC instead of IAID, or missing the server DUID, or something else? Just trying to understand this better, but it does look like a problem. Thanks.

Oladimeji Fayomi (fayomidimeji) wrote :

Hi Brian,

It's the v6 entry using a MAC instead of IAID and also v6 addresses being placed inside square brackets. The server DUID is included just fine, I omitted it in my paste, my apologies.

According to the DHCPv6 RFC 3315, The DHCP client creates the IAID and includes it in a solicit message to the DHCP server when requesting an address, so there is no way for use to determine the IAID before hand like we do for MAC addresses. There are three possible solutions that I see:

1.) Generate the IAID for the ports attached to instances like it's currently done for MAC addresses and find a way to make the instance OS aware of the IAIDs

2.) Extract the IAID from the initial DHCP Solicit message sent by the instances when negotiating an IPv6 address, store it and subsequently use it to generate IPv6 lease entries for the instances later.

3.) Filter out IPv6 addresses from being written to the lease file when a DHCP agent is restarted or moved from one node to the other.

Brian Haley (brian-haley) wrote :

So here's my take on the options you listed:

# 1 - there is not really a way for us to inject a value into the instance like this from neutron

# 2 - is the IAID known to dnsmasq after the solicit such that it could write it somewhere? although it seems difficult to get right.

# 3 - i guess this looks like the best option - don't write any IPv6 leases to the file on bootstrap

Do you want to work on this change?

Oladimeji Fayomi (fayomidimeji) wrote :

Hi Brian,

I agree, option #1 seems impossible and #2 is complicated. #3 is the easiest, I'll have to confirm the effect of doing that on IPv6 and/or DHCPv6 and let you. But yes, I'm happy to work on this change.

Thanks
Dimeji

Fix proposed to branch: master
Review: https://review.openstack.org/630951

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/630951
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4747de23d80ab32eea073846221adf94987f755b
Submitter: Zuul
Branch: master

commit 4747de23d80ab32eea073846221adf94987f755b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000

    Remove IPv6 addresses in dnsmasq leases file

    IPv6 address format in dnsmasq leases file is incorrect (correct format
    is described in bug description). This bad formatting generates the
    following error when initializing dnsmasq:
      dnsmasq[20603]: failed to parse lease database, invalid line: \
        1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:8295:5339::43] * ...

    This patch removes the IPv6 addresses from the leases file, as proposed
    in the bug, because the DHCP agent does not have the IAID (identity
    association identifier) of each IPv6 address assigned.

    In case of agent restart, dnsmasq won't have any IPv6 address in the
    leases file, but the hosts file and the additional hosts file will
    contain all MAC/IPv6 previous assignations. When the IPv6 client sends
    a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
    At the same time, the client will request to the server the same address:
      DHCPDISCOVER(tap2c14823a-e6) fa:16:3e:54:c6:8e
      DHCPOFFER(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPREQUEST(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPACK(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e \
        host-fd5b-1fd5-8295-5339--43

    Once dnsmasq updates the leases database, rewrites the leases file with the
    new IPv6 address (including the IAID) and the server DUID (if not present).

    Change-Id: Ib1b2f284ab81f1c4af7b08b5257b45a3f6e79c3e
    Closes-Bug: #1722126

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/633201
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=828daf9f133c51fb18b0147760c5a2d4f3b9f663
Submitter: Zuul
Branch: stable/queens

commit 828daf9f133c51fb18b0147760c5a2d4f3b9f663
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000

    Remove IPv6 addresses in dnsmasq leases file

    IPv6 address format in dnsmasq leases file is incorrect (correct format
    is described in bug description). This bad formatting generates the
    following error when initializing dnsmasq:
      dnsmasq[20603]: failed to parse lease database, invalid line: \
        1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:8295:5339::43] * ...

    This patch removes the IPv6 addresses from the leases file, as proposed
    in the bug, because the DHCP agent does not have the IAID (identity
    association identifier) of each IPv6 address assigned.

    In case of agent restart, dnsmasq won't have any IPv6 address in the
    leases file, but the hosts file and the additional hosts file will
    contain all MAC/IPv6 previous assignations. When the IPv6 client sends
    a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
    At the same time, the client will request to the server the same address:
      DHCPDISCOVER(tap2c14823a-e6) fa:16:3e:54:c6:8e
      DHCPOFFER(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPREQUEST(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPACK(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e \
        host-fd5b-1fd5-8295-5339--43

    Once dnsmasq updates the leases database, rewrites the leases file with the
    new IPv6 address (including the IAID) and the server DUID (if not present).

    Change-Id: Ib1b2f284ab81f1c4af7b08b5257b45a3f6e79c3e
    Closes-Bug: #1722126
    (cherry picked from commit 4747de23d80ab32eea073846221adf94987f755b)

tags: added: in-stable-queens

Reviewed: https://review.openstack.org/633200
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=18f2cea730ae1dcc8c5c557078ab29acb89a356d
Submitter: Zuul
Branch: stable/rocky

commit 18f2cea730ae1dcc8c5c557078ab29acb89a356d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 14 18:47:21 2019 +0000

    Remove IPv6 addresses in dnsmasq leases file

    IPv6 address format in dnsmasq leases file is incorrect (correct format
    is described in bug description). This bad formatting generates the
    following error when initializing dnsmasq:
      dnsmasq[20603]: failed to parse lease database, invalid line: \
        1547121093 fa:16:3e:a0:3a:9a [fd5b:1fd5:8295:5339::43] * ...

    This patch removes the IPv6 addresses from the leases file, as proposed
    in the bug, because the DHCP agent does not have the IAID (identity
    association identifier) of each IPv6 address assigned.

    In case of agent restart, dnsmasq won't have any IPv6 address in the
    leases file, but the hosts file and the additional hosts file will
    contain all MAC/IPv6 previous assignations. When the IPv6 client sends
    a DHCPDISCOVER, dnsmasq will offer the same IPv6 address to this client.
    At the same time, the client will request to the server the same address:
      DHCPDISCOVER(tap2c14823a-e6) fa:16:3e:54:c6:8e
      DHCPOFFER(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPREQUEST(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e
      DHCPACK(tap2c14823a-e6) fd5b:1fd5:8295:5339::43 fa:16:3e:54:c6:8e \
        host-fd5b-1fd5-8295-5339--43

    Once dnsmasq updates the leases database, rewrites the leases file with the
    new IPv6 address (including the IAID) and the server DUID (if not present).

    Change-Id: Ib1b2f284ab81f1c4af7b08b5257b45a3f6e79c3e
    Closes-Bug: #1722126
    (cherry picked from commit 4747de23d80ab32eea073846221adf94987f755b)

tags: added: in-stable-rocky

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.

This issue was fixed in the openstack/neutron 13.0.3 release.

This issue was fixed in the openstack/neutron 12.0.6 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers