DHCPNAK after neutron-dhcp-agent restart

[Han Zhou]

After rolling out a configuration change, we restarted neutron-dhcp-agent service, and then dnsmasq logs start flooding: DHCPNAK ... lease not found.
DHCPNAK is replied by dnsmasq for all DHCPREQUEST renews from all VMs. However the MAC and IP pairs exist in host files.
The log flooding increases when more and more VMs start renewing and they keep retrying until IP expire and send DHCPDISCOVER and reinit the IP.
The log flooding gradually disappears when the VMs IP expire and send DHCPDISCOVER, to which dnsmasq respond DHCPOFFER properly.

Analysis:
I noticed that option --leasefile-ro is used in dnsmasq command when started by neutron dhcp-agent. According to dnsmasq manual, this option should be used together with --dhcp-script to customize the lease database. However, the option --dhcp-script was removed when fixing bug 1202392.
Because of this, dnsmasq will not save lease information in persistent storage, and when it is restarted, lease information is lost.

Solution:
Simply replace --leasefile-ro by --dhcp-leasefile=<path to dhcp runtime files>/lease would solve the problem. (patch attached)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/108272

[Brian Haley (brian-haley)]

This seems to be higher than a low priority. We recently did an upgrade and started seeing this today. One of the side-effects is that a VM could de-configure it's IP address for a short time (right after the NAK), leading to unreachability. It always seems to get the same IP on the next DISCOVER, but it's still noticable to customers.

[Armando Migliaccio (armando-migliaccio)]

Grenade is affected because we need an upgrade patch to reflect the change in dhcp filters.

[Ihar Hrachyshka (ihar-hrachyshka)]

I guess we should kill release note at: https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#Known_Issues_6 Correct?

[Han Zhou (zhouhan)]

We should not kill the release note but update it so that dhcp.filter is applied for the dead code cleanup: https://review.openstack.org/#/c/152398/

[Ihar Hrachyshka (ihar-hrachyshka)]

@Han, it seems that rootwrap filter update is not required though? Meaning, Kilo agent will still work with old filter?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152080
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=74a16fde1c9972dc3c5d07215ca9d5e8f2e23d70
Submitter: Jenkins
Branch: master

commit 74a16fde1c9972dc3c5d07215ca9d5e8f2e23d70
Author: Alexey I. Froloff <email address hidden>
Date: Mon Feb 2 13:44:14 2015 +0300

    Pass '--dhcp-authoritative' option to dnsmasq

    When dnsmasq is restarted, it forgets about all leases (since it runs
    with leasefile-ro option). When client tries to renew its lease, dnsmasq
    sends DHCPNAK reply with message "lease not found". Then client shuts
    down the network and re-request lease from DHCP server (and gets exactly
    same IP address). There's a small network downtime which affects
    services, like zookeeper, running in VMs.

    Change-Id: Ieff0236670c1403b5d79ad8e50d7574c1b694e34
    Closes-Bug: #1345947

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/153182

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/154052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153182
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed799e38fe740621776aab51c95ec1db248af997
Submitter: Jenkins
Branch: stable/juno

commit ed799e38fe740621776aab51c95ec1db248af997
Author: Alexey I. Froloff <email address hidden>
Date: Mon Feb 2 13:44:14 2015 +0300

    Pass '--dhcp-authoritative' option to dnsmasq

    When dnsmasq is restarted, it forgets about all leases (since it runs
    with leasefile-ro option). When client tries to renew its lease, dnsmasq
    sends DHCPNAK reply with message "lease not found". Then client shuts
    down the network and re-request lease from DHCP server (and gets exactly
    same IP address). There's a small network downtime which affects
    services, like zookeeper, running in VMs.

    Change-Id: Ieff0236670c1403b5d79ad8e50d7574c1b694e34
    Closes-Bug: #1345947
    Co-Authored-By: Kevin Bringard <email address hidden>
    (cherry picked from commit 74a16fde1c9972dc3c5d07215ca9d5e8f2e23d70)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154052
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c3a88a578f3e484b5dc2a227a46e8546e581f8a3
Submitter: Jenkins
Branch: stable/icehouse

commit c3a88a578f3e484b5dc2a227a46e8546e581f8a3
Author: Alexey I. Froloff <email address hidden>
Date: Mon Feb 2 13:44:14 2015 +0300

    Pass '--dhcp-authoritative' option to dnsmasq

    When dnsmasq is restarted, it forgets about all leases (since it runs
    with leasefile-ro option). When client tries to renew its lease, dnsmasq
    sends DHCPNAK reply with message "lease not found". Then client shuts
    down the network and re-request lease from DHCP server (and gets exactly
    same IP address). There's a small network downtime which affects
    services, like zookeeper, running in VMs.

    Change-Id: Ieff0236670c1403b5d79ad8e50d7574c1b694e34
    Closes-Bug: #1345947
    (cherry picked from commit 74a16fde1c9972dc3c5d07215ca9d5e8f2e23d70)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Carl Baldwin (<email address hidden>) on branch: master
Review: https://review.openstack.org/108272
Reason: Han, your work here has been greatly appreciated. I hope to see more contributions from you. Sometimes we do find a different way to go.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/186003
Reason: The Kevin's patch is merged, so we don't need that one anymore.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/185996
Reason: Kevin's patch is merged, now it's not needed to revert.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/108272
Reason: https://review.openstack.org/#/c/185486/ is merged handling both the scenario that this patch tried to solve, and a regression that was introduced by an alternative patch that ended merged in the tree; so no need for this patch anymore.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/185971
Reason: Not needed anymore, Kevin's patch is merged.

[Alexander Bozhenko (alexbozhenko)]

The fix is introducing new issue that is fixed here:
https://bugs.launchpad.net/neutron/+bug/1457900

[Sean Dague (sdague)]

This grenade bug was last updated over 180 days ago, as grenade
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.