dnsmasq does not remove leases for deleted VMs - leases and host files point to different MACS
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
Medium
|
Unassigned |
Bug Description
We see this sporadically, sometimes it works, sometimes it doesn't. We delete VMs, then create new ones. The new fixed IP can't get an IP via DHCP, because the leases file still points to MAC of some old, deleted VMs. The host file is correctly updated.
For example we see in syslog/messages:
Jul 26 16:45:04 stan dnsmasq-
Jul 26 16:45:04 stan dnsmasq-
And tcpdump on tap interface of qdhcp indicates it is receiving the DHCP requests, but nothing is sent out.
On inspection of the leases and host file, we see the MAC address differs for the fixed IP:
[root@stan ~]# cat /var/opt/
fa:16:3e:
[root@stan ~]# cat /var/opt/
1532649702 fa:16:3e:ae:c4:b0 10.96.0.21 test-pf9-
On another dhcp-agent host, we saw the leases file pointed to some OTHER old third MAC.
We then deleted this VM. We observed that the lease was removed on one host, but it remained on the other host. So it appears to be random. Enabling DEBUG logs shows the dhcp_release command is being invoked, without error:
2018-07-26 16:52:06.214 10660 DEBUG neutron.
2018-07-26 16:52:06.217 10660 DEBUG neutron.
2018-07-26 16:52:06.498 10660 DEBUG neutron.
2018-07-26 16:52:06.505 10660 DEBUG neutron.
The entry was removed from hosts file, but remains in leases despite above dhcp_release remaining. If we spin up a new VM with this fixed IP, it can't get an IP due to old lease:
root@krusty:~# cat /var/opt/
root@krusty:~# cat /var/opt/
1532735433 fa:16:3e:2f:29:37 10.96.0.21 test-pf9-
root@krusty:~#
Changed in neutron: | |
status: | Incomplete → New |
We see this on both Ubuntu14 and Centos 7.2 and Centos 7.4. Kernel and OS versions:
[root@stan pf9]# uname -a 693.11. 6.el7.x86_ 64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux 2.66-14. el7_1.x86_ 64 6.16-7. el7.x86_ 64 utils-2. 66-14.el7_ 1.x86_64
Linux stan.platform9.sys 3.10.0-
[root@stan pf9]# rpm -qa | grep dns
dnsmasq-
ldns-1.
dnsmasq-
root@krusty:~# uname -a dfsg-3ubuntu0. 8 amd64 Clients provided with BIND dfsg-3ubuntu0. 8 amd64 DNS Shared Library used by BIND
Linux krusty 3.13.0-58-generic #97-Ubuntu SMP Wed Jul 8 02:56:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@krusty:~# dpkg -l | grep dns
ii dnsmasq 2.68-1ubuntu0.2 all Small caching DNS proxy and DHCP/TFTP server
ii dnsmasq-base 2.68-1ubuntu0.2 amd64 Small caching DNS proxy and DHCP/TFTP server
ii dnsmasq-utils 2.68-1ubuntu0.2 amd64 Utilities for manipulating DHCP leases
ii dnsutils 1:9.9.5.
ii libdns100 1:9.9.5.