VM interface goes down because of short dhcp lease
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Hi,
We have an issue with our cloud VM, where eth0 goes down a short time, because they didn't get DHCP reply quickly enough.
The cause seems to be dnsmasq not responding to unicast dhcprequest and only getting reply with broadcast dhcprequest, which may be sent too late in time.
Here is the detailed problem and our installation:
* All server are ubuntu lucid.
* nova version are version 2011.3 from ppa (2011.3-
* lease time is 120 second (default - hard coded in nova/network/
* from the 120 second of lease timelife, dnsmasq generate the renew to a value near 60s (120/2) and rebind time to a value near 105s (7/8 * 120)
* we are using vlan and then we have several dnsmasq running, one for each vlan, serving only the given vlan
* guest is a simple ubuntu lucid with default dhcp3-client
First issue: dnsmasq don't reply to UNICAST dhcprequest: because when an unicast request come to nova-network node, since we have several dnsmasq running on some port (0.0.0.0:67), the last started will get the message. This is probably more a dnsmasq bug.
But because of this, dhcp client won't get any answer before trying to rebind. In this case, rebind send BROADCAST dhcprequest, and in this case all dnsmasq will get the message. Since dnsmasq only reply when it know the host, when all dnsmasq get the broadcast request, only the good one send the reply.
In most case everything is fine, dhcp client send the broadcast few seconds before lease expire time. But something (I'm pretty sure because of backoff time - interval between two dhcprequest) it don't send the broadcast request before lease expire time.
This never last too long, because if dhcp client would backoff after the lease expire time, it alter the backoff interval to match the expire time... + 1 second. So interface goes down and a second later dhcp request is send.
A solution could be to increase the lease time to more than 120s, but this seems to be hard-coded (I don't known if their is some reason). Also, it's seem to be an flag for dhcp_lease_time, but it's not used for dnsmasq command line parameter.
We are testing a workaround, which is to limit the backoff interval of dhclient to 10 second instead of default 2 minutes.
If something is not clear, don't hesitate to ask for more information.
I also add detail on software version and extract of dhcp client log
PS: we have trouble with interface going down, because we add a secondary IP (a VIP) with pacemaker. When interface goes goes, even the shortest time, secondary IP get lost.
Changed in nova: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
I have the same issue, especially on windows guests, sometimes the DHCPREQUEST is sent too late and the DHCP server answers with a NAK because the lease is not found (lease is expired).
We are testing with a longer lease time (yes you have to overwrite the hard-coded value of 120 in linux_net.py).
Hopefully a longer lease time will fix this issue and the dhcp_release switch will be used to free up private IPs of terminated instances before the lease expired so we don't waste fixed ips.
I don't think we should work on a client-based solution as they are many OS involved, but a switch to overwrite the lease time would be effective and simple to implement.