dnsmasq option --dhcp-lease-max prevents startup for more than 1 network

[Brian J. Murrell]

If one configures more than one network to use DHCP, multiple dnsmasq processes are started, one for each network. The problem is that each process has --dhcp-lease-max configured for the number of leases in it's own network. But since multiple dnsmasq processes share the standard lease file it's possible that more leases are already in the lease file than the newly configured network yielding an error when the new dnsmasq tries to start:

dnsmasq: too many stored leases

Steps to Reproduce:
1. define a /24 network in libvirt
2. obtain (or simulate by populating the leases file) 30 leases
3. define a second /28 network

Actual results:
libvirtd will report:

09:46:47.004: error : virRunWithHook:857 : internal error '/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/test.pid --conf-file= --listen-address 10.0.1.1 --except-interface lo --dhcp-range
10.0.1.1,10.0.1.14 --dhcp-lease-max=14' exited with non-zero status 5 and signal 0:
dnsmasq: too many stored leases

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: libvirt-bin 0.8.3-1ubuntu14
ProcVersionSignature: Ubuntu 2.6.35-24.42-generic 2.6.35.8
Uname: Linux 2.6.35-24-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Fri Feb 4 07:53:44 2011
ProcEnviron:
 PATH=(custom, user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: libvirt

[Brian (brian-redhat-bugs)]

Description of problem:
If one configures more than one network to use DHCP, multiple dnsmasq processes are started, one for each network. The problem is that each process has --dhcp-lease-max configured for the number of leases in it's own network. But since multiple dnsmasq processes share the standard lease file it's possible that more leases are already in the lease file than the newly configured network yielding an error when the new dnsmasq tries to start:

dnsmasq: too many stored leases

Version-Release number of selected component (if applicable):
0.8.5

How reproducible:
100%

Steps to Reproduce:
1. define a /24 network in libvirt
2. obtain (or simulate by populating the leases file) 30 leases
3. define a second /28 network

Actual results:
libvirtd will report:

09:46:47.004: error : virRunWithHook:857 : internal error '/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/test.pid --conf-file= --listen-address 10.0.1.1 --except-interface lo --dhcp-range 10.0.1.1,10.0.1.14 --dhcp-lease-max=14' exited with non-zero status 5 and signal 0:
dnsmasq: too many stored leases

Expected results:
Should add the network and start the dnsmasq process without error.

Additional info:
Probably the "-l" option to dnsmasq would be useful here to create per network lease files.

[Brian (brian-redhat-bugs)]

Created attachment 476445
allocate a lease file per dnsmasq process

This patch uses the --dhcp-leasefile option to give each dnsmasq it's own lease file.

[Brian J. Murrell (brian-interlinx)]

There is a patch to fix this problem in the upstream bug report.

[Serge Hallyn (serge-hallyn)]

A natty libvirt tree with this patch added is at https://code.launchpad.net/~serge-hallyn/ubuntu/natty/libvirt/bug-713071/ .

I will create a ppa with the maverick libvirt with all patches for various bugs applied. I just don't know how to best approach this in terms of ordering SRUs.

[Serge Hallyn (serge-hallyn)]

A package for maverick with the proposed patch is building at ppa:serge-hallyn/libvirt-mav. The source is at https://code.launchpad.net/~serge-hallyn/ubuntu/maverick/libvirt/bugall/. The package and source tree also include the fix for bug 668032.

[Kenni (kenni-redhat-bugs)]

Duplicate of ticket #663664 - but that ticket doesn't contain a patch.

[Laine (laine-redhat-bugs)]

An updated patch to fix this has been committed upstream:

   commit 13c00dde3171b3a38d23cceb3f9151cb6cac3dad
   Author: Laine Stump <email address hidden>
   Date: Fri Mar 11 13:20:48 2011 -0500

   network driver: Use a separate dhcp leases file for each network

This will be in libvirt-0.9.0.

[Laine (laine-redhat-bugs)]

*** This bug has been marked as a duplicate of bug 663664 ***

[Brian (brian-redhat-bugs)]

(In reply to comment #3)
>
> This will be in libvirt-0.9.0.

It's really quite a pity that libvirt-0.9.0 didn't get cut in time for FC15 (or an 0.8.x.x bugfix release with this fix in it). Since I have already fixed this bug on my Ubuntu 10.10 installation I'm afraid I'm not going to the effort to fix it again, on a brand new O/S release (FC15) and will just revert to my Ubuntu installation where this at least works.

In fact, without having done a terrible amount of debugging in FC15 (since I am abandoning it anyway), it would appear, at least, that multiple dnsmasq processes sharing a dnsmasq.leases file actually overwrite the file from under each other, rather than the previous behavior where they all (at least mostly) cooperatively shared the file and one just ran out of leases quickly.

[Laine (laine-redhat-bugs)]

(In reply to comment #5)
>
> It's really quite a pity that libvirt-0.9.0 didn't get cut in time for FC15
> (or an 0.8.x.x bugfix release with this fix in it).

There is a new libvirt release once each month. There is also a cutoff date for rebasing packages for each new Fedora release. Whatever is the most recent release of libvirt on the cutoff date, that is the release that is put into Fedora for the release. In this case, libvirt-0.8.8 was the most recent on the F15 cutoff date.

virt-related packages like libvirt are not rebased for already-released versions of Fedora, but there are two methods of getting bugfixes:

1) request that a bugfix be pulled in as a patch to the release in use on that version of Fedora. When this happens, a patch release will be made (e.g., the version will go from 0.8.8-1 to 0.8.8-2). There is no automated method for bugfixes to go into libvirt of a released Fedora, an explicit request must be made.

2) use the fedora-virt-preview repo when doing "yum update". This will give you a build of the most recent upstream release of each of the packages. The F14 instructions on the wiki page will also work for F15 (as soon as the F15 version of the repo is online):

  http://fedoraproject.org/wiki/Virtualization_Preview_Repository

For a list of all the packages included in virt-preview, see:

http://repos.fedorapeople.org/repos/jforbes/virt-preview/fedora-14/x86_64/

I am following up with jforbes to learn when the F15 incarnation of the virt-preview repo will be created.

[Serge Hallyn (serge-hallyn)]

The fix for this bug is in the oneiric package. The linked bzr tree can still be used if we choose to SRU for maverick and natty. However, I'd like to hear from anyone who needs this SRUd before risking a regression for a bug which doesn't seem to be affecting most people's uses.