Please run dnsmasq in such a way that it can also be used on the host — to look up the VMs' names

[Seth Arnold]

First, apologies for the complicated configuration.

I use the 'uvt' front-end to libvirt-managed kvm guest machines. I also have a LAN with a router that does local dynamic DNS configuration with the client-provided dhcp hostnames.

I would like to resolve hostnames on my machine for both VM guests and LAN hosts. This configuration worked for 12.04 LTS, 12.10, but does not work for Raring. I have followed the directions at https://wiki.ubuntu.com/SecurityTeam/TestingEnvironment#Networking_with_libvirt :

> Put a line into /etc/dhcp/dhclient.conf like so:
>
> prepend domain-name-servers 192.168.122.1;
>
> Disable the system dnsmasq to prevent it from looping with libvirt's dnsmasq by modifying
> /etc/NetworkManager/NetworkManager.conf to comment out the following line:
>
> #dns=dnsmasq

My /etc/resolv.conf when the wireless is up:
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.122.1
nameserver 192.168.1.1

My router / DHCP / Dynamic DNS server is on 192.168.1.1.
My laptop ("hunt") is currently allocated 192.168.1.14.
The VMs on my laptop run in 192.168.122.0/24.
An instance of dnsmasq started by libvirt runs on 192.168.122.1 on my laptop.
My panda ("dean") is currently allocated 192.168.1.236.

Here's trying to look up dean from my laptop:

$ host dean
Host dean not found: 3(NXDOMAIN)
$ host dean 192.168.122.1
Using domain server:
Name: 192.168.122.1
Address: 192.168.122.1#53
Aliases:

Host dean not found: 3(NXDOMAIN)
$ host dean 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:

$ host 192.168.1.236
236.1.168.192.in-addr.arpa domain name pointer dean.
$ host 192.168.1.236 192.168.122.1
Using domain server:
Name: 192.168.122.1
Address: 192.168.122.1#53
Aliases:

236.1.168.192.in-addr.arpa domain name pointer dean.
$ host 192.168.1.236 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:

236.1.168.192.in-addr.arpa domain name pointer dean.
$

I'm surprised that 192.168.122.1 knows the reverse information for dean.

I want the query for plain "dean" to succeed. (As that's how 'ssh' works..)

If I swap the order of the lines in /etc/resolv.conf to put 192.168.1.1 first,
I get the opposite situation:

$ host dean
dean has address 192.168.1.236
$ uvt start sec-precise-amd64
Sleeping 5 seconds to give 'sec-precise-amd64' a chance to start
$ ssh sec-precise-amd64 "echo hello"
ssh: Could not resolve hostname sec-precise-amd64: No such file or directory
$ ssh sec-precise-amd64. "echo hello"
ssh: Could not resolve hostname sec-precise-amd64.: No such file or directory
$ ssh sec-precise-amd64.local "echo hello"
hello

I do not want to use the .local form for all my VMs. That is annoying and the
scripts we have written around uvt assume that the .local is not necessary.

12.04 LTS and 12.10 had this working well. I would like Raring to work that
well again.

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: dnsmasq (not installed)
ProcVersionSignature: Ubuntu 3.8.0-15.25-generic 3.8.4
Uname: Linux 3.8.0-15-generic x86_64
ApportVersion: 2.9.2-0ubuntu5
Architecture: amd64
Date: Tue Apr 2 00:04:49 2013
InstallationDate: Installed on 2012-10-18 (166 days ago)
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MarkForUpload: True
ProcEnviron:
 TERM=rxvt-unicode
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: dnsmasq
UpgradeStatus: Upgraded to raring on 2013-03-18 (15 days ago)

[Robie Basak (racb)]

Thank you for filing this bug and helping to make Ubuntu better.

I appreciate that this behaviour has changed for you. I understand what you're trying to do/have been doing.

I always saw this use of dnsmasq as a hack. Does anybody know if it is actually documented as supported behaviour upstream?

[Thomas Hood (jdthood)]

> Put a line into /etc/dhcp/dhclient.conf like so:
>
> prepend domain-name-servers 192.168.122.1;

The following doesn't address the main issue but does seem to me to be worth mentioning.

Since the introduction of resolvconf in Ubuntu 12.04 it is more advisable to configure nameserver addresses via the network configuration tool, i.e., either ifup or NetworkManager.

If you are using ifup then you add extra nameserver addresses to logical interface definitions in /etc/network/interfaces as arguments to the "dns-nameservers" option.

If you are using NetworkManager (as you seem to be doing, given that you are editing NetworkManager.conf) then add nameserver addresses in Edit Connections | <connection> | Edit... | IPv4 Settings | Additional DNS servers; if necessary, remove DHCP-acquired nameserver addresses by setting Method to "Automatic (DHCP) addresses only".

[Thomas Hood (jdthood)]

Where did you get the uvt program?

[Thomas Hood (jdthood)]

The instructions say you should comment out "dns=dnsmasq" in order to disable "the system dnsmasq". But commenting out that line (and restarting network-manager) will only disable the NetworkManager-controlled dnsmasq instance (which listens at 127.0.1.1 in Quantal and higher, at 127.0.0.1 in Precise). If you have the "dnsmasq" package installed then you have another system dnsmasq instance running which listens by default on all network interfaces. Make sure that the "dnsmasq" package is not installed if you don't want this dnsmasq instance to be running.

[Thomas Hood (jdthood)]

I guess this is the same as bug #1126488.

[Seth Arnold (seth-arnold)]

Robie, this very well might be an abuse of dnsmasq; however, the upstream author discusses a libvirt-managed dnsmasq instance as a perfectly well supported configuration in https://bugzilla.redhat.com/show_bug.cgi?id=833033#c11 .

Thomas, I don't think this is a duplicate of bug #1126488 since I made the conscious decision to _not_ upgrade to Raring until after that bug had been fixed in the main archive. :) I didn't want to deal with dnsmasq problems...

I had used the resolvconf mechanism in the past, but switched to using prepend nameservers in dhclient.conf to match the instructions on the uvt wikipage while trying to debug this. (I had used both mechanisms without problems on 12.04 LTS and 12.10.)

I do not have the dnsmasq package installed.

uvt is available in the ubuntu-qa-tools bazaar tree: http://bazaar.launchpad.net/~ubuntu-bugcontrol/ubuntu-qa-tools/master/view/head:/vm-tools/uvt It may not have the polish of the usual GUI tools but it does make deploying and maintaining a dozen qemu/kvm snapshot-capable Ubuntu installs extremely easy and convenient.

[Thomas Hood (jdthood)]

For the record, please say which version of the uvt script you are using.

In the description you say:
> This configuration worked for 12.04 LTS, 12.10, but does not work for Raring.

In comment #6 you say:
> I don't think this is a duplicate of bug #1126488 since I made the
> conscious decision to _not_ upgrade to Raring

Can you be more clear about whether you are running 12.04 or Raring or some combination of the two? In the latter case, which packages have you upgraded to Raring?

[Seth Arnold (seth-arnold)]

Thomas, I'm experiencing this problem on Raring. The uvt version (in
the source code) is:
script_version="2013021401"

I upgraded to Raring completely -- this isn't a frankenmonster :) --
after having successfully working 12.04 LTS and 12.10 configurations.
(Obviously, with earlier versions of uvt, as well; the changes to uvt have
been mainly to support secure boot and turn off some whizbang graphic
effects in the preseed to make vnc more tolerable for the times we need
to perform GUI application tests.)

Thanks again, and sorry for the confusion of my last comment.

[Robie Basak (racb)]

If you're still having the issue, the other bug is marked as Fix Released and you don't think it's a duplicate, I think it would be fair to unmark this as a duplicate rather than have the bug appear as if it is resolved.

[Thomas Hood (jdthood)]

Hi Seth,

Just read your bug report again.

It appears that you have two nameservers, neither of which has complete DNS information. You need to configure things such that each nameserver listed in resolv.conf can resolve all names you want resolved.

On a LAN, for example, there is sometimes a nameserver that resolves names in a TLD such as ".private". Such a nameserver has to be so configured that it *also* resolves Internet DNS names. Likewise in your setup.

P.S. Can you please edit your description to say which machine has which IP address? And provide more details about how you have configured your nameservers?

[Seth Arnold (seth-arnold)]

Thanks Thomas,

> You need to configure things such that each nameserver listed in resolv.conf can resolve
> all names you want resolved.

This requirement feels impossible to meet; the dnsmasq spawned by libvirt on my laptop knows only the VMs running on my laptop. It is not possible for this dnsmasq to know the hosts on my LAN -- I have a standard consumer router on my network to provide DHCP and dynamic DNS services. I do not want to require my laptop to be on and connected to my LAN for other hosts to continue using the LAN.

The DNS server on my router does not know about the VMs I host on my laptop. They are "behind" the libvirt-default NAT configuration, so they aren't even routable addresses on the LAN. Sometimes my laptop leans the LAN, and I still want to resolve my VM names.

I do not know how to configure my router to serve hostnames with a fake TLD. (I haven't used a fake TLD since about 2002, I didn't know anyone still did that..) If you really think this is a solution, I can look into it, but this all worked flawlessly with 12.04 LTS. :(

[Thomas Hood (jdthood)]

>> You need to configure things such that each nameserver listed in resolv.conf
>> can resolve all names you want resolved.

> This requirement feels impossible to meet; the dnsmasq spawned by libvirt
> on my laptop knows only the VMs running on my laptop. It is not possible
> for this dnsmasq to know the hosts on my LAN

That instance of dnsmasq, call it DV, provides services to VMs. If the VMs are going to access the LAN or the Internet then DV needs to forward DNS queries for which it isn't authoritative to a nameserver on the LAN or on the Internet.

> I have a standard consumer router on my network to provide DHCP
> and dynamic DNS services. I do not want to require my laptop to be
> on and connected to my LAN for other hosts to continue using the LAN.

Agreed. Your laptop should ultimately send DNS queries to the nameserver on the router, just as any other machine on the LAN does.

You want to be able to resolve VM names on your laptop as well as LAN and Internet names. In that case you will have to use DV as your laptop's nameserver and configure DV to forward DNS queries to the router and not use the nameserver that the laptop is using (which is DV itself — that would be a loop).

This should be implemented cleanly using resolvconf. Here is an outline of how that should be done, blindly assuming a few things such as that the "dnsmasq" package is not installed on your machine. I blindly assume, that is, that libvirt runs an instance of dnsmasq using the binary in the dnsmasq-base package.

Basically, DV should behave similarly to how the server instance of dnsmasq (from the "dnsmasq" package) behaves with respect to resolvconf.

1. DV should register with resolvconf a loopback address at which it listens.

Configure it to listen at 127.0.0.1 on the laptop and modify its control script to run

    echo "nameserver 127.0.0.1" | resolvconf -a lo.dnsmasq

on start and

    resolvconf -d lo.dnsmasq

on stop.

2. DV should have a hook script in /etc/resolvconf/update.d/ which generates a list of forwarders for DV to use. The resolvconf hook script from the dnsmasq package can be used verbatim for this purpose; simply put the file in update.d/ and configure DV to use /var/run/dnsmasq/resolv.conf as its only forwarders-list file. If you want details I can help you, or you can follow the good example of the dnsmasq package.

[Thomas Hood (jdthood)]

I think that this is ultimately an enhancement request for libvirt.

[Thomas Hood (jdthood)]

Having re-read your original description I have a couple of questions for you.

1. How did it come to pass that "nameserver 192.168.122.1" is first in your resolv.conf, as given in the description? Does libvirt add that line, or did you customize something?

2. You say that you can resolve the VM FQDNs ending in ".local". Does this go via mDNS or via dnsmasq?

[Seth Arnold (seth-arnold)]

I believe the 192.168.122.1 comes first due to the /etc/dhcp/dhclient.conf configuration: prepend domain-name-servers 192.168.122.1;

It might also have come first as a result of my previous debugging efforts. (Sorry. :)

I think the .local VM FQDN lookups are using mDNS -- at least, wireshark showed a fair amount of mDNS traffic but no DNS traffic when I just now tested.

I've made some changes to my /etc/init/lxc-net script, stolen the dnsmasq package's resolvconf hook script -- I'll test it shortly. Thanks!

[Thomas Hood (jdthood)]

> I believe the 192.168.122.1 comes first due to the /etc/dhcp/dhclient.conf configuration

Doh — yes, of course. I overlooked the following bit.

>> Put a line into /etc/dhcp/dhclient.conf like so:
>> prepend domain-name-servers 192.168.122.1;
>> Disable the system dnsmasq to prevent it from looping with libvirt's dnsmasq by modifying
>> /etc/NetworkManager/NetworkManager.conf to comment out the following line:
>> #dns=dnsmasq

Note that in connection with my comment #12 you should remove the "prepend domain-name-servers 192.168.122.1;" from dhclient.conf. The "resolvconf -a" command takes its place.

If things are set up properly then it won't be necessary to comment out "dns=dnsmasq" in NetworkManager.conf. Everything should work either with dns=dnsmasq or without dns=dnsmasq.

For clarity I will describe both cases in detail.

Note that in what follows I have replaced the string "dnsmasq" (which I used in comment #12) with "dnsmasq-libvirt", this in order to avoid conflicting with the dnsmasq package; this with an eye to inclusion of this feature in libvirt and, in so doing, avoiding conflict with the dnsmasq package. :) Because of this change you will also have to add a line to /etc/resolvconf/interface-order: just above "lo.dnsmasq" add a line "lo.dnsmasq-libvirt". Also you will have to do

    echo "nameserver 127.0.3.1" | resolvconf -a lo.dnsmasq-libvirt

and

    resolvconf -d lo.dnsmasq-libvirt

Why have I used the address 127.0.3.1 here? Because dnsmasq listens at 127.0.0.1; dnsmasq-NM listens at 127.0.1.1; and dnscrypt-proxy listens at 127.0.2.1.

Also you will have to create a directory /var/run/dnsmasq-libvirt/ and modify the hook script to write /var/run/dnsmasq-libvirt/resolv.conf instead of /var/run/dnsmasq/resolv.conf.

OK. Assume that your ISP has a nameserver NI which resolves Internet names. Assume your router's nameserver NR resolves your LAN names and forwards queries for other names to NI. Assume machines on your LAN have a resolv.conf containing "nameserver addrR" where addrR is the address of the router.

1. Consider the case where you have "dns=dnsmasq" in NetworkManager.conf on your laptop which runs NetworkManager (NM). Assume NM gets the laptop's IP address and nameserver address addrR via DHCP from the DHCP server running on the router. NM starts a local dnsmasq instance (dnsmasq-NM) and passes addrR to dnsmasq-NM. dnsmasq-NM then listens locally at 127.0.1.1 and forwards all queries to addrR. NM adds a record called "NetworkManager" to resolvconf's database containing "nameserver 127.0.1.1".

Now you start libvirt. Libvirt starts a dnsmasq instance, dnsmasq-libvirt, and adds a record called "lo.dnsmasq-libvirt" to resolvconf's database by means of the command 'echo "nameserver 127.0.3.1" | resolvconf -a lo.dnsmasq-libvirt'. When the database is updated the resolvconf update hook scripts are run. dnsmasq-libvirt has such a hook script which reads in the database records in the order determined by /etc/resolvconf/interface-order, excludes the one called 'lo.dnsmasq-libvirt' and generates /var/run/dnsmasq-libvirt/resolv.conf containing "nameserver 127.0.1.1". When the latter file changes, the dnsmasq-l...

[Thomas Hood (jdthood)]

I wrote in comment #16:
> 3. Consider now case #1 where you add dnsmasq-server (from the
> "dnsmasq" package) into the mix. Dnsmasq-server forwards to
> dnsmasq-NM — this already works. Dnsmasq-libvirt should forward
> to dnsmasq-server. If you have set things up as decribed above
> then this should happen.

On second thought I realize that there is a small problem. If dnsmasq server is installed on a system as in case #1 then dnsmasq server will forward to dnsmasq-libvirt, thus creating a loop. In order to prevent this from happening, dnsmasq-server's resolvconf hook script has to be modified to exclude the record "lo.dnsmasq-libvirt" (just as it already excludes its own record "lo.dnsmasq").

Once libvirt is enhanced as we are proposing here (bug #1163147), I will see to it that the necessary change is made to dnsmasq: I think that the change should be made via an enhancement to resolvconf that I already had in mind but I won't go into detail here about that.

[Thomas Hood (jdthood)]

Resolvconf 1.74 will include a new feature which will make it easy to adapt dnsmasq server (as requested by me in Debian bug report #716908) to work properly in the situation where it is in the middle of a chain: dnsmasq-libvirt -> dnsmasq server -> dnsmasq-NM.

[Seth Arnold (seth-arnold)]

Thomas, sorry for the slow response, I had a family issue come up.

I believe I followed the directions in comment #12 and when I rebooted, I found myself without any working DNS at all -- no Internet hosts, no LAN hosts, no VM guests.

I didn't troubleshoot too far because of your additional advice in comment #16 -- which, again, I think I followed, and also does not work. I could not resolve Internet hosts, LAN hosts. (I must admit I haven't yet tried VM guests.)

root@hunt:~# netstat -nlp | grep :53
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1966/dnsmasq
udp 0 0 0.0.0.0:5353 0.0.0.0:* 1388/avahi-daemon:
udp 0 0 192.168.122.1:53 0.0.0.0:* 1966/dnsmasq
udp6 0 0 :::5353 :::* 1388/avahi-daemon:

root@hunt:~# ps auxw | grep dnsmasq
118 1966 0.0 0.0 26080 996 ? S 18:45 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
root@hunt:~# getent passwd 118
libvirt-dnsmasq:x:118:127:Libvirt Dnsmasq,,,:/var/lib/libvirt/dnsmasq:/bin/false

I don't understand at all where this dnsmasq is being started from, because the --conf-file in the /etc/init/lxc-net is left blank. There are no other instances of 'dnsmasq' in /etc/init/* or /etc/init.d/*.

I've attached the files I recall modifying.

[Thomas Hood (jdthood)]

1. Note that my instructions in comment #12 are partly superseded by my comments in #16. I think you understood that. To make it clear for other readers I will repeat the instructions below.

2. In comment #16 I wrote

> Also you will have to do
> echo "nameserver 127.0.3.1" | resolvconf -a lo.dnsmasq-libvirt
> and
> resolvconf -d lo.dnsmasq-libvirt

without saying *when* these commands need to be run. You understood what I meant, but to make it clear for others: one has to do

    echo "nameserver 127.0.3.1" | resolvconf -a lo.dnsmasq-libvirt

after dnsmasq-libvirt has started and

    resolvconf -d lo.dnsmasq-libvirt

before stopping it.

3. Your etc-init-lxc-net.conf starts dnsmasq with incorrect options. You need to be careful with how "bind-interfaces", "interface" and "listen-address" options are used. You do the following

    --bind-interfaces --listen-address=10.0.3.1 --interface=lxcbr0 --interface=127.0.3.1

3.1. You presumably meant the following.

    --bind-interfaces --listen-address=10.0.3.1 --interface=lxcbr0 --listen-address=127.0.3.1

The argument of "--interface=" must be the name of an interface, not an IP address; the "--interface=127.0.3.1" will cause dnsmasq to fail to start. On my machine, dnsmasq 2.65-1ubuntu1 with "--interface=127.0.3.1" prints "unknown interface 127.0.3.1".

3.2. In bind-interfaces mode dnsmasq binds all and only the addresses on interfaces that it is configured to listen on. By default it listens on all interfaces that are present when it starts, but when you give an "interface" option, dnsmasq listens only that interface plus interface lo. You can add listen addresses by giving additional "interface" and "listen-address" options. So if you give "--listen-address=10.0.3.1 --interface=lxcbr0 --listen-address=127.0.3.1" then dnsmasq will listen on 10.0.3.1, 127.0.3.1, 127.0.0.1 plus the address of lxcbr0.

What you want is for it to listen on the virtual-machine-facing address(es) and on 127.0.3.1 but not on 127.0.0.1. Assuming that the virtual-machine-facing addresses are 10.0.3.1 and the address of lxcbr0, the options should be:

     --bind-interfaces --listen-address=10.0.3.1 --interface=lxcbr0 --except-interface=lo --listen-address=127.0.3.1

3.3. (We would be better off using "bind-dynamic" than "bind-interfaces" so that dnsmasq-libvirt continues to listen on the correct addresses when interfaces go up and down and have their addresses changed. However, my testing seems to indicate that you can't use "bind-dynamic" along with "listen-address". That's a bug, I think, which I will report, but until it's fixed we can't use "bind-interfaces" along with "listen-address", alas.)

4. Now create the script /etc/resolvconf/update.d/dnsmasq-libvirt (not .../dnsmasq which will conflict with the script installed by the dnsmasq package). The script you posted (etc-resolvconf-update.d-resolvconf) is good for that except that one line has to be changed. The line

    RSLVCNFFILES="$(/lib/resolvconf/list-records | sed -e '/^lo.dnsmasq$/d')"

must be

    RSLVCNFFILES="$(/lib/resolvconf/list-records | sed -e '/^lo.dnsmasq-libvirt$/d')"

to reflect the fact that we are going to use the record name "dn...

[Serge Hallyn (serge-hallyn)]

Just a note to say that
  1. IIUC there is not yet a conclusion about a workable and proper fix (frankly I'm not yet convinced that the original behavior as used by sarnold was not the sanest)
  2. I think at this point the bug should still be marked as affecting both dnsmasq and libvirt

[Thomas Hood (jdthood)]

Hi Serge,

I agree that it's still debatable what the default behavior should be. There are at least two behaviors which would be sane.

1. VMs have the same view of DNS as their host except that they can resolve names of VMs.

2. VMs have the same view of DNS as their host including being able to resolve names of VMs.

AIUI, #1 is the current default behavior. The submitter seems to claim that #2 used to be the default behavior; in any case the submitter requests that #2 be the default behavior in the future.

Behavior #2 is obviously more convenient. Implementing it requires that the host send DNS queries to dnsmasq-libvirt. That may not be what maintainers are willing to configure as the default, but it's something that I think should at least be offered as an option.

[Serge Hallyn (serge-hallyn)]

Quoting Thomas Hood (<email address hidden>):
> Hi Serge,
>
> I agree that it's still debatable what the default behavior should be.
> There are at least two behaviors which would be sane.
>
> 1. VMs have the same view of DNS as their host except that they can
> resolve names of VMs.
>
> 2. VMs have the same view of DNS as their host including being able to
> resolve names of VMs.
>
> AIUI, #1 is the current default behavior. The submitter seems to claim
> that #2 used to be the default behavior;

I don't think it used to be the default behavior, rather it was trivial
to make it the behavior by adding 192.168.122.1 as a secondary name
server.

> in any case the submitter
> requests that #2 be the default behavior in the future.

I don't think (as one of the libvirt packaging maintainers) I'd want it
to be the default behavior, but trivially configurable to do so.

FWIW I only ever ssh to vms, so I just use a .ssh/config section like

Host *.libvirt
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand nc $(host $(echo %h | sed "s/\\\\.libvirt//g") 192.168.122.1 | tail -1 | awk '{print $NF}') %p

to do it for me, but I did use to use resolv.conf like sarnold does,
years ago. (The .ssh/config solution obviously doesn't work for web, for
instance.)

> Behavior #2 is obviously more convenient. Implementing it requires that
> the host send DNS queries to dnsmasq-libvirt. That may not be what
> maintainers are willing to configure as the default, but it's something
> that I think should at least be offered as an option.

If there is a simple way to do so (that is, without a (non-negligable)
delta against upstream that would be great. Ideally it could be done
trivially for any local dnsmasqs (libvirt's, lxc's, I'm guess there are
nova and juju involved for some people...)

It seems like there should be a simple config command to tell the
host dnsmasq that *.libvirt goes to 192.168.122.1 (and then ignore
such requests from 192.168.122.1), *.lxc goes to 10.0.3.1, etc.

[Thomas Hood (jdthood)]

> I don't think it used to be the default behavior, rather it was
> trivial to make it the behavior by adding 192.168.122.1

I think you are right. I misspoke.

>> in any case the submitter requests that #2 be the default
>> behavior in the future.
>
> I don't think (as one of the libvirt packaging maintainers) I'd want
> it to be the default behavior, but trivially configurable to do so.

Agreed.

And I think I put the wrong words in the submitter's mouth. Re-reading what he originally wrote I think he wants what you just suggested: that it be *easy to activate* behavior #2 (i.e., that VMs and host are both able to resolve names of VMs and of "upstream" DNS names).

But this isn't the main issue, of course. The main issue is that the submitter couldn't obtain behavior #2 at all.

> It seems like there should be a simple config command to tell the
> host dnsmasq that *.libvirt goes to 192.168.122.1 (and then ignore
> such requests from 192.168.122.1), *.lxc goes to 10.0.3.1, etc.

Actually that is possible with dnsmasq, and what you suggest here is an alternative approach to the one I have been proposing.

What I have been proposing is that the host resolver consult dnsmasq-libvirt which forwards to upstream DNS. The VM resolvers do exactly the same. That way both the host resolver and the VM resolvers can resolve both VM names and upstream names.

The alternative approach you suggest is, IIUC, for the host resolver to consult a host dnsmasq instance ("dnsmasq server") which forwards queries regarding VM names to dnsmasq-libvirt and forwards queries regarding other names to the upstream nameserver.

The alternative approach has several drawbacks, in my opinion. First, it only works if the host runs dnsmasq (i.e., an instance distinct from dnsmasq-libvirt). Second, it absolutely requires that VM names have their own TLD(s). Third, if the user wants to resolve short VM names then the search path has to be set correctly in resolv.conf. Fourth, support for dynamic configuration (e.g., the right thing is done when dnsmasq-libvirt is stopped or started) requires either that libvirt reconfigure dnsmasq server or that resolvconf be enhanced to do this tidily (see Debian bug report #710960). Finally, it means that the host and the VMs achieve exactly the same ends by different means. This approach thus seems more complex and error-prone than the one I was proposing. But perhaps it also has advantages.

[Seth Arnold (seth-arnold)]

Thomas, thanks for the advice: I took the fix from 3.2 and 4 and the advice to rename my update.d script to /etc/resolvconf/update.d/dnsmasq-libvirt

What works:
On the host, Internet host name lookups
On the host, LAN host name lookups
In KVM guests, Internet host name lookups
In KVM guests, other KVM guest name lookups

What doesn't work:
On the host, KVM guest name lookups
In KVM guests, LAN name lookups

Now that your fix from 3.2 is in place, I can see that the situation is rather more complicated than I expected: Once I fixed the dnsmasq command line, it now starts, and I have two dnsmasq servers for virtual environments, one for LXC and one for libvirt. I modified the command line arguments for the LXC-aimed dnsmasq. The libvirt dnsmasq uses a configuration file, /var/lib/libvirt/dnsmasq/default.conf

##WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
##OVERWRITTEN AND LOST. Changes to this configuration should be made using:
## virsh net-edit default
## or other application using the libvirt API.
##
## dnsmasq conf file created by libvirt
strict-order
domain-needed
user=libvirt-dnsmasq
local=//
pid-file=/var/run/libvirt/network/default.pid
except-interface=lo
bind-dynamic
interface=virbr0
dhcp-range=192.168.122.2,192.168.122.254
dhcp-no-override
dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases
dhcp-lease-max=253
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

I haven't yet tried the simplification of getting rid of 127.0.3.1. I have a feeling that I need to further configure the libvirt-owned dnsmasq instance before getting too fancy. (For the same reason, I've also held off the managed upstart for the lxc-owned dnsmasq.)

I'm now too sleepy to see the modifications that might be needed. I thought I should document all this progress before I sleep, though.

Thanks Thomas and Serge!

[Seth Arnold (seth-arnold)]

Oh yes, I also have a feeling that I'll need a second update.d/ file -- I'm using the name dnsmasq-libvirt for the one I have now, but I think that the one I have now is actually being used by the LXC-owned dnsmasq. There's a lot of mentions of libvirt in the /etc/init/lxc-net.conf that now feel incorrect -- they're starting a dnsmasq for LXC use.

libvirt starts its own dnsmasq somewhere else, and these names should probably be applied to it instead.

[Seth Arnold (seth-arnold)]

I tried adding --server 192.168.122.1 to the lxc-net.conf dnsmasq server, in the hopes that it would forward queries for my VM guests to the libvirt dnsmasq server, but that broke everything: VM guests, LAN hosts, and Internet hosts.

[Thomas Hood (jdthood)]

All my previous advice was given without knowing that you needed a distinct dnsmasq instance for LXC.

[Thomas Hood (jdthood)]

So the situation is more complex than we thought. No problem, though: we should be able to handle any number of dnsmasq instances in daisy chain.

Do you think you understand the various components well enough to hand-craft a solution on your machine? If so then the next step is to escort the needed changes into the various packages involved, as we have discussed earlier.

If you doubt that you can get it working on your own then I'd be happy to work with you on the solution. Send me e-mail (gmail username jdthood) so we can communicate with each other directly.

[Serge Hallyn (serge-hallyn)]

So really isn't this just a result of bug 1003842?

[Serge Hallyn (serge-hallyn)]

Hi Seth,

So based on some reading for related bugs, I found the following to work!

1. I added

server=/lxc/10.0.3.1
server=/libvirt/192.168.122.1

to /etc/dnsmasq.conf

2. I added "-s lxc" to the dnsmasq command in /etc/init/lxc-net.conf

3. restarted both of the dnsmasqs

Now I was able to resolve r0.lxc correctly from 10.0.3.1!

[Thomas Hood (jdthood)]

I don't know what you mean by 'this', so I can't answer your question directly.

What I can say is that bug #1003842 implies that where you have dnsmasq-A, dnsmasq-B, dnsmasq-C which you want to operate in daisy chain, A forwarding to B and B to C, and you want to give C's listen address to A as a second forwarding address (so that A can contact C if B crashes) then you have to run A in strict-order mode.

[Seth Arnold (seth-arnold)]

Thomas, the more I've read about dnsmasq and the complaints from other
users in bug 1003842, the more I think that we might be better off taking
a different approach entirely. Please let me know what you think.

Your dnsmasq-A, dnsmasq-B, dnsmasq-C daisy-chain approach could probably
work. But guest VMs or guest LXC domains that are using dnsmasq-B couldn't
then look up hosts registered with dnsmasq-C.

This daisy chain would be brittle: if dnsmasq-B is stopped by the
administrator or otherwise dies, dnsmasq-C's configuration would need
to be updated to forward to dnsmasq-A instead.

And, libvirt-controlled guests and lxc-controlled guests would also
need networking properly configured to route between libvirt guests and
lxc guests. This might not be desirable. (In that case, you wouldn't
care that they couldn't look up hostname information from the other
virtualization technology anyway.)

So, instead of a chain, how about a flatter layout?

I haven't yet wrapped my head around the entirety of the implementation.
But I wanted to share what I'd thought of so far to get your feeling if
this is even useful to pursue to the end.

Have dnsmasq-lxc, dnsmasq-libvirt, authoritative for their own little
networks. If an lxc guest wants to resolve libvirt guests, the lxc guest
needs to have /etc/resolv.conf configured to query both dnsmasq-lxc
and dnsmasq-libvirt. If a libvirt guest wants to resolve lxc guests,
the libvirt guest needs to have /etc/resolv.conf configured to query
both dnsmasq-libvirt and dnsmasq-lxc.

If those guests also want to resolve LAN hosts or Internet hosts, they'll
need to add more "nameserver" lines. Don't abuse dnsmasq as a forwarder,
because it doesn't handle it as smoothly as it could.

The glibc resolver appears to handle multiple nameservers all configured
as 'nameserver' lines in /etc/resolv.conf even if they serve up different
data. dnsmasq does not appear to handle this case well (I think this is
what you called "NNN" in bug 1003842), and thus we should probably try to
avoid using dnsmasq in this way. If it were instead authoritative-only
for the dhcp-registered hosts, I have a feeling we could construct a
simpler answer.

We may need to increase MAXNS in /usr/include/resolv.h and rebuild nearly
everything. So I guess there are downsides. :)

If the glibc resolver doesn't actually handle multiple nameservers
with different data in them well, then probably Someone needs to write
a DNS resolver that can handle this well. Bug #1003842 would probably
also benefit from a dns resolver that forwards to all configured DNS
servers and tries to give the "best" of all available responses. (This
could eliminate the "rebuild everything" step, so it might be worthwhile
anyway.)

So, what do you think? Is there an argument to be made for a flat approach
that does not use the dnsmasq instances as forwarders? Or is the better
approach still the chaining approach, and hooking together libvirt, lxc,
and the various VPN servers in one long chain?

Thanks

[Thomas Hood (jdthood)]

On Thu, Aug 8, 2013 at 7:26 PM, Seth Arnold <email address hidden> wrote:
> Your dnsmasq-A, dnsmasq-B, dnsmasq-C daisy-chain approach could probably
> work. But guest VMs or guest LXC domains that are using dnsmasq-B couldn't
> then look up hosts registered with dnsmasq-C.

Yes, it's a characteristic of the daisy chain approach that query-upstream nameservers can consult query-downstream ones but not vice versa.

> This daisy chain would be brittle: if dnsmasq-B is stopped by the
> administrator or otherwise dies, dnsmasq-C's configuration would need
> to be updated to forward to dnsmasq-A instead.

It's the intent that this be automated using resolvconf. That takes care of the case where an instance of dnsmasq is stopped correctly. It doesn't take care of the case where a dnsmasq instance crashes or is stopped improperly. If a query-downstream dnsmasq instance crashes then upstream instances won't be able to resolve any downstream names.

> And, libvirt-controlled guests and lxc-controlled guests would also
> need networking properly configured to route between libvirt guests and
> lxc guests. This might not be desirable. (In that case, you wouldn't
> care that they couldn't look up hostname information from the other
> virtualization technology anyway.)

I don't understand you here. Can you elaborate?

> So, instead of a chain, how about a flatter layout?
> [...]
> Have dnsmasq-lxc, dnsmasq-libvirt, authoritative for their own little
> networks. If an lxc guest wants to resolve libvirt guests, the lxc guest
> needs to have /etc/resolv.conf configured to query both dnsmasq-lxc
> and dnsmasq-libvirt. If a libvirt guest wants to resolve lxc guests,
> the libvirt guest needs to have /etc/resolv.conf configured to query
> both dnsmasq-libvirt and dnsmasq-lxc.
>
> If those guests also want to resolve LAN hosts or Internet hosts, they'll
> need to add more "nameserver" lines.

The glibc resolver is not a good place to route queries to different nameservers. The glibc resolver can only consult one nameserver at a time. It queries the first nameserver in the list, waits five seconds, then queries the next nameserver in the list if it hasn't received a reply. Queries to the second nameserver are delayed by TIMEOUT every single time. Queries to the third nameserver are delayed by TIMEOUT * 2. There can be no fourth nameserver. That is not acceptable. And shortening the timeout is no solution.

> Don't abuse dnsmasq as a forwarder,
> because it doesn't handle it as smoothly as it could.

Dnsmasq works very well as a forwarder. And unlike the glibc resolver, dnsmasq can route queries by domain suffix without introducing delays, e.g., it can be configured to consult nameserver A for *.lbvrt and nameserver B for *.lxc.

> The glibc resolver appears to handle multiple nameservers all configured
> as 'nameserver' lines in /etc/resolv.conf even if they serve up different
> data. dnsmasq does not appear to handle this case well (I think this is
> what you called "NNN" in bug 1003842), and thus we should probably try to
> avoid using dnsmasq in this way.

Dnsmasq handles this case either as well as, or better than, the glibc resolver. If...

[Serge Hallyn (serge-hallyn)]

Quoting Seth Arnold (<email address hidden>):
> Thomas, the more I've read about dnsmasq and the complaints from other
> users in bug 1003842, the more I think that we might be better off taking
> a different approach entirely. Please let me know what you think.
>
> Your dnsmasq-A, dnsmasq-B, dnsmasq-C daisy-chain approach could probably
> work. But guest VMs or guest LXC domains that are using dnsmasq-B couldn't
> then look up hosts registered with dnsmasq-C.
>
> This daisy chain would be brittle: if dnsmasq-B is stopped by the
> administrator or otherwise dies, dnsmasq-C's configuration would need
> to be updated to forward to dnsmasq-A instead.
>
> And, libvirt-controlled guests and lxc-controlled guests would also
> need networking properly configured to route between libvirt guests and
> lxc guests. This might not be desirable. (In that case, you wouldn't
> care that they couldn't look up hostname information from the other
> virtualization technology anyway.)
>
> So, instead of a chain, how about a flatter layout?
>
> I haven't yet wrapped my head around the entirety of the implementation.
> But I wanted to share what I'd thought of so far to get your feeling if
> this is even useful to pursue to the end.
>
> Have dnsmasq-lxc, dnsmasq-libvirt, authoritative for their own little
> networks. If an lxc guest wants to resolve libvirt guests, the lxc guest
> needs to have /etc/resolv.conf configured to query both dnsmasq-lxc
> and dnsmasq-libvirt. If a libvirt guest wants to resolve lxc guests,
> the libvirt guest needs to have /etc/resolv.conf configured to query
> both dnsmasq-libvirt and dnsmasq-lxc.
>
> If those guests also want to resolve LAN hosts or Internet hosts, they'll
> need to add more "nameserver" lines. Don't abuse dnsmasq as a forwarder,
> because it doesn't handle it as smoothly as it could.

I guess the reason this confused me when I first read it last week is
because I *thought* this was how we expected things to work before,
and what your original request was for. But I think I see the
distinction now.

[Thomas Hood (jdthood)]

Seth, the ball is in your court, I think. Try both approaches and see how well either or both of them works.

I have favored the daisy-chain approach for the reasons given in comment #27 and because I can easily see how to support that approach using resolvconf such that it just works(tm) when packages are installed. If the routing-by-domain-name approach is preferred then we will have to figure out a j.w. solution for that.

[Serge Hallyn (serge-hallyn)]

@Seth,

any updates?

[Serge Hallyn (serge-hallyn)]

This might be something to discuss on #ubuntu-devel.

[Thomas Hood (jdthood)]

Hmm, well I don't think that this bug report is "incomplete". We know what the problem is. What is unfinished is the implementation of the solution.

[Seth Arnold (seth-arnold)]

Sorry I haven't made much progress on this myself; I just don't install lxc on the system where I use libvirt. It's not a pleasing solution but it's mostly worked.

I think part of the problem is that what I want is .. not what DNS software authors are expecting.

I want a local DNS service that knows to query multiple authoritative servers and recursive resolvers simultaneously and somehow determine which of the responses should be given back to clients. It's not a usual problem to have. (Someone else may run mixed lxc and libvirt and expect all guests to be able to resolve all other guests, but they might just run their own dns infrastructure and eschew the built-in offerings, to avoid this specific issue.)

I suspect what I want does not exist and would likely take significant engineering resources to make it work.

Thanks

[Thomas Hood (jdthood)]

@Seth: Unless you prefer to drop the matter entirely, I'd suggest you edit the bug description to say exactly what it is that you want, as you know see it. This isn't very clear in the original posting, and after re-reading many of the comments above I don't feel I have a completely clear picture of what you would like. Once you've done that, then even if we don't take any action to implement I'd say we should keep this report open at importance Wishlist.

Second, I think that with limited effort we can provide at least *part* of the desired functionality. I am happy to help with that.