DNS proxy on Ubuntu precise

[Miika Komu]

DNS proxy does not work in Ubuntu precise because dnsmasq has mandatory and has been integrated in rather interesting way. Note: fixing this should not affect, e.g., Debian or Fedora.

[Paul Tötterman (ptman)]

Trying to reproduce in various Ubuntu Precise environments. First off, virtual machine with minimal ubuntu install from <https://help.ubuntu.com/community/Installation/MinimalCD> . Package selection is roughly ubuntu-minimal + stuff needed to build hipl.

Cannot reproduce.

[Paul Tötterman (ptman)]

Adding ubuntu-desktop + recommended packages. Does include dnsmasq-base, but doesn't appear to be running.

Cannot reproduce.

[Miika Komu (miika-iki)]

Hi Paul,

did you try to use as a binary package (make bin)? I am using binary packages.

Also, did you try to look up e.g. crossroads.infrahip.net twice? Artturi had problems with this.

My problems appear when reconnect to the network with NetworkManager. This breaks DNS proxy.

See also:

http://sokratisg.wordpress.com/2012/03/31/ubuntu-precise-12-04-get-rid-of-nms-dnsmasq-and-setup-your-own/

[Miika Komu (miika-iki)]

Disclaimer: this could something specific to my environment, I did not try with a clean installation of Ubuntu.

[Miika Komu (miika-iki)]

My system has been upgraded several times, so it's definitely not a clean install anymore.

mkomu@bling:~$ dpkg -l resolvconf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-==============-==============-============================================
ii resolvconf 1.63ubuntu14 name server information handler
mkomu@bling:~$ dpkg -l dnsmasq
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-==============-==============-============================================
ii dnsmasq 2.59-4 Small caching DNS proxy and DHCP/TFTP server
mkomu@bling:~$ ps axu|grep dnsmasq
nobody 13946 0.0 0.0 28824 1292 ? S 09:31 0:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec
mkomu 14931 0.0 0.0 9392 924 pts/0 S+ 09:33 0:00 grep --color=auto dnsmasq
mkomu@bling:~$ ls -ld /etc/resolv.conf
lrwxrwxrwx 1 root root 28 Jul 11 09:30 /etc/resolv.conf -> /var/run/dnsmasq/resolv.conf

Note: I am not sure if the resolv.conf link is the vanilla configuration but it was the easiest way to fix totally busted DNS configuration.

[Paul Tötterman (ptman)]

I tried with a simple desktop install of Ubuntu Precise instead of a minimal install with added packages. This time, dnsmasq is running. So installing ubuntu-desktop with recommended packages is not the same as installing from the default install image. Who knew?

Tried ping and ping6 for crossroads.infrahip.net.

Anyway, could not reproduce. At least yet. It's getting late and I'm giving up for today.

[Paul Tötterman (ptman)]

It just popped to my mind that I forgot to mention the fact that I've been using the binary packages all the time. Though, I don't build them using make bin. The command I've been using is: dpkg-buildpackage -us -uc -rfakeroot

[Miika Komu (miika-iki)]

Yes, I am using desktop install as well. I don't think the install method makes any difference.

I compared a Ubuntu version that I have been running for a long time with several upgrades (DNS proxy package now removed) and an installation from scratch. With the regard to DNSmasq and resolvconf configuration files, I noticed the following:

                                                                Upgraded Scratch
/etc/default/dnsmasq yes no
/etc/resolvconf/run/resolv.conf no no
/var/run/dnsmasq/resolv.conf yes no

I tried with a new installation from scratch and it works.

[Miika Komu (miika-iki)]

Sorry, the last one was:
/var/run/dnsmasq/resolv.conf yes yes

So the difference is a left over /etc/default/dnsmasq. When I removed it manually, everything started to work alright. I'll report again if I experience something strange.

Paul, can you revisit the logic of the file checking before we close this bug? Remember that it affects only Ubuntu Precise on onwards and not e.g. Debian. Thanks for you patience.

[Miika Komu (miika-iki)]

Actually, DNS requests started failing when I rebooted the Ubuntu installed from scratch. So, the problem persists and I think it's about the new placement or missing configuration files due to different start up DNS masq in Precise. So, please try install (all) packages and reboot (which effectively kills dnsmasq and something weird happens).

I had to remove DNS proxy and to recover the following file (and restart services or reboot the VM) to make DNS work at all:

ln -s /var/run/dnsmasq/resolv.conf /etc/resolv.conf

[Paul Tötterman (ptman)]

Ok, so sometimes the dns proxy seems to get confused if dnsmasq hasn't started yet and there's nothing in the resolv.conf file. Some times the system even ends up with a resolv.conf without anything but comments. I guess the proper way to fix this would be to register the dns proxy with resolvconf or try to order stuff so that there is no race.

[Miika Komu (miika-iki)]

Yes, preferably in a distro-independent way as it is now.

[Paul Tötterman (ptman)]

There is a version of hipdnsproxy on the resolvconf branch now that uses resolvconf(8) to update /etc/resolv.conf if it is present. This seems to solve the problem on Debian wheezy (with resolvconf, without dnsmasq) and Ubuntu precise (with resolvconf and network-manager, which implies dnsmasq in some capacity).

More work is required to handle all corner cases so that this can be merged to trunk.

[Miika Komu (miika-iki)]

Please list the corner cases?

[Miika Komu (miika-iki)]

I built the binary and installed it, and then I get the attached crash report in normal Ubuntu LTS Desktop. I tried it also from command line and then I get the following error:

mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ sudo -s ./hipdnsproxy
Traceback (most recent call last):
  File "./hipdnsproxy", line 102, in <module>
    main()
  File "./hipdnsproxy", line 98, in main
    dnsp.mainloop(args)
  File "/usr/lib/python2.7/dist-packages/hipdnsproxy/dnsproxy.py", line 488, in mainloop
    self.dnsmasq.setup_hook()
  File "/usr/lib/python2.7/dist-packages/hipdnsproxy/dnsmasq.py", line 137, in setup_hook
    self.backup()
  File "/usr/lib/python2.7/dist-packages/hipdnsproxy/dnsmasq.py", line 92, in backup
    os.link(Dnsmasq.defaults_path(), self.backup_name())
OSError: [Errno 2] No such file or directory
mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ ps axu|grep dnsmasq
dnsmasq 12211 0.0 0.0 28832 720 ? S Oct22 0:03 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf
mkomu 17093 0.0 0.0 9400 924 pts/4 S+ 15:08 0:00 grep --color=auto dnsmasq
mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ ps axu|grep ^C
mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ aptitude search resolvconf
i resolvconf - name server information handler
mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ ls -ld /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Aug 20 18:42 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
mkomu@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search cs.hut.fi

I could debug this also myself but I am joggling with too many tasks at the moment.

[Paul Tötterman (ptman)]

So hipdnsproxy expects you to have /etc/default/dnsmasq since you
a) have /usr/sbin/dnsmasq (from dnsmasq-base)
b) have /etc/init.d/dnsmasq (from dnsmasq)
and
c) are on a debian family distro

I've added os.path.exists checks now, but can you explain how come you don't have /etc/default/dnsmasq? What is wrong in the assumption?

[Paul Tötterman (ptman)]

Fixed the recovery code, so the -k switch (used by the init script) works again. Which distributions and releases should we try to support? Should I test with dnsmasq in any other capacity than spawned by network-manager?

[Miika Komu (miika-iki)]

bzr co lp:~ptman/hipl/resolvconf and compilation gives me:

root@bling:~/projects/hipl-bzr/resolvconf/tools/hipdnsproxy# ./hipdnsproxy
/etc/init.d/dnsmasq: 1: /etc/default/dnsmasq: Syntax error: Unterminated quoted string
Traceback (most recent call last):
  File "./hipdnsproxy", line 104, in <module>
    main()
  File "./hipdnsproxy", line 100, in main
    dnsp.mainloop(args)
  File "/home/mkomu/projects/hipl-bzr/resolvconf/tools/hipdnsproxy/dnsproxy.py", line 510, in mainloop
    self.dnsmasq.setup_hook()
  File "/home/mkomu/projects/hipl-bzr/resolvconf/tools/hipdnsproxy/dnsmasq.py", line 145, in setup_hook
    self.restart()
  File "/home/mkomu/projects/hipl-bzr/resolvconf/tools/hipdnsproxy/dnsmasq.py", line 103, in restart
    subprocess.check_call([Dnsmasq.INIT, 'restart'])
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)

bzr log -v tells me
revno: 6399 [merge]
committer: Paul Tötterman <email address hidden>
branch nick: resolvconf
timestamp: Sun 2012-10-28 18:05:11 +0200
message:
  Merged trunk
modified:
  .commitguards/copyright-guard-space.py
  doc/HOWTO.xml.in

Binary compilation installs properly but does not seem to hook correctly. Looking up crossroads.infrahip.net just returns normal IP addresses.

I am running 64-bit Ubuntu LTS in my laptop (that has been upgraded several times, not a "clean" install). Besides this, we need Debian and Fedora supported with the DNS proxy.

[Miika Komu (miika-iki)]

> I've added os.path.exists checks now, but can you explain how come you don't have /etc/default/dnsmasq? What is wrong in the assumption?

Besides my laptop, this does not appear in my PC (Ubuntu LTS) either.

[Paul Tötterman (ptman)]

I noticed the following on Ubuntu Precise:

hipdnsproxy gets started before NetworkManager manages to lift up the network connection using DHCP, so there is no usable nameserver in /etc/resolv.conf. This means that hipdnsproxy needs to install a script into /etc/resolvconf/update[-libc].d/ to get notified of changes to resolv.conf. And that script must take care not to end up in infinite recursion.

Also resolvconf normally stops processing the inputs once it hits the first nameserver entry that starts with 127., so if that is the line for hipdnsproxy, then we need to figure out the upstream nameserver in some other way. resolvconf cannot easily be told to dump the result anywhere else, so I think we should figure that out from resolvconf's inputs without involving resolvconf.

resolvconf works fine if you are either a provider of nameserver information, or a consumer, but being both is a bit tricky.

[Miika Komu (miika-iki)]

Thanks for the clarifications. This gets really complicated if you consider that we should be able to run DNS proxy manually from the command line and automatically from /etc/init.d/ in a binary installation. Rather than trying to live along with DNS masq, I would suggest living without it to make things less complicated:

1. Remove all dnsmasq integration from the python code. If HIP dns proxy cannot bind to port, exit with some warnings:

"Port xx is occupied. Please remove dnsmasq, or any other software, if you're running it or change the default invocation parameter --xx in /etc/default/hipdnsproxy."

2. Modify Debian and Fedora packaging to mark "dnsmasq" as a conflicting package.
* debian/control (hipl-dnsproxy package): Conflicts: dnsmasq (see http://www.debian.org/doc/debian-policy/ch-relationships.html#s-conflicts)
* trunk/packaging/hipl.spec (dnsproxy package): Conflicts dnsmasq (see http://www.rpm.org/max-rpm/s1-rpm-depend-manual-dependencies.html)

3. Test installation at least with Ubuntu with dnsmasq installed.

Does this work for you?

[Miika Komu (miika-iki)]

Ubuntu has two packages, dnsmasq and dnsmasq-base. Only the former can be removed, and removal of the latter will break the system pretty badly. Network manager starts dnsmasq from the dnsmasq-base package, and thus the previous suggestion does not entirely work (removal of "dnsmasq" disabled it only temporarily because it re-appeared on next login).

To make this work, I would suggest to grep also the /etc/NetworkManager/NetworkManager.conf for traces of dnsmasq, and give bail out with a big warning if it exists:

  Please comment out dnsmasq line in /etc/NetworkManager/NetworkManager.conf
  and run 'sudo restart network-manager'. You may have to logout and login to make the
  changes effective.

By following the instructions in the following article, I noticed that logout/login procedure was necessary to get network manager to actually to work:

http://www.ubuntugeek.com/how-to-disable-dnsmasq-in-ubuntu-12-04precise.html

So, you can automatize the configuration procedure if you want to, but the user has to logout from the Desktop to make the changes effective.

I hope we can resolve this soon as possible, and maybe you could send a new merge proposal directly when think it's working? Thanks!

[Paul Tötterman (ptman)]

Here's a branch with dnsmasq support ripped out but still resolvconf(8) support added. Tested on precise.

[Miika Komu (miika-iki)]

Great, it seems to be working, thanks. Could you still detect the installation of dnsproxy and give a more descriptive warning? This could be done either in DNS proxy itself or in debian/hipl-dnsproxy.postinst. Now DNS proxy just gives a mysterious error:

root@bling:~/projects/hipl-bzr/nodnsmasq/tools/hipdnsproxy# ./hipdnsproxy
2013-01-15 23:36:15,391 ERROR Port 53 occupied

Please consider documenting the issue and giving a pointer to the URL (see above) in HOWTO.xml.in.

I think the branch is read for a merge proposal after this. Thanks!

[Miika Komu (miika-iki)]

Btw, if "dnsmasq" (not "dnsmasq-base") package has been installed and then removed, it still has to be purged with apt-get or aptitude. Otherwise, it will be restarted via /etc/rc2.d/ links again in boot. After purging, I don't get any weird issues with dnsmasq anymore and login/logout procedure is not required, though I had to "killall dnsmasq" since init.d script was gone . So I think this could be also automatized in the preinst/postinst scripts. You can leave the automation to me if you're getting bored with this :)

[Paul Tötterman (ptman)]

"Neither Breaks nor Conflicts should be used unless two packages cannot be installed at the same time or installing them both causes one of them to be broken or unusable. Having similar functionality or performing the same tasks as another package is not sufficient reason to declare Breaks or Conflicts with that package." <http://www.debian.org/doc/debian-policy/ch-relationships.html#s-conflicts>

So we shouldn't use that to communicate the incompatibility of dns proxy and dnsmaq.

"Keep usage notes to what they belong: the NEWS.Debian, or README.Debian file. Only use notes for important notes which may directly affect the package usability. Remember that notes will always block the install until confirmed or bother the user by email." <http://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#s6.5.1>

This is what I've done anyway. But I'm against touching something that belongs to another package. That should be left up to the user.

[Miika Komu (miika-iki)]

I have tested the branch and still has two problems that need be fixed before merge proposal:

1. The condition should be just about "dnsmasq" and not about "dnsmasq-base". As indicated earlier in the discussion, the removal of dnsmasq-base is problematic because it removes quite a many packages.
2. I tried changing the condition from 'dnsmasq*' to 'dnsmasq', but "dpkg -i hipl-dnsproxy*deb" hangs and never completes.

I have tried these on a cleanly installed Ubuntu 64-bit system (latest LTS). I have some strange problems on my Ubuntu machine (which has been upgraded several time), but suggest to ignore them.

[Miika Komu (miika-iki)]

I forgot to say that the installation also hangs if I have 'dnsmasq' installed. A warning prompt appears, but hangs after I press enter.

[Miika Komu (miika-iki)]

Installation of the DNS proxy on the trunk works, so the problem really is the new branch.

[Paul Tötterman (ptman)]

The difference in the environments points to DEBCONF_REDIR as in <http://us.generation-nt.com/answer/bug-439763-debconf-hangs-puppet-installs-preseed-install-help-167594241.html> and closing fd3 works. But I'll dig deeper.

[Paul Tötterman (ptman)]

No longer hangs under dpkg. I resorted to special casing when DEBCONF_REDIR is present in the environment. I tried forking into the background as early as possible, but there are still several file descriptors open so I thought it unwise to just close everything.

As to why I'm _showing_ the warning in case dnsmasq-base is installed, it's because that's where the binary comes from so it can be run, wether it be from an init script, via network-manager or by the user himself.

Feel free to reword the documentation and warning as you like.

Also merged in trunk.

[Miika Komu (miika-iki)]

Test report:

* CASE1: I tested the branch binaries in another system where network-manager starts dnsmasq. It works as expected; installation prompts warning and dnsmasq does nothing. Success!
* CASE2: I have tested the branch binaries with in Ubuntu where I have uninstalled dnsmasq (but not dnsmasq-base), and disallowed network-manager to start it. It seems to work good. Success!

Suggestion for improvement:

* In the second case, the administrator is prompted unnecessarily. I wonder if we could test "pidof dnsmasq" instead of "dpkg -l" debian/hipl-dnsproxy.config? Please note that "service dnsmasq" does not work when network-manager starts dnsmasq.

Suggestion on how to proceed with the bug report:

* I suggest issuing a merge proposal after this small fix. Please give one week for reviews and merge if no further comments.

Good work!

[Paul Tötterman (ptman)]

Done, review is pending.

Additionally I shifted the resposibility of daemonizing to start-stop-daemon. This gets around the problem with DEBCONF_REDIR in a cleaner way, as start-stop-daemon can just close all filehandles before before execing hipdnsproxy. When forking in python code I never could track all the open filehandles and be certain that I didn't step on some other code's toes. So I had to special case for DEBCONF_REDIR.

I can just as easily shift the job back to hipdnsproxy, I just wanted to properly explore the options.