No resolving after upgrade from Ubuntu-Secure

[Jason Rogers]

******* PLEASE READ BEFORE POSTING:
- this particular bug applies only if you installed Ubuntu-Secure-Remix, and upgraded from it. If you had installed a standard version of Ubuntu, please create a new bug report here: https://bugs.launchpad.net/ubuntu/+source/resolvconf/+filebug
- this bug should concern only people who installed Ubuntu-Secure-Remix ISOs dated <04/10/2012. If you installed a newer Ubuntu-Secure-Remix ISO (downloaded after 04/10/2012), and upgrade from it, you should not get this bug.
- If you reproduce it with a recent (>04/10/2012) ISO, please create a new bug report here: https://bugs.launchpad.net/ubuntu/+source/network-manager/+filebug

******* ORIGINAL BUG REPORT:
I'm assuming (based on the prefix of /var/run/nm-dns-dnsmasq.conf) that this is a network-manager issue, though I'm not certain.

I recently upgraded to Ubuntu 12.10 Beta 1 from a more-or-less stock installation of Ubuntu 12.04 (it had only been installed for about 3-4 hours before the upgrade). This means that I am now using network-manager 0.9.6.0-0ubuntu7. On startup, network-manager connects to my router without any issues, but I am then unable to resolve any domain names (beyond localhost); I can't browse to any websites, ping any servers using their domain names, etc. There were no problems doing any of this while using 12.04, and I did not do anything other than perform the upgrade to 12.10 Beta 1 before this happened, so what I was expecting to happen was that I would continue to be able to use DNS as usual.

On poking around a bit, I discovered that dnsmasq is running as it should be (listening on 127.0.1.1) but that the config file that it is trying to use, /var/run/nm-dns-dnsmasq.conf, is completely empty. This appears to be the source ofthe problem, as I can fix the issue by doing the following:

1) Kill the /usr/sbin/dnsmasq process that is automatically started (by network-manager?).
2) Manually create /var/run/nm-dns-dnsmasq.conf and add lines such as the following (this line identifies an OpenDNS server):

    server=208.67.222.222

3) Manually start the /usr/sbin/dnsmasq using exactly the same options that were passed to it when it was started automatically.

So, I'm guessing this file should be non-empty. (Another workaround, by the way, is to manually create an /etc/resolv.conf file with appropriate nameserver lines and then restart the networking service, but AFAIK that solution simply bypasses dnsmasq entirely.)

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: network-manager 0.9.6.0-0ubuntu7
ProcVersionSignature: Ubuntu 3.5.0-14.19-generic 3.5.3
Uname: Linux 3.5.0-14-generic x86_64
ApportVersion: 2.5.1-0ubuntu7
Architecture: amd64
Date: Sat Sep 15 13:02:05 2012
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
IpRoute:
 default via 192.168.1.1 dev wlan0 proto static
 169.254.0.0/16 dev wlan0 scope link metric 1000
 192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.21 metric 9
NetworkManager.state:
 [main]
 NetworkingEnabled=true
 WirelessEnabled=true
 WWANEnabled=true
 WimaxEnabled=true
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: network-manager
UpgradeStatus: Upgraded to quantal on 2012-09-15 (0 days ago)
nmcli-con:
 NAME UUID TYPE TIMESTAMP TIMESTAMP-REAL AUTOCONNECT READONLY DBUS-PATH
 Wired connection 1 500ef8b0-f9cf-48f6-9409-aabea4121a84 802-3-ethernet 1347727737 Sat 15 Sep 2012 12:48:57 PM EDT yes no /org/freedesktop/NetworkManager/Settings/1
 NotThat! 968fa0a6-be87-47bb-b306-daf2188393e9 802-11-wireless 1347728337 Sat 15 Sep 2012 12:58:57 PM EDT yes no /org/freedesktop/NetworkManager/Settings/0
nmcli-dev:
 DEVICE TYPE STATE DBUS-PATH
 eth0 802-3-ethernet unavailable /org/freedesktop/NetworkManager/Devices/1
 wlan0 802-11-wireless connected /org/freedesktop/NetworkManager/Devices/0
nmcli-nm:
 RUNNING VERSION STATE NET-ENABLED WIFI-HARDWARE WIFI WWAN-HARDWARE WWAN
 running 0.9.6.0 connected enabled enabled enabled enabled disabled

[Thomas Hood (jdthood)]

NM no longer uses nm-dns-dnsmasq.conf to tell nm-dnsmasq its forwarding addresses. NM now uses D-BUS for that purpose. So that's not the problem.

That you have no domain name service is obviously a problem.

That you could restore service by sticking nameserver addresses into nm-dns-dnsmasq.conf and restarting dnsmasq is very useful to know.

[Jason Rogers (jrog)]

I'm sorry, but it is now looking like what I posted as the first "workaround" -- adding server addressed into /var/run/nm-dns-dnsmasq.conf -- does not actually work. It turned out that I still had a manually-created /etc/resolv.conf file present on my system when I was using that "solution." Once I removed the manually-created resolv.conf, DNS stopped working again, regardless of what I did with nm-dns-dnsmasq.conf.

So, in short, it looks like the only real workaround that I've discovered right now is manually creating an /etc/resolv.conf file with lines with appropriate nameserver lines. nm-dns-dnsmasq.conf appears not to have an effect. I hope that my original suggestion otherwise didn't send anyone barking up the wrong tree too long!

[Thomas Hood (jdthood)]

The problem could be simply that the symbolic link at /etc/resolv.conf is missing. This should be a symbolic link with value "../run/resolvconf/resolv.conf". Create it by doing

    ln -nsf ../run/resolvconf/resolv.conf /etc/resolv.conf

and then rebooting. After rebooting do the following in a terminal

    ls -l /etc/resolv.conf
    cat /etc/resolv.conf
    ls -l /run/resolvconf
    ls -l /run/resolvconf/interface
    for F in /run/resolvconf/interface/* ; do echo === $F === ; cat $F ; done
    ls -l /etc/resolvconf/resolv.conf.d
    for F in /etc/resolvconf/resolv.conf.d/* ; do echo === $F === ; cat $F ; done
    cat /etc/NetworkManager/NetworkManager.conf
    ps -elfww|grep dnsmasq

and post the results here.

[Jason Rogers (jrog)]

It looks like you're on to something! I created the symlink and rebooted, and domain names are now resolving just fine. Here are the outputs of the commands that you requested:

     $ ls -l /etc/resolv.conf
     lrwxrwxrwx 1 root root 29 Sep 20 09:51 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

     $ cat /etc/resolv.conf
     # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
     # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
     nameserver 127.0.1.1

     $ ls -l /run/resolvconf
     total 4
     -rw-r--r-- 1 root root 0 Sep 20 09:53 enable-updates
     drwxr-xr-x 2 root root 60 Sep 20 09:53 interface
     -rw-r--r-- 1 root root 172 Sep 20 09:53 resolv.conf

     $ ls -l /run/resolvconf/interface
     total 4
     -rw-r--r-- 1 root root 21 Sep 20 09:53 NetworkManager

     $ for F in /run/resolvconf/interface/* ; do echo === $F === ; cat $F ; done
     === /run/resolvconf/interface/NetworkManager ===
     nameserver 127.0.1.1

     $ ls -l /etc/resolvconf/resolv.conf.d
     total 4
     -rw-r--r-- 1 root root 0 Mar 29 16:37 base
     -rw-r--r-- 1 root root 151 Sep 15 12:13 head

     $ for F in /etc/resolvconf/resolv.conf.d/* ; do echo === $F === ; cat $F ; done
     === /etc/resolvconf/resolv.conf.d/base ===
     === /etc/resolvconf/resolv.conf.d/head ===
     # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
     # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

     $ cat /etc/NetworkManager/NetworkManager.conf
     [main]
     plugins=ifupdown,keyfile
     dns=dnsmasq

     [ifupdown]
     managed=false

     $ ps -elfww|grep dnsmasq
     4 S nobody 2736 910 0 80 0 - 8268 poll_s 09:53 ? 00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
     0 S jrog 5804 3150 0 80 0 - 3396 pipe_w 09:56 pts/0 00:00:00 grep --color=auto dnsmasq

[Thomas Hood (jdthood)]

@Jason: From the log you posted it appears that resolvconf is now configured correctly.

Was /etc/resolv.conf simply non-existent on your system before your created the symlink?

Do you have any idea *why* /etc/resolv.conf was not there, or not a symlink to ../run/resolvconf/resolv.conf? Normally the resolvconf package postinst script creates this symlink.

When /etc/resolv.conf is not present, glibc falls back to the default of "nameserver 127.0.0.1". In Precise the NetworkManager-controlled dnsmasq process listens at 127.0.0.1, so if the resolv.conf file is absent and NM-dnsmasq is running then you still have name service. In Quantal the NetworkManager-controlled dnsmasq process listens at 127.0.1.1, so if the resolv.conf file is absent then you end up with no name service. As a result of this change we can expect a number of bug reports to be filed by people who had no /etc/resolv.conf on their system, still had name service under Precise, and no longer have name service after upgrade to Quantal. Get ready for this, especially in light of bug report #1000244.

[Jason Rogers (jrog)]

@Thomas: Yes, /etc/resolv.conf was simply non-existent before I created the symlink, at least in Quantal. I unfortunately don't know whether it was there in Precise and somehow ended up getting removed on upgrade or was simply never there to begin with (for the reasons that you mentioned; even if it wasn't there in Precise, DNS still would have worked, so I wouldn't have noticed). And no, I have no idea why it wasn't there. :) I didn't do anything that ought to have removed it, AFAIK!

If the absence of an /etc/resolv.conf file/symlink is widespread, then I think your warning about a number of bug reports is quite right, unless something is done to make sure that such a file/link is in place when Quantal goes official. Now, I suppose it's possible that some unique quirk happened on my machine to either remove /etc/resolv.conf or prevent it from being created in the first place, but I think it's unlikely; I literally had 12.04 on the system for about 3-4 hours at the most, and I wasn't doing any serious tinkering with it (or much of anything with it at all), so this was pretty much a vanilla install and vanilla upgrade. I had Ubuntu Tweak installed from its PPA, but I had used it to make only some minor cosmetic changes, and I also had the Faenza icon theme installed from its PPA.

[Thomas Hood (jdthood)]

Since the disappearance of your /etc/resolv.conf symlink is a mystery I have merged this report with the other reports of mysterious resolv.conf disappearance.

Exactly which image did you originally use to install Ubuntu 12.04?

Are you running a VPN client, especially a not-Ubuntu-packaged one? Or any other non-Ubuntu software?

You wrote:
> unless something is done to make sure that such a file/link is in place when Quantal goes official

That's a good idea. I think it would avoid a lot of trouble to add a check for /etc/resolv.conf-being-a-symlink somewhere in the Quantal upgrade process. Beside your case and the cases reported in #1000244 there are all the cases of the /etc/resolv.conf symlink being gone simply because the admin deleted it, because she didn't know that she's not supposed to fiddle with that file any more.

[Jason Rogers (jrog)]

No, I'm not running any VPN client, and I don't have any non-Ubuntu software installed except for Ubuntu Tweak, Faenza Icons, and YannUbuntu's Boot Repair. I don't recall when (or whether) I installed YannBuntu's Boot Repair, which makes me slightly uncertain about which image I installed from when I installed 12.04. There are definitely only two possibilities here: either (i) I installed from ubuntu-12.04.1-desktop-amd64.iso (http://releases.ubuntu.com/12.04/ubuntu-12.04.1-desktop-amd64.iso) or (ii) I installed from YannUbuntu's "Ubuntu Secure Remix" version (which includes Boot Repair), meaning from ubuntu-secure-remix-12.04.1-64bits.iso (here: http://sourceforge.net/projects/ubuntu-secured/files/). I am *almost* positive that I installed from the first one (the standard 64-bit 12.04 iso), but the fact that Boot Repair is on my system and I don't recall how it got there, together with the fact that I do have an iso for YannUbuntu's Ubuntu Secure Remix, now has me uncertain. So, sorry that I can't be definite about that. (My initial bug report, filed using apport, says this: "InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1).")

By the way, I skimmed through the duplicate bug report (#1000244) and noticed something that seems to be common to my case and all of the other cases involving Ubuntu. All of the Ubuntu cases seem to involve amd64 iso's. There are Linux Mint cases that involve 32-bit iso's, and there is an Ubuntu case that is 32-bit, but it appears to involve someone's unique issues with creating their own iso's from remastersys. The only cases in that bug's thread where someone actually had the problem after installing from an official Ubuntu source -- again, as I *think* I did, though I can't be absolutely sure given the above -- and didn't have an /etc/resolv.conf file are cases where the installer was amd64. Not sure whether that's even possibly significant or not, though.

[Jason Rogers (jrog)]

Oh, and I want to emphasize that it's possible that I missed cases in skimming through #1000244, but what I say above seemed to be true from what I could tell.

[Thomas Hood (jdthood)]

On the Ubuntu Secure Remix iso

    Name: ubuntu-secure-remix-12.04.1-64bits.iso
    Source: sourceforge.net
    Date: 2012-08-31
    Size: 735.9 MB
    MD5 sum: f331f796d721d533ee887ac590918265

the file /etc/resolv.conf is missing (whereas it is present, as it should be, on the official Ubuntu 12.04.1 images). That makes it even more plausible that you installed from the Ubuntu Secure Remix image.

[Jason Rogers (jrog)]

Thomas, thanks for checking into that. I agree with you; given both the presence of Boot Repair on my system and this otherwise seemingly-inexplicable problem, it is now looking like I must have mixed up my iso files and installed from the Ubuntu Secure Remix when I thought I was installing from the official Ubuntu image. I will report this to the Ubuntu Secure Remix project.

[Thomas Hood (jdthood)]

Thanks.

[YannUbuntu (yannubuntu)]

Thanks a lot jrog for your bug report.
I confirm there was no /etc/resolv.conf in Ubuntu-Secure-Remix iso.
I will fix it and upload new Ubuntu-Secure ISOs in the coming days.

[Sander Stuurwold (sstuurwold)]

On my machine the /etc/resolv was present. It said:

nameserver 127.0.0.1
search lokaal

I changed in

nameserver 8.8.8.8
search lokaal

And now i get connected and going to update.

[Francisco (aztlanow)]

Thanks for this post. I have used Ubuntu for 2 years, had upgraded to 12.04 and yesterday to 12.10. And for the first time I could not get online although I was connected to my router,and I tried your suggestions and It's working great. Thanks Sander Sturwold.

I'm running 12.10 on an EeePc 1005PBE, Atheros Communications Inc AR9285 Wireless Network, ath9k driver, Linux Aztlanow 3.5.0-18-generic #29-Ubuntu SMP, if that's any help to anybody.

[Thomas Hood (jdthood)]

Francisco: Are you running Ubuntu Secure Remix?

[Sebastian Geiger (lanoxx)]

Just upgraded my ubuntu 12.10 and was hit by this bug. I'm not running secure remix, just the normal Ubuntu 64

[Sebastian Geiger (lanoxx)]

sudo aptitude purge resolveconf

has solved the problem. But I'm not sure if this package is actually required for something.

[Thomas Hood (jdthood)]

Lanoxx: Resolvconf is the part of the base system that manages resolv.conf dynamically. Removing it is never the right solution. Try installing it again and then doing dpkg-reconfigure resolvconf and then rebooting. If you still have problems then we should investigate the cause. In a new bug report. This one is about a bug in Ubuntu Secure Remix.

[Leon Maurer (leon-n-maurer)]

I had a similar problem and running "dpkg-reconfigure resolvconf" seems to have fixed it. Running that asked me two yes/no questions -- I think one of them was about making the missing symlink, and I answered yes to both. (I'm running regular 12.10, so I realize this shouldn't be the place to post about that, but it's where I found the fix, so I'm posting here anyway.)

[Thomas Hood (jdthood)]

Hi Leon. Please look at bug #1000244, e.g., at my comment #66.

[Ben (bhubu)]

Comment #21 fixed the issue for me. Bug does also apply to current 12.10 release (upgrade: 16/11/2012).

[YannUbuntu (yannubuntu)]

@all:
- this particular bug applies only if you installed Ubuntu-Secure-Remix, and upgraded from it. If you had installed a standard version of Ubuntu, please create a new bug report here: https://bugs.launchpad.net/ubuntu/+source/resolvconf/+filebug
- this bug should concern only people who installed Ubuntu-Secure-Remix ISOs dated <04/10/2012. If you installed a newer Ubuntu-Secure-Remix ISO (downloaded after 04/10/2012), and upgrade from it, you should not get this bug.
- If you reproduce it with a recent (>04/10/2012) ISO, please create a new bug report here: https://bugs.launchpad.net/ubuntu/+source/network-manager/+filebug