Name resolving not working after 12.04 LTS -> 14.04 LTS if /etc/resolv.conf was absent

[Timo Jyrinki]

[Impact]

Name resolving using the NetworkManager-controlled nameserver discontinues working after upgrading from 12.04 (precise) to 14.04 (trusty) in the case where /etc/resolv.conf is absent.

Precise works fine with that file absent because the resolver defaults to using "nameserver 127.0.0.1" — so it is possible to run Precise for a long time without even noticing that the file is absent.

In Trusty, resolving does not work if /etc/resolv.conf is absent because the NetworkManager-controlled nameserver listens at 127.0.1.1.

People suffering from bug #1000244 may lack /etc/resolv.conf on upgrade.

[Test Case]

1. Install 12.04 LTS
2. rm /etc/resolv.conf and observe everything works fine
3. update-manager -c -d
4. Observe that in 14.04 LTS name resolving does not work.

[Test Case for testing the fix on 14.04]

1. wget http://de.archive.ubuntu.com/ubuntu/pool/main/r/resolvconf/resolvconf_1.63ubuntu16_all.deb
2. dpkg -i resolvconf_1.63ubuntu16_all.deb
3. Running that precise version, test the possible cases like rm /etc/resolv.conf (note you lose name resolving at this point in 14.04)
4. Run 'bzr bd' (apt-get install bzr-builddeb first) in the proposed branch, dpkg -i ../build-area/*.deb
5. Check how either, depending on the case being tested, /etc/resolv.conf is intact / non-touched or it's created when not-existing, restoring network connectivity

[Regression Potential]

Seems low considering the limiting of the proposed fix to 12.04 -> 14.04 upgrades only, and only in the case of the user/3rd party app/something intentionally removed /etc/resolv.conf (instead of just modifying it).

---

I fixed it manually with:
cd /etc
sudo ln -s /run/resolvconf/resolv.conf .

It turns out 12.04 LTS works fine without /etc/resolv.conf, so that increases the likelihoods that 12.04 LTS systems do not have that file. When such a system is upgraded to 14.04 LTS, network seemingly stops working to the user.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1308378

[Thomas Hood (jdthood)]

Thanks for the report.

Either a third-party package or the administrator futzed with /etc/resolv.conf independently of the resolvconf package.

[Timo Jyrinki (timo-jyrinki)]

Right, thanks for the quick confirmation.

[Timo Jyrinki (timo-jyrinki)]

Adjusted the description, I think there's a potential problem than can handled better from user perspective.

The proposed branch adds upgrade) preinst rule for resolvconf < 1.64, and creates the symlink if and only if /etc/resolv.conf does not exist at all (normal file or symlink). I've tested that it does nothing when upgrading from newer resolvconf, and also leaves /etc/resolv.conf untouched if it exists as a normal file or a symlink, when upgrading from the precise version. It fixes the issue I saw on upgrade.

[Thomas Hood (jdthood)]

What happens if the admin does `rm /etc/resolv.conf`? The package should not undo this on a subsequent upgrade.

[Timo Jyrinki (timo-jyrinki)]

If an admin does rm /etc/resolv.conf on the 14.04, name resolving stops working and no upgrade will undo that change, since the preinst check is only there when upgrading from the 12.04's version of the package. This can also be tested by installing the new package multiple times, removing resolv.conf in between - it does not get back in that case, because the upgrade) is called with the 1.69 version which is higher than what's checked.

If admin does rm /etc/resolv.conf in 12.04, name resolving continues to work because the name resolving does not stop working at that like it does on 14.04 (possibly because /run/resolvconf/resolv.conf is always watched in 12.04 instead or some other detail that has changed?). If the system is then upgraded to 14.04, either: a) without this fix name resolving will simply stop working, because of the difference in behavior in 14.04, b) with this fix the symlink gets reinstated and name resolving continues to work after the upgrade.

[Thomas Hood (jdthood)]

> If an admin does rm /etc/resolv.conf on the 14.04, name resolving stops working

Actually it doesn't. In the absence of /etc/resolv.conf the resolver simply defaults to the configuration equivalent to "nameserver 127.0.0.1".

The case I have in mind is:

* Install 12.04
* Do `rm /etc/resolv.conf`
* Upgrade resolvconf to 1.63ubuntuNewWithYourChange

After this will there be something at /etc/resolv.conf? If so then this is contrary to the Debian/Ubuntu principle that packages don't undo changes made by admins.

[Timo Jyrinki (timo-jyrinki)]

> Actually it doesn't. In the absence of /etc/resolv.conf the resolver simply defaults to
> the configuration equivalent to "nameserver 127.0.0.1".

Weirdly, I've multiple 14.04 machines (including freshly installed) here that lose name resolving abilities after removing /etc/resolv.conf. Only Firefox continues to work until restart, but eg. ping www.google.com immediately reports unknown host. If I move the symlink back, resolving works.

> Install 12.04
> Do `rm /etc/resolv.conf`
> Upgrade resolvconf to 1.63ubuntuNewWithYourChange

This wouldn't need any changes in 12.04, and the version would be 1.69NewWithMyChange in 14.04. So 12.04 users would continue to use as normal, but this fix would prevent network getting seemingly broken for no obvious reason upon upgrade to 14.04, as a one-time upgrade change of restoring the symlink that by default is there in both 12.04 and 14.04. But only 14.04 seems to rely on it existing, thus 12.04 users are unaware they'd hit this problem when upgrading.

[Thomas Hood (jdthood)]

> Weirdly, I've multiple 14.04 machines (including freshly installed) here
> that lose name resolving abilities after removing /etc/resolv.conf.

Ah, a misunderstanding. What I meant is that the resolver still functions: it tries to reach a nameserver at 127.0.0.1. In your case there is no nameserver listening at 127.0.0.1 so resolving doesn't succeed.

[Thomas Hood (jdthood)]

OK, I think I know what you are talking about now. (When you wrote "The proposed branch adds upgrade) preinst rule for resolvconf < 1.64" I thought you meant that you were going to update resolvconf 1.63 in Precise via precise-updates. This threw me off.)

In 12.04 the NetworkManager-controlled nameserver listened locally at 127.0.0.1 so resolving worked even if /etc/resolv.conf was absent.

In 12.10 and later the NetworkManager-controlled nameserver listens at 127.0.1.1. If /etc/resolv.conf is absent then resolving fails.

If you are running 12.04 using the NetworkManager-controlled nameserver and /etc/resolv.conf is absent and you upgrade to 14.04 then resolving no longer works and you are sad.

[Timo Jyrinki (timo-jyrinki)]

Thanks for describing the exact problem instead of my more vague "symptoms" approach. Indeed updated trusty package that only takes effect during 12.04 LTS -> 14.04 LTS upgrade is in question.

[Steve Langasek (vorlon)]

I'm not happy with the proposed change to make resolvconf recreate the missing symlink on upgrade from precise to trusty without understanding why the symlink was absent in the first place. Timo, why was this missing for you? How do we know that whatever caused this problem for you, won't happen again after upgrade to trusty?

[Timo Jyrinki (timo-jyrinki)]

I think I know the actual reason (or one of those) it may be missing for many people including me in 12.04 LTS: https://help.ubuntu.com/community/LiveCDCustomization <- as an example, this in a modified form (just installing additional packages and resolv.conf as the only modified/removed file) was started to be used for Finnish Remix CD:s around 2007 and probably many other remixes have used the same wiki page as guidance. I'm not sure about the various more automated remix tools around. The instructions use copying host's resolv.conf to etc so that network works inside chroot, and then it "cleans up" by removing the copied /etc/resolv.conf. This was maybe ok when /etc/resolv.conf was not a symlink and Network Manager modified it directly, like in 10.04 LTS.

This happened to still work during 12.04 LTS, even though by default resolv.conf was already a symlink and not a real file in 12.04 LTS, and Network Manager didn't modify it anymore. When these users upgrade to 14.04 LTS, they seemingly have a broken network with no Network Manager configuration helping the situation because of this changed Network Manager fallback 127.0.0.1/127.0.1.1.

If the wiki instructions are tried in remastering 14.04 LTS image, it becomes obvious something is broken since network doesn't work on the remastered image.

[Steve Langasek (vorlon)]

On Thu, Apr 24, 2014 at 07:49:17AM -0000, Timo Jyrinki wrote:
> I think I know the actual reason (or one of those) it may be missing for
> many people including me in 12.04 LTS:
> https://help.ubuntu.com/community/LiveCDCustomization <- as an example,
> this in a modified form (just installing additional packages and
> resolv.conf as the only modified/removed file) was started to be used
> for Finnish Remix CD:s around 2007 and probably many other remixes have
> used the same wiki page as guidance. I'm not sure about the various more
> automated remix tools around. The instructions use copying host's
> resolv.conf to etc so that network works inside chroot, and then it
> "cleans up" by removing the copied /etc/resolv.conf. This was maybe ok
> when /etc/resolv.conf was not a symlink and Network Manager modified it
> directly, like in 10.04 LTS.

Ok. Would be great if someone could fix up such broken documentation; but
if you say this was because of a non-standard install method I'm satisfied
that there's no reason to expect a recurrence of the problem on upgrade.

[Timo Jyrinki (timo-jyrinki)]

In this case at least there shouldn't be a recurrence.

I fixed the wiki page by putting warnings to all places that discussed resolv.conf mangling, and also recommended that the hosts file wouldn't be touched either.

[Michael Vogt (mvo)]

I uploaded the SRU to trusty-proposed just now. A utopic version is not needed as there is no direct 12.04->14.10 upgrade path.

[Chris J Arges (arges)]

Hello Timo, or anyone else affected,

Accepted resolvconf into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/resolvconf/1.69ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Timo Jyrinki (timo-jyrinki)]

I tested this again with the -proposed's .deb, and it continues to work as advertised.

[Launchpad Janitor (janitor)]

This bug was fixed in the package resolvconf - 1.69ubuntu1.1

---------------
resolvconf (1.69ubuntu1.1) trusty-proposed; urgency=medium

  * debian/preinst: On 12.04 LTS the system works without /etc/resolv.conf.
    Upon upgrading such a system to 14.04 LTS network stops working.
    Therefore, create a symlink upon upgrade from resolvconf <= 1.64 in the
    case that /etc/resolv.conf does not exist at all. (LP: #1308378)
 -- Timo Jyrinki <email address hidden> Thu, 17 Apr 2014 14:45:17 +0300

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for resolvconf has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[hamish (hamish-b)]

Hi,

unfortunately the updated package just makes things worse for us. (14.04 Lubuntu)

as soon a it is installed in our live iso chroot build process as part of 'apt-get upgrade' the original chroot session's resolv.conf is gone. Before it was only when the live user session started up at run-time that resolv.conf was gone.

our work-around for the original run-time version of missing resolv.conf was to add an autostart .desktop launcher to run "sudo dhclient eth0", but that makes a rather large assumption that eth0 is what you're connecting through.

Hamish
   //
OSGeo Live GIS
http://live.osgeo.org

[hamish (hamish-b)]

this really wants me to open a new bug report rather than reopen?

bah, barriers to entry :-(

[hamish (hamish-b)]

yeah, we were following those live cd wiki instructions too and removing resolv.conf before mastering. I just put the symlink back, we'll see how the next nightly build goes.

but that doesn't explain why the latest 'apt-get upgrade' to the newly released resolvconf package spontaneously nukes DNS lookups. I've added some debug messages to our build to try and figure out what's happening there; will report back when I know more.

thanks for all the detective work,
Hamish

[Timo Jyrinki (timo-jyrinki)]

Hey hamish. The addition in this bug fix only triggers when upgrading resolvconf from the 12.04 LTS version (1.63), so it doesn't get used in your case. Maybe your problem is simply because resolvconf gets upgraded in the first place in a stable release, and your live cd creation/upgrade process doesn't like that?

For what it's worth, I always create a new live cd based off normal Ubuntu release for point releases, see https://code.launchpad.net/~timo-jyrinki/ubuntu-fi-remix/main - and I only temporarily, manually, edit /run/resolvconf/resolv.conf during the preparation nowadays. I don't know if that helps, but maybe the idea of doing your resolv.conf changes inside /run/ would help you too.

[Timo Jyrinki (timo-jyrinki)]

Just in case some people find this bug and still see a similar problem:

It seems like if the original installation is 10.04 LTS, it's possible that 12.04 LTS -> 14.04 LTS will yield similarly non-working name resolving. This is because in 10.04 LTS /etc/resolv.conf was a file, and it was not modified into a symlink during 10.04 LTS -> 12.04 LTS upgrade. While this did not hurt during usage of 12.04 LTS, it starts to break name resolving when the 12.04 LTS is further upgraded to 14.04 LTS.

[Tristan Schmelcher (tschmelcher)]

I had a similar issue but not quite the same. I upgraded from Precise (installed at the factory on a Dell Sputnik device) to Trusty. After upgrade, DNS did not work. I did have the /etc/resolv.conf symlink, but the content generated by resolvconf just listed 127.0.0.1 twice (the old address). Following the suggestion at http://askubuntu.com/a/462377/75121, I purged and reinstalled resolvconf--which didn't immediately fix it--and then rebooted. After reboot, the resolv.conf now lists the correct address of 127.0.1.1 followed by the old address of 127.0.0.1, and DNS resolution works.