during recovery mode, enable network failed due to /etc/resolv.conf not being present

[themusicgod1]

Something went wrong that required me to boot to recovery mode via grub. The important part here, is that while I got as far as the recovery screen asking to "Enable Networking" and other options fsck filesystems, drop to root shell, etc.

and selected "Enable Networking":

the result was:

grep: /etc/resolv.conf: No such File or directory.

Unknown group "power" in message bus configuration file.

(Networking did not enable, leaving me stranded at root shell without network which would have made adding/removing packages to troubleshoot easier)

Ubuntu: zesty 17.04
Linux: Linux Hedy 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:04:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
network-manager: 1.4.4-1ubuntu3

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: network-manager 1.4.4-1ubuntu3
ProcVersionSignature: Ubuntu 4.10.0-19.21-generic 4.10.8
Uname: Linux 4.10.0-19-generic x86_64
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
CurrentDesktop: XFCE
Date: Thu Apr 13 16:40:19 2017
EcryptfsInUse: Yes
IfupdownConfig:
 # interfaces(5) file used by ifup(8) and ifdown(8)
 auto lo
 iface lo inet loopback
InstallationDate: Installed on 2014-07-09 (1009 days ago)
InstallationMedia: Ubuntu-GNOME 14.10 "Utopic Unicorn" - Alpha amd64 (20140708)
IpRoute:
 default via 192.168.250.1 dev wlan1 proto static metric 600
 10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown
 169.254.0.0/16 dev lxcbr0 scope link metric 1000 linkdown
 192.168.250.0/24 dev wlan1 proto kernel scope link src 192.168.250.3 metric 600
NetworkManager.conf:
 [main]
 plugins=ifupdown,keyfile

 [ifupdown]
 managed=false
NetworkManager.state:
 [main]
 NetworkingEnabled=true
 WirelessEnabled=true
 WWANEnabled=false
SourcePackage: network-manager
UpgradeStatus: Upgraded to zesty on 2017-04-13 (0 days ago)
nmcli-dev:
 DEVICE TYPE STATE DBUS-PATH CONNECTION CON-UUID CON-PATH
 lxcbr0 bridge connected /org/freedesktop/NetworkManager/Devices/3 lxcbr0 46595dd8-757b-4d93-ade3-c066f72d9e2e /org/freedesktop/NetworkManager/ActiveConnection/0
 wlan1 wifi connected /org/freedesktop/NetworkManager/Devices/2 Brisbane House 2b25e748-f9c5-4c84-9fe6-0f64071fcf0b /org/freedesktop/NetworkManager/ActiveConnection/1
 eth1 ethernet unavailable /org/freedesktop/NetworkManager/Devices/1 -- -- --
 lo loopback unmanaged /org/freedesktop/NetworkManager/Devices/0 -- -- --
nmcli-nm:
 RUNNING VERSION STATE STARTUP CONNECTIVITY NETWORKING WIFI-HW WIFI WWAN-HW WWAN
 running 1.4.4 connected started full enabled enabled enabled enabled disabled

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[aleandro (aleandrodasilva)]

Here the same. I upgraded from 16.10 to 17.04. Network is not active. I cannot surf in internet.

[themusicgod1 (themusicgod1)]

but did you boot into recovery mode in grub? Might be a different bug if not

[aleandro (aleandrodasilva)]

I booted into recovery to see if there the network was working again.

It is a bug of Networkmanager. Probably the username 'power' doesn't affect the connection.

Solution:

sudo -H gedit /etc/NetworkManager/NetworkManager.conf

At the bottom of this file, copy and paste the following:

[device]
wifi.scan-rand-mac-address=no

Restart the networkmanager

sudo service network-manager restart

[themusicgod1 (themusicgod1)]

So editing NetworkManager.conf didn't help.
I did some digging though and found that in /etc/dbus-1/system.d in element busconfig in org.freedesktop.thermald.conf:

        <policy group="power">
                <allow send_destination="org.freedesktop.thermald"/>
                <allow receive_sender="org.freedesktop.thermald"/>
        </policy>

this is owned by thermald

thermald: 1.5.4-4

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[starkus (starkus)]

Booting to recovery mode and trying to enable networking tells me Unknown group "power" in message bus configuration file.

And networking will not be available running 17.04 ubuntu-gnome.

[Daniel Pietrucha (texar)]

To to establish connection run:

dpkg-reconfigure resolvconf

eth connection is working in my case

[Hannu E K N (hannu-n)]

I can confirm "Daniel Pietrucha (texar)"'s experience.

Boot into the recovery console and run
# dpkg-reconfigure resolvconf

...
# apt update # worked immediately afterwards.

[Domizio Demichelis (domizio)]

None of the above worked here! Without the /etc/resolv.conf nothing works obviously, then the dpkg-reconfigure resolvconf does nothing (i.e. there is still no file after running it. Adding it manually solved a part of the problem, but the Unknown group "power" still blocks starting of the network. How to fix that one?

[Woodrow Shen (woodrow-shen)]

The issue still can be reproduced by 18.04.

[Woodrow Shen (woodrow-shen)]

Giving the working note what I'm doing currently:

In order to analyze the issue, I tried to remove the /etc/resolv.conf in the normal mode and traced the syslog if any systemd service restored the file. What I observed is NM will go through this line of writing DNS, and reported the warning from calling /etc/resolvconf/update.d/libc:

<info> [1516604239.0578] dns-mgr: Writing DNS information to /sbin/resolvconf
warning "/etc/resolv.conf is not a symbolic link to /run/resolvconf/resolv.conf"

I'm thinking the /etc/resolv.conf will be dynamically recreated, and then NM moved on to finish its service (so NM is able to keep alive). However, under the recovery mode, the NM can't normally start its service and exit due to catch the signal SIGTERM (not sure if it's related to nm_dispatcher). So seems like we have two problems here:

1. why does NM fail to start
2. why does NM not try to call /etc/resolvconf/update.d/libc to restore the file in the recovery mode

[Woodrow Shen (woodrow-shen)]

Above should fix the situation as I just found out the resolvconf service seems not be triggered when recovery mode is applied...

If resolvconf.service can be triggered normally, then it would execute "/sbin/resolvconf --enable-updates". This creates a symbolic link /etc/resolv.conf for /run/resolvconf/resolv.conf. Basically the step is a implementation of "dpkg-reconfigure resolvconf".

[Woodrow Shen (woodrow-shen)]

The issue is not NM, but in the friendly-recovery, and I already prepare the patch to be reviewed.

[Woodrow Shen (woodrow-shen)]

My understanding is recovery-mode should do a task to update its resolv.conf when "Enable networking" option was chosen,
just like normal boot will trigger resolvconf service to update resolv.conf (so /sbin/resolvconf calls /etc/resolvconf/update.d/libc).
What I patch the friendly-recovery is just to add one line, see the patch I attached.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix-resolvconf-not-found.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Launchpad Janitor (janitor)]

This bug was fixed in the package friendly-recovery - 0.2.38

---------------
friendly-recovery (0.2.38) unstable; urgency=medium

  * Make friendly-recovery block the systemd emergency / rescue shell
    modes. LP: #1662137
  * Bump debhelper and starndards version, drop dh-systemd build-dep.
  * Support activating networking using systemd units for resolved,
    networkd, NetworkManager, ifupdown. LP: #1682637

 -- Dimitri John Ledkov <email address hidden> Thu, 29 Mar 2018 14:38:38 +0100

[Eric Desrochers (slashd)]

@xnox,

I have notice a block in Bionic when choosing 'Enable Network' option in recovery mode on different bionic vanilla system and I can reproduce all the time.

I also asked colleagues to give it a try (for a second pair of eye on this) and they have the same result as me.

Basically, when choosing 'Enable Network' the user got block or lock.
If we hit 'ctrl-c', then a shell arrive and the system has network connectivity.

Here's my colleague observation about this behaviour so far :

"
I also observe a lock. Plus I enabled the systemd debug-shell so I got some info:

pstree:

    systemd(1)-+-bash(391)---pstree(928)
               |-lvmetad(438)
               |-recovery-menu(711)---network(897)---systemctl(899)---systemd-tty-ask(900)
               |-systemd-journal(406)
               `-systemd-udevd(432)

I also observed that the behavior is the same with and without /e/n/i config, so it's not related to the config itself most likely.

So I also straced the systemd-tty-ask process:

    strace: Process 900 attached
    restart_syscall(<... resuming interrupted restart_syscall ...>
"

[Eric Desrochers (slashd)]

pstree command seems to shorten the processes name. So in fact 'systemd'tty-ask' is 'systemd-tty-ask-password-agent'

[Eric Desrochers (slashd)]

Here's a screenshot of what it looks like when block. [bionic-recovery-mode.png]

[Eric Desrochers (slashd)]

Here's a screenshot of what it looks like when block. [bionic-recovery-mode.png]

[Eric Desrochers (slashd)]

I did the same exercise on my reproducer and got the following pstree output:

When is switch to vtty with systemd.debug-shell=1, here what I got :

# pstree
systemd-+-bash---pstree
        |-recovery-menu---network---systemctl---systemd-tty-ask
        |-systemd-journal
        ....

# ps
root 486 473 0 08:29 tty1 00:00:00 /bin/systemd-tty-ask-password-agent

# PID 473 being systemctl start dbus.socket :
root 473 466 0 08:29 tty1 00:00:00 systemctl start dbus.socket

# PID 466 being recovery-mode/options/network
root 466 283 0 08:29 tty1 00:00:00 /bin/sh /lib/recovery-mode/options/network

[Eric Desrochers (slashd)]

Additionally,

When doing a systemd-analyze blame:
"Bootup is not yet finished. Please try again later"

systemctl list-jobs is showing a 100 jobs in 'waiting' state

Seems like systemd is not fully initialise in 'Recovery Mode'

[Eric Desrochers (slashd)]

I filed a new bug LP: #1766872 .

If someone experience the same problem, please provide some feedbacks in the new bug.

[Simon Quigley (tsimonq2)]

Unsubscribing sponsors as there seems to be nothing else to sponsor.

Artful is also EOL.

[Mathew Hodson (mhodson)]

This bug was fixed in the package friendly-recovery - 0.2.31ubuntu2

---------------
friendly-recovery (0.2.31ubuntu2) xenial; urgency=medium

  [ Dimitri John Ledkov ]
  * Actually use a friendly-recovery.target which systemd boots to
    using a generator for default.target symlink. This ensures that
    one is not in the middle of a boot transaction during recovery
    and can start/stop/change systemd units without interference.
  * Cleanup lintian warnings. (LP: #1766872)

 -- Eric Desrochers <email address hidden> Tue, 02 Oct 2018 14:43:12 -0400

[David Chmelik (dchmelik)]

I've had this problem many years. Still happens in 20.04... but it's not like there's an attempt to be like a real Unix (rather than 'real' GUI 'OS') so I guess I can't expect much (but the users do.)