netplan using incorrect renderer

[Yannik]

After upgrading ubuntu desktop from 19.10 to 20.04, it's impossible to enable screen sharing.
It used to work fine on 19.10.

When clicking the enable/disable toggle, the following message is logged in journalctl:

Failed to enable service vino-server: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Sharing cannot be enabled on this network, status is '0'

This is reproducible on both computers I've upgraded. They're both on the latest 20.04 package updates as of today.

I've done some research to rule out common faults:

$ cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

(Yes, I've ofc ran netplan apply, but the file already had this content anyway.)

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

Also, this seems unrelated to #1871787:
$ dpkg -l |grep gnome-settings-daemon
ii gnome-settings-daemon 3.36.0-1ubuntu2
ii gnome-settings-daemon-common 3.36.0-1ubuntu2
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2020-01-17 (108 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
Package: gnome-control-center 1:3.36.1-1ubuntu5
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 5.4.0-28.32-generic 5.4.30
Tags: focal
Uname: Linux 5.4.0-28-generic x86_64
UpgradeStatus: Upgraded to focal on 2020-04-28 (6 days ago)
UserGroups:

_MarkForUpload: True

[Yannik (yannik-5)]

I've also tried setting

[ifupdown]
managed=true

in /etc/NetworkManager/NetworkManager.conf

and

`touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf`.

Even this did not fix the problem. (I ran 'netplan generate', 'netplan apply' and rebooted after doing these changes)

[Daniel van Vugt (vanvugt)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1876884

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Yannik (yannik-5)]

apport information

[Yannik (yannik-5)]

apport information

[Yannik (yannik-5)]

apport information

[Sebastien Bacher (seb128)]

Thank you for your bug report. To what wifi are you connected? The warning indicates that n-m belives that the wifi can't be trusted and is not safe for enabling sharing, which shouldn't be the case on a private access point

[Yannik (yannik-5)]

No wifi, the computer is using a wired connection.

[Yannik (yannik-5)]

No wifi, the computer is using a wired connection.

[Sebastien Bacher (seb128)]

Could you add the output from those commands
$ journalctl -f
click the toggle and copy what was printed in the log

$ nmcli d status

do you get any error if you start manually?
$ /usr/lib/vino/vino-server

also what interface is the settings panel listing at the bottom is that your correct eth one? the return code seems to suggest n-m thinks you are offlin

[Yannik (yannik-5)]

$ journalctl -f
May 05 14:32:30 work-pc1 gnome-control-c[2052]: Failed to enable service vino-server: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Sharing cannot be enabled on this network, status is '0'

$ nmcli d status
DEVICE TYPE STATE CONNECTION
enp2s0 ethernet unmanaged --
lo loopback unmanaged --

$ /usr/lib/vino/vino-server

05/05/2020 15:10:09 Autoprobing TCP port in (all) network interface
05/05/2020 15:10:09 Listening IPv6://[::]:5900
05/05/2020 15:10:09 Listening IPv4://0.0.0.0:5900
05/05/2020 15:10:09 Autoprobing selected port 5900
05/05/2020 15:10:09 Advertising security type: 'TLS' (18)
05/05/2020 15:10:09 Re-binding socket to listen for VNC connections on TCP port 5900 in (all) interface
05/05/2020 15:10:09 Listening IPv6://[::]:5900
05/05/2020 15:10:09 Listening IPv4://0.0.0.0:5900
05/05/2020 15:10:09 Clearing securityTypes
05/05/2020 15:10:09 Advertising security type: 'TLS' (18)
05/05/2020 15:10:09 Clearing securityTypes
05/05/2020 15:10:09 Advertising security type: 'TLS' (18)
05/05/2020 15:10:09 Advertising authentication type: 'No Authentication' (1)
05/05/2020 15:10:09 Re-binding socket to listen for VNC connections on TCP port 5900 in (all) interface
05/05/2020 15:10:09 Listening IPv6://[::]:5900
05/05/2020 15:10:09 Listening IPv4://0.0.0.0:5900

also what interface is the settings panel listing at the bottom is that your correct eth one?
The "network" part of the settings panel reads: "Es sind keine Netzwerke zur Freigabe gewählt." (german locale) which can be translated to "No network has been selected for sharing".

[Sebastien Bacher (seb128)]

Ok, so the issue is

$ nmcli d status
DEVICE TYPE STATE CONNECTION
enp2s0 ethernet unmanaged --
lo loopback unmanaged --

It looks like network-manager isn't handling the network, I'm going to reassigning to netplan since you are using it to apply the configuration

Could you also add the 'journalctl -b 0' from a session having the issue? it might help giving some clue why network-manager is bailing out of handling the connection

[Yannik (yannik-5)]

I did some additional debugging on this and found the file `/run/NetworkManager/conf.d/netplan.conf` with contents:
[keyfile]
# devices managed by networkd
unmanaged-devices+=interface-name:enp1s0

Removing that file leads to the device being listed as managed by `nmcli d status`, but this file get's created again every reboot by netplan, so it does not hold for long. (And then the device is unmanaged again)

[Yannik (yannik-5)]

This is my `/etc/netplan/01-network-manager-all.yaml` file:
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

Here's the output of `netplan apply --debug`:
Found cgroup2 on /sys/fs/cgroup/unified, unified hierarchy for systemd controller
root@arge-pc2:/run# netplan --debug apply
** (generate:2327): DEBUG: 15:42:52.416: Processing input file /etc/netplan/01-network-manager-all.yaml..
** (generate:2327): DEBUG: 15:42:52.416: starting new processing pass
** (generate:2327): DEBUG: 15:42:52.416: Processing input file /run/netplan/enp1s0.yaml..
** (generate:2327): DEBUG: 15:42:52.416: starting new processing pass
** (generate:2327): DEBUG: 15:42:52.416: We have some netdefs, pass them through a final round of validation
** (generate:2327): DEBUG: 15:42:52.416: enp1s0: setting default backend to 1
** (generate:2327): DEBUG: 15:42:52.416: Configuration is valid
** (generate:2327): DEBUG: 15:42:52.416: Generating output files..
** (generate:2327): DEBUG: 15:42:52.416: NetworkManager: definition enp1s0 is not for us (backend 1)
(generate:2327): GLib-DEBUG: 15:42:52.416: posix_spawn avoided (fd close requested)
DEBUG:netplan generated networkd configuration changed, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:enp1s0 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    enp1s0:
      critical: true
      dhcp-identifier: mac
      dhcp4: true
      nameservers:
        addresses:
        - 192.168.2.1
        search:
        - fritz.box
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device enp1s0 operstate is up, not changing
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for enp1s0
Found cgroup2 on /sys/fs/cgroup/unified, unified hierarchy for systemd controller

[Dimitri John Ledkov (xnox)]

What is your /proc/cmdline ?

Do you have `ip=` there? what is it?

It feels like you are booting with `ip=dhcp` set on the kernel commandline, when you do that, we bring netwokring up in the initramfs, and generate netplan.yaml for it.

Can you please confirm if you have any /run/netplan/*.yaml files?

Also can you please check what you have in /etc/cloud.cfg.d/* ? anything that has 'network:' key inside it?

[Yannik (yannik-5)]

What is your /proc/cmdline ?
-----------
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.4.0-29-generic root=/dev/mapper/vgubuntu-root ro quiet splash vt.handoff=7

Do you have `ip=` there? what is it?
-----------
No ip= option as you can see above.

It feels like you are booting with `ip=dhcp` set on the kernel commandline, when you do that, we bring netwokring up in the initramfs, and generate netplan.yaml for it.
-------------
I have an idea about this: The computers have their harddrive encrypted and therefore dropbear-initramfs is installed for remote unlocking. Because of that networking must be brought up during initramfs.

Can you please confirm if you have any /run/netplan/*.yaml files?
-------------------------
$ ls /run/netplan/*.yaml
/run/netplan/enp1s0.yaml

$ cat /run/netplan/enp1s0.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: true
      dhcp-identifier: mac
      critical: true
      nameservers:
        addresses: ["192.168.2.1"]
        search: ["fritz.box"]

Also can you please check what you have in /etc/cloud.cfg.d/* ? anything that has 'network:' key inside it?
----
The directory /etc/cloud.cfg.d/ does not exist.

[Lukas Märdian (slyon)]

dropbear-initramfs sounds interesting...
How did you setup networking inside your initramfs? Apparently you did not use the kernel CMDLINE, did you configure something inside /etc/initramfs-tools/initramfs.conf instead?

[Lukas Märdian (slyon)]

After some more research, I found this "Ask Ubuntu" topic, where the exact same problem is reported: https://askubuntu.com/questions/1228433/what-is-creating-run-netplan-eth0-yaml

It seems like "initramfs-tools" is generating the /run/netplan/enp1s0.yaml file, managed by the "system-dnetworkd" renderer. This is why the "NetworkManager" is not able to manage it at runtime, after the system is booted.

As a workaround, you should be able to override this behavior to set "renderer: NetworkManager" in a file sorting lexically after 'enp1s0.yaml' and redirect this network interface to NetworkManager that way.

But generally speaking I think initramfs-tools should cleanup its side-effects after it is done...

[Yannik (yannik-5)]

> How did you setup networking inside your initramfs? Apparently you did not use the kernel CMDLINE, did you configure something inside /etc/initramfs-tools/initramfs.conf instead?

I have not done any modifications at all to the initramfs or the kernel cmdline except for installing dropbear-initramfs:

$ apt install dropbear-initramfs
$ ln -s /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys
$ update-initramfs -u

That's it.

> As a workaround, you should be able to override this behavior to set "renderer: NetworkManager" in a file sorting lexically after 'enp1s0.yaml' and redirect this network interface to NetworkManager that way.

/run is a tmpfs, so at what point should I do that?

[Lukas Märdian (slyon)]

Hi Yannik,

I think you have two options:
1) Workaround by adding /etc/netplan/xxx.yaml (or any other name which lexically sorts after "enp1s0.yaml"):
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0:
      dhcp4: true

2) Cleanup dropbear-initramfs interfaces (as described at "Ask Ubuntu"):
/etc/initramfs-tools/scripts/init-bottom/deconfigure-interfaces:
rm -f /run/netplan/enp1s0.yaml && ip -f inet address flush dev enp1s0

You probably need to run 'update-initramfs -u' afterwards again...

IMO option (2) is actually a proper solution, this is why I am assigning this bug to 'dropbear', as I think it should automatically cleanup the interfaces/configs it created, once they are not needed anymore.

[Yannik (yannik-5)]

I created a file /etc/initramfs-tools/scripts/init-bottom/deconfigure-interfaces:
#!/bin/sh

rm -f /run/netplan/*

-----

That works fine as a workaround! Flushing the device is not needed.

> IMO option (2) is actually a proper solution, this is why I am assigning this bug to 'dropbear', as I think it should automatically cleanup the interfaces/configs it created, once they are not needed anymore.

The package is called 'dropbear-initramfs', not dropbear.

I checked what dropbear-initramfs is doing:
It installs a script `/usr/share/initramfs-tools/scripts/init-premount/dropbear` , which uses the function `configure_networking()` defined in `/usr/share/initramfs-tools/scripts/functions`.

The latter script is owned by the `initramfs-tools-core` package.

I would assign this bug to that package, as it should imo take care of leaving the networking it has setup in a sane state when starting the OS proper.

[Yannik (yannik-5)]

Any update on this?

[Dimitri John Ledkov (xnox)]

I feel like dropbear-initramfs should learn how to detect if one is on desktop or not, and ensure renderer network-manager is used.

Or like make it a "non-critical" network connection, meaning that no networking configuration is propagated to the host machine.

Howe important is it for you that networking is never torn down, once configured by dropbear whilst booting into the real system?

[Dimitri John Ledkov (xnox)]

alternative, we can drop the renderer setting, then it will be inherited from the installed system.

or I guess we should reorder the file =/

[Dimitri John Ledkov (xnox)]

I think what dropbear does is "right", and initramfs-tools should do the right thing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.136ubuntu8

---------------
initramfs-tools (0.136ubuntu8) groovy; urgency=medium

  * Drop renderer from netplan yaml, such that netplan uses the default
    rendered for the target system, i.e. NetworkManager on the
    desktop. LP: #1876884

 -- Dimitri John Ledkov <email address hidden> Wed, 20 May 2020 15:41:09 +0100

[Steve Langasek (vorlon)]

The initramfs-tools change is reasonable, but I disagree that the dropbear behavior is "correct". The use of 'critical' for network interfaces should be limited to those that actually need to be up for the root filesystem to be available. Using dropbear for a one-time unlock of the rootfs at boot is not that. Especially given that the dropbear bottom hook /specifically deconfigures/ the network interface, it should not be inducing netplan to output yaml that sets 'critical'.

Either the configure_networking function should be extended to take an option to skip the 'critical' setting, or dropbear should do some cleanup in its bottom script.

[Dimitri John Ledkov (xnox)]

@vorlon

I see. Imho, we need to add API to configure "non-critical" internet. Because for example, netboot in subiquity should be "non-critical" too.

[Yannik (yannik-5)]

Will there be a fix for this on focal too?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.