Network Manager is not able to manage the devices on Ubuntu 18.04

[bugproxy]

NetworkManager is not able to manage the devices on latest Ubuntu(18.04)

---uname output---
Linux (none) 4.15.0-12-generic #13-Ubuntu SMP Wed Mar 7 21:36:36 UTC 2018 s390x s390x s390x GNU/Linux

Machine Type = z14 s390

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1. Install the latest Ubuntu(18.04) with Network Manager(1.10.4).
2. Configure a network device and login to the partition through ssh.
3. Now you can see the following output
        root@(none):~# nmcli d s
        DEVICE TYPE STATE CONNECTION
        eth0 ethernet unmanaged --
        eth1 ethernet unmanaged --
        lo loopback unmanaged --

Userspace tool common name: 1.10.6-2ubuntu1: amd64 arm64 armhf i386 ppc64el s390x

The userspace tool has the following bit modes: 64-bit

Userspace rpm: NetworkManager --version 1.10.4

Userspace tool obtained from project website: na

Some more information about the issue:

Network device has been configured manually after the image is up from Support Element(SE):
    - znetconf -a <dev_id>
    - cat /sys/bus/ccwgroup/drivers/qeth/<dev_id>/if_name
    - ifconfig <interface_name> <ip_address> netmask 255.255.255.0
    - route add default gw <gateway_address> <interface_name>
    - SSH service has been configured

This helped us to login to the Lpar. In Lpar
    - output of znetconf -c
        Device IDs Type Card Type CHPID Drv. Name State
        -------------------------------------------------------------------------------------
        0.0.1a80,0.0.1a81,0.0.1a82 1731/01 OSD_10GIG A8 qeth eth0 online
        0.0.1810,0.0.1811,0.0.1812 1731/01 OSD_1000 D0 qeth eth1 online

    - output of nmcli c s
        root@(none):~# nmcli c s
        NAME UUID TYPE DEVICE

    - output of nmcli d s
        root@(none):~# nmcli d s
        DEVICE TYPE STATE CONNECTION
        eth0 ethernet unmanaged --
        eth1 ethernet unmanaged --
        lo loopback unmanaged --

        * The above output shows that devices are not managed by nmcli

After some investigation we found couple of suggestions like
1. Ubuntu(version <17.04): Creating an empty file(/etc/NetworkManager/conf.d/10-globally-managed-devices.conf) and restarting NM,
   solved the issue.

2. Ubuntu(version 17.10): Copying the said file(10-globally-managed-devices.conf) from /usr/lib to /etc/ and modifying the
   "unmanaged-devices" to none, resolved the issue.

* link for reference: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1638842

For the latest version(18.04), none of the above solutions worked.

[Frank Heimes (fheimes)]

First of all some comments:
- why was znetconf used and not chzdev ?
- I'm a bit puzzled reading about "rpm" in combination with Ubuntu
  even if a universe rpm package exists, I assume a network-manager deb package was used, right?
- afaik the network-manager package (that also incl. the nmcli) is a package that belongs to the desktop-packages pocket (need to check this ...)
- is the use of the ip tools (from iproute2) an alternative?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-01 03:11 EDT-------
Team,
Any update on this issue? This is blocking our experimentation of Network Manager on Ubuntu platform.

[Frank Heimes (fheimes)]

So this is more a configuration thing, because with netplan the default renderer is networkd (and not NetworkManager):

ubuntu@zlin:~$ grep renderer /etc/netplan/01-netcfg.yaml
  renderer: networkd

This leads to the fact that no connections are managed by nm by default:

ubuntu@zlin:~$ nmcli con show
NAME UUID TYPE DEVICE
ubuntu@zlin:~$ nmcli d s
DEVICE TYPE STATE CONNECTION
enP1p0s0 ethernet unmanaged --
enP1p0s0d1 ethernet unmanaged --
enP2p0s0 ethernet unmanaged --
enP2p0s0d1 ethernet unmanaged --
encc000 ethernet unmanaged --
lo loopback unmanaged --
encc000.2653 vlan unmanaged --

Changing the renderer from networkd to NetworkManager is probably what you are looking for:

# default:
ubuntu@zlin:~$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    encc000:
      dhcp4: no
      dhcp6: no
  vlans:
    encc000.2653:
      link: encc000
      id: 2653
      addresses: [ 10.245.236.14/24 ]
      gateway4: 10.245.236.1
      nameservers:
          search: [ canonical.com ]
          addresses:
              - "10.245.236.1"

# change the renderer form 'networkd' to 'NetworkManager':

ubuntu@zlin:~$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    encc000:
      dhcp4: no
      dhcp6: no
  vlans:
    encc000.2653:
      link: encc000
      id: 2653
      addresses: [ 10.245.236.14/24 ]
      gateway4: 10.245.236.1
      nameservers:
          search: [ canonical.com ]
          addresses:
              - "10.245.236.1"

# restart netplan / dry-run, to look for any potential config errors
ubuntu@zlin:~$ sudo netplan --debug generate
DEBUG:command generate: running ['/lib/netplan/generate']
** (generate:2472): DEBUG: 13:47:43.846: Processing input file //etc/netplan/01-netcfg.yaml..
** (generate:2472): DEBUG: 13:47:43.846: starting new processing pass
** (generate:2472): DEBUG: 13:47:43.846: encc000.2653: setting default backend to 2
** (generate:2472): DEBUG: 13:47:43.846: encc000: setting default backend to 2
** (generate:2472): DEBUG: 13:47:43.846: Generating output files..
** (generate:2472): DEBUG: 13:47:43.846: networkd: definition encc000.2653 is not for us (backend 2)
** (generate:2472): DEBUG: 13:47:43.846: networkd: definition encc000 is not for us (backend 2)

# restart netplan in case no error are detected
ubuntu@zlin:~$ sudo netplan apply
ubuntu@zlin:~$

# now nm / nmcli has control:
ubuntu@zlin:~$ nmcli dev show
GENERAL.DEVICE: encc000.2653
GENERAL.TYPE: vlan
GENERAL.HWADDR: 02:00:00:33:B5:DD
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: netplan-encc000.2653
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/Acti...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-06 03:26 EDT-------
Team,

We understand Network Manager is good for desktop but not servers.
But given the stability NetworkManager is bringing in and ease of use across different operating systems, i assume users are not restricted not to use NetworkManager for servers. The reason being we going with network manager as explained is to reuse the code for all distros.

Note : Netplan is not present/installed in our ubuntu. So by default we are expecting network manager to work.

root@57fb2571ae8a:~# ls -l /etc | grep netplan
root@57fb2571ae8a:~#

If we need to start using netplan, probably we have to re-write the code to configure yaml files which can be understood by netplan.

So as a workaround, we would like to understand why network manager is failing to manage the devices?
This is definitely to do with open issue, https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1638842

Let me know if more data is needed for analysis, Thanks

[Frank Heimes (fheimes)]

Since the ticket was opened against 18.04, and since 18.04 installations come with and use netplan by default with again networkd as the default renderer (and not NetworkManager), NetworkManager just cannot work by default.

If you want to work with NetworkManager on an 18.04 installation that's using netplan, the described re-configuration is needed and the renderer needs to be changed from networkd to NetworkManager - that's not a bug, that's intended and caused by the introduction of netplan (since about 17.10).
With that change NetworkManager should be able to manage qdio devices (see end of comment #3 - devices are listed as managed).

If you still have problems managing the devices _after_ doing the re-config there might be a bug, but I don't see that right now (but I for sure didn't covered your entire use case that I just don't know).

We understand that behavior is now a bit different compared to other distributions.

There were some bugs in the past on NetworkManager and netplan on previous Ubuntu releases, like the one mentioned by you in the bug description:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1638842
which is a duplicate of:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1676547
and already 'Fix Release' (since quite some time).
And btw. comment 26: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1676547/comments/26 confirms that the above renderer change works.

If issues still occur on other Ubuntu releases (not 18.04, but for example 16.04) that are not yet addressed in an LP bug, please open a separate bug on them.
And if there are still issues managing qdio devices after the change above, we may address them is a separate ticket, too.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-06 14:43 EDT-------
Team,

We understand Network Manager is good for desktop but not servers.
But given the stability NetworkManager is bringing in and ease of use across different operating systems, i assume users are not restricted not to use NetworkManager for servers. The reason being we going with network manager as explained is to reuse the code for all distros.

Note : Netplan is not present/installed in our ubuntu. So by default we are expecting network manager to work.

root@57fb2571ae8a:~# ls -l /etc | grep netplan
root@57fb2571ae8a:~#

If we need to start using netplan, probably we have to re-write the code to configure yaml files which can be understood by netplan.

So as a workaround, we would like to understand why network manager is failing to manage the devices?
This is definitely to do with open issue, https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1638842

Let me know if more data is needed for analysis, Thanks

[Frank Heimes (fheimes)]

So your system is obviously not an 18.04 default installation from scratch - otherwise you would have netplan.io installed and this folder and file "/etc/netplan/01-netcfg.yaml" will exist on your system.

There is no known way to me to install an 18.04 system without having netplan (initially) as default.

Did you have removed netplan manually from that installation?
Or is your 18.04 system and upgrade from an older Ubuntu release (like 16.04 or 17.10)?

If it's an 18.04 installation from scratch please provide the installation logs.
They are during the installation available at /var/log/*
and after the installation still available at /var/log/installer/*

[Dimitri John Ledkov (xnox)]

"We understand Network Manager is good for desktop but not servers." I would not phrase it this away. On Ubuntu, Desktop defaults to NetworkManager and Server defaults to netplan&networkd. Previously we defaulted to NetworkManager & ifupdown.

"But given the stability NetworkManager is bringing in and ease of use across different operating systems, i assume users are not restricted not to use NetworkManager for servers." I would significantly challenge those statements. NetworkManager is extremely flakey, and consistently fails to reliably bring complex network configurations up, as exercised by our stress testing. It also is far from easy to use. Wrt. users restrictions, Ubuntu tries to guide users to an obviously correct, easy, default and obvious ways of doing things. However, we do indeed allow users to break their systems and keep all the broken pieces.

"The reason being we going with network manager as explained is to reuse the code for all distros." I don't believe that's the right thing to do, as that does not validate how end customers will use respective distros on z hardware. All customers will most likely use whatever each distro installer sets up, and will not tear that out and go out of their way to use networkmanager in a lowest common denominator configuration (e.g. Ubuntu's network-manager is patched, and is at a different version level than other distros)

"Note : Netplan is not present/installed in our ubuntu. So by default we are expecting network manager to work."

After installing a system, if you want netplan to be out of the way - and have the ability to use straight-up netwokrd/NM/etc, I'd rather recommend removing /etc/netplan/* configs and rebooting, without uninstalling netplan.io package. This would guarantee that interfaces are undeclared as managed-on-unmanaged by anything.

[Dimitri John Ledkov (xnox)]

(given architecture-all tag; replace "z hardware" with "any server hardware" - above comments are architecture independent across all Ubuntu products)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-07 03:18 EDT-------
"So your system is obviously not an 18.04 default installation from scratch" This is correct. We are not using a standard installation but build an appliance with a minimal footprint. Our build process builds the appliance using debootstrap and we are using a local apt repository with the deb packages from UBUNTU 18.04 as input for debootstrap. So we are basically controlling what packages are installed in our appliances and right now netplan is not included.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-07 06:47 EDT-------
Hi there,

Now Network Manager(NM) is able to manage the devices with the following changes.
Change the contents of the file /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf to
----------------------------------------------------
[keyfile]
unmanaged-devices=none
----------------------------------------------------

from
----------------------------------------------------
[keyfile]
unmanaged-devices=*,except:type:wifi,except:type:wwan
----------------------------------------------------

and restart NM(service network-manager restart) did the trick.

Why it didn't work earlier:
We added interfaces to the file /etc/network/interfaces.
So NM is not able to manage the interfaces updated in the said file.

When we removed the interfaces from the above file, It started working.

Is this approach correct ??

[Dimitri John Ledkov (xnox)]

> We added interfaces to the file /etc/network/interfaces.
> So NM is not able to manage the interfaces updated in the said file.
> When we removed the interfaces from the above file, It started working.
>
> Is this approach correct ??

Yes.

NetworkManager, as shipped in Ubuntu, comes with multiple plugins and has networkd/ifupdown co-operation enabled by default. Out of the box, NetworkManager will not manage any regular eth devices that are already managed by somebody else which includes networkd and ifupdown. One should only ever configure one device using one of the technologies, as usually, multiple configurations of the same device using different stacks is an indication of a configuration mistake.

For the eth devices, we currently prefer networkd&ifupdown, over NM. And the preference of wifi/3g/4g/lte-like adhoc devices is with NM, over other stacks.

If you are exercising testing of various stacks, do ensure you cleanup systems to a pristine state, without any configs, for any of the stacks first, before proceeding automating using a specific network stack.

Removing or not installing netplan, should not be required to achieve what you want. All Ubuntu systems should have ubuntu-minimal meta-package installed, which means netplan.io package should be left installed too. Remvoing / not-installing ubuntu-minimal, may result in upgrade and runtime issues, as such systems are harder to correctly support. Please evaluate and ensure that ubuntu-minimal is installed in your case. Given NetworkManager usage, it shouldn't add too much cruft, if installed without recommends.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-19 07:30 EDT-------
IBM bugzilla status -> closed, Problem solved.....

[Nicholas Roberts (niccolox)]

I have a clean Ubuntu 18.04 Desktop install with defaults on an HP Envy

I get the same error

WIFI works

PCI Ethernet and USB Ethernet connectors both fail when adding them static or DHCP via Network Manager

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

wlp3s0: connected to Big_House
        "Intel Wireless 7265 (Dual Band Wireless-AC 7265)"
        wifi (iwlwifi), 7C:76:35:32:8E:4E, hw, mtu 1500
        ip4 default, ip6 default
        inet4 10.1.10.92/24
        route4 0.0.0.0/0
        route4 10.1.10.0/24
        route4 169.254.0.0/16
        inet6 2603:3024:1404:3800::5e1f/128
        inet6 2603:3024:1404:3800:7cd5:1de8:c4a3:ca53/64
        inet6 2603:3024:1404:3800:c9a2:5e0b:bbb:edab/64
        inet6 fe80::5d74:8f35:8843:bbd7/64
        route6 2603:3024:1404:3800::/64
        route6 ::/0
        route6 ff00::/8
        route6 fe80::/64
        route6 fe80::/64
        route6 2603:3024:1404:3800::5e1f/128

enp2s0f1: disconnected
        "Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller"
        1 connection available
        ethernet (r8169), B4:B6:86:88:79:C1, hw, mtu 1500

enx8cae4cff12e2: unavailable
        "ASIX Elec. AX88178"
        ethernet (asix), 8C:AE:4C:FF:12:E2, hw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 75.75.75.75 75.75.76.76
        domains: hsd1.ca.comcast.net
        interface: wlp3s0

        servers: 2001:558:feed::1 2001:558:feed::2
        interface: wlp3s0

/usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf

[keyfile]
unmanaged-devices=none

I also keep getting the Connection Failed notification, even when only in WIFI mode

its a bug, clean install

[Nicholas Roberts (niccolox)]

cancel that, netplan/network-manager worked for the ethernet after installing later version of the drivers

[Frank Heimes (fheimes)]

Hi Nicholas, since the original ticket is already closed and the particular problem solved, I strongly suggest to open a separate new Launchpad bug for your case.

[Sumit Solomon (ssolomon)]

My version ubuntu server 18.04. Below worked for me:

1. Creating an empty file(/etc/NetworkManager/conf.d/10-globally-managed-devices.conf)

2. removing the file(10-globally-managed-devices.conf) from /usr/lib

and finally

3. changing Network renderer to NetworkManager in Yaml file.

Yaml file location: /etc/netplan

My yaml was like this:

network:
    ethernets:
      eno1:
       dhcp4: true
    version: 2

Changed to:

network:
 version: 2
 renderer: NetworkManager
 ethernets:
  eno1:
   dhcp4: true

and I change it to:

[Alkis Georgopoulos (alkisg)]

While working on a remote server, it took me 2-3 hours to locate this bug report and apply its workarounds. It's certainly not a good default behavior.

In case someone has already removed netplan, the recommended steps to get network-manager to manage the interfaces, as I understood them, are:

```
sudo -i
apt install ubuntu-minimal
echo "# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager" >/etc/netplan/01-network-manager-all.yaml
update-initramfs -u # Not sure if this is needed or not
reboot
```

I.e. I think that /etc/netplan/01-network-manager-all.yaml comes from some installer and it's not part of some package and it needs to be manually re-created.