No network connectivity after upgrade from 16.04 to 16.10

Bug #1676547 reported by Brian Murray on 2017-03-27
192
This bug affects 69 people
Affects Status Importance Assigned to Milestone
network-manager (Ubuntu)
Critical
Brian Murray
Yakkety
Critical
Unassigned
Zesty
High
Brian Murray

Bug Description

Impact
------
When upgrading from Xenial (with all updates installed) to Yakkety/Zesty you will boot to a system without networking - sad!

Test Case
---------
1) Boot a xenial desktop system
2) Ensure network-manager version 1.2.6-0ubuntu0.16.04.1 from -updates is installed
3) Upgrade to yakkety or zesty
3.5) Enable -proposed and download the new version of n-m "apt-get download network-manager"
4) Reboot
5) Observe you have no network

If you install the version of network-manager from yakkety-proposed and then reboot you should have a network connection.

1) Boot a xenial desktop system.
2) ensure network-manager version 1.2.6-0ubuntu0.16.04.1 from -updates is installed.
3) Install nplan, configure to set up static network on an ethernet device using nplan.
4) Upgrade to yakkety or zesty
5) Reboot
6) Observe that your system is still being managed by networkd rather than NetworkManager.

Regression Potential
--------------------

Any failure to manage ethernet or wireless devices after upgrade, or issues with the use of netplan in conjunction with NetworkManager should be investigated as potential regressions. For example, if after upgrading, ethernet devices are no longer managed, or if devices that should be explicitly ignored by NetworkManager due to existing user configuration are now managed, these would indicate a possible regression caused by this update.

Original Description
--------------------

nplan was installed during the upgrade process but I had no network connectivity after upgrading. Apparently, no package wanted to manage my ethernet connection.

ProblemType: BugDistroRelease: Ubuntu 16.10
Package: ubuntu-release-upgrader-core 1:16.10.10
ProcVersionSignature: Ubuntu 4.8.0-41.44-generic 4.8.17
Uname: Linux 4.8.0-41-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.3-0ubuntu8.2
Architecture: amd64
CrashDB: ubuntu
CurrentDesktop: Unity
Date: Mon Mar 27 11:34:04 2017
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-01-16 (1531 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Alpha amd64 (20130116)
PackageArchitecture: allSourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: Upgraded to yakkety on 2017-03-17 (10 days ago)
VarLogDistupgradeTermlog:

modified.conffile..etc.update-manager.meta-release: [modified]
mtime.conffile..etc.update-manager.meta-release: 2013-10-08T06:39:09.818913

Brian Murray (brian-murray) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
tags: added: xenial2yakkety

Triaged/Critical: This will affect upgrades from Xenial to 18.04. It needs to be addressed properly.

Currently, upgrade fails this way because the postinst for network-manager is looking for a lower version than what is now in xenial-updates to gate the "migration" to netplan and writing an override file.

Changed in ubuntu-release-upgrader (Ubuntu):
status: Confirmed → In Progress
importance: Undecided → Critical
assignee: nobody → Mathieu Trudel-Lapierre (cyphermox)

I'm reasonably certain that I can fix this in NM directly; for now I've uploaded a new version of network-manager for yakkety to my PPA (ppa:cyphermox/sru) to do the upgrade testing from Xenial to Yakkety. It involves first making sure xenial is fully updated to network-manager 1.2.6.

Changed in ubuntu-release-upgrader (Ubuntu):
milestone: none → ubuntu-17.04
affects: ubuntu-release-upgrader (Ubuntu) → network-manager (Ubuntu)
Skewray (ubuntu-skewray) wrote :

Any suggestions on how those of us affected by this can retroactively fix our systems? I am using "sudo dhclient p1p1" after every boot to get the ethernet up, but some programs such as "evolution" still do not have internet access.

James Gross (xeboc) wrote :

Info I've gathered about this issue so far:

The netplan should be written to this location:

/etc/netplan/01-network-manager-all.yaml

And should contain:

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

See this reference:

http://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/auto/build#L196

If the file is missing, it can be manually added.

Ubuntu transitioned to netplan right before version 1.2.2-0ubuntu8. See here:

https://lists.ubuntu.com/archives/ubuntu-devel/2016-July/039464.html

Additionally, this is related to these 2 bugs:

https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1638842
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1658921

When 1.2.2 was released, a globally managed file was created (/etc/NetworkManager/conf.d/10-globally-managed-devices.conf), and a canceling file was also created (/etc/NetworkManager/conf.d/10-globally-managed-devices.conf).

https://git.launchpad.net/network-manager/tree/debian/network-manager.postinst#n48

For some reason, either the override file or the netplan file is missing on upgrades. Between the two of them, users are left without fully managed Ethernet and networking when upgrading from 16.04.

That's indeed what seems to be the case, and this has already been debugged. The issue is that the code that does the override on upgrade looks for the wrong version of NetworkManager, since it has been updated in Xenial.

This change has been in the process of being tested; I'll upload to the release proper as soon as I know that it really does work.

The netplan code itself should be created only on new installs; for an upgrade, it's not there, and thus we need to explicitly write an override for NM via NM postinst.

James Gross (xeboc) wrote :

Ah, so this code:

 if dpkg --compare-versions "$2" le-nl "1.2.2-0ubuntu4"; then
  mkdir -p /etc/NetworkManager/conf.d || true
  # for old versions, override the global config with a null config
  touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf

Should have ge-nl rather than le-nl? Seems like it's backwards and that all new installs need the override, rather than old installs.

Also, why the decision to only do a netplan on install?

Łukasz Zemczak (sil2100) wrote :

Could this bug get its description updated to include the SRU details? (as per https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template). We can't really reliably review this without it.
Thanks!

description: updated
description: updated
Changed in network-manager (Ubuntu Yakkety):
status: New → Triaged
importance: Undecided → Critical
description: updated

Hello Brian, or anyone else affected,

Accepted network-manager into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.2.6-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in network-manager (Ubuntu Yakkety):
status: Triaged → Fix Committed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

I upgraded a Xenial system to Yakkety, then enabled -proposed and installed the new version of network-manager.

bdmurray@clean-xenial-amd64:~$ apt-cache policy network-manager
network-manager:
  Installed: 1.2.6-0ubuntu1
  Candidate: 1.2.6-0ubuntu1.1
  Version table:
     1.2.6-0ubuntu1.1 500
        500 http://192.168.10.7/ubuntu yakkety-proposed/main amd64 Packages
 *** 1.2.6-0ubuntu1 500
        500 http://192.168.10.7/ubuntu yakkety-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.4-0ubuntu1 500
        500 http://192.168.10.7/ubuntu yakkety/main amd64 Packages
bdmurray@clean-xenial-amd64:~$ sudo apt-get install network-manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  network-manager
1 upgraded, 0 newly installed, 0 to remove and 52 not upgraded.
Need to get 1,996 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://192.168.10.7/ubuntu yakkety-proposed/main amd64 network-manager amd64 1.2.6-0ubuntu1.1 [1,996 kB]
Fetched 1,996 kB in 0s (82.5 MB/s)
(Reading database ... 188256 files and directories currently installed.)
Preparing to unpack .../network-manager_1.2.6-0ubuntu1.1_amd64.deb ...
Unpacking network-manager (1.2.6-0ubuntu1.1) over (1.2.6-0ubuntu1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up network-manager (1.2.6-0ubuntu1.1) ...
Processing triggers for systemd (231-9ubuntu4) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for dbus (1.10.10-1ubuntu1.1) ...
bdmurray@clean-xenial-amd64:~$ sudo shutdown -r now
Connection to clean-xenial-amd64.local closed by remote host.
Connection to clean-xenial-amd64.local closed.

After the reboot I had network connectivity.

 $ ssh clean-xenial-amd64.local
Welcome to Ubuntu 16.10 (GNU/Linux 4.8.0-49-generic x86_64)

Setting to verification-done.

description: updated

Verified network-manager 1.2.6-0ubuntu1.1 in yakkety-proposed. Upgrades with and without nplan installed appear to work as expecte.d

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 1.2.6-0ubuntu1.1

---------------
network-manager (1.2.6-0ubuntu1.1) yakkety; urgency=medium

  * debian/network-manager.postinst: drop in an empty override file for
    NetworkManager to manage all devices for upgrade from any version, as long
    as there is no netplan configuration yet. (LP: #1676547)

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 27 Mar 2017 17:02:27 -0400

Changed in network-manager (Ubuntu Yakkety):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for network-manager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Stefan Leitner (s-o-l) wrote :

The problem persists if you upgrade from 16.04.3 directly to 17.04 ( as described https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1658921/comments/23 )

Felipe Reyes (freyes) wrote :

> The problem persists if you upgrade from 16.04.3 directly to 17.04 ( as described https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1658921/comments/23 )

I can confirm this. My upgrade path was xenial -> zesty -> artful and I had to run "touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf" so network-manager could handle my enp2s0 device.

Changed in network-manager (Ubuntu Zesty):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Changed in network-manager (Ubuntu):
assignee: Mathieu Trudel-Lapierre (cyphermox) → Brian Murray (brian-murray)
milestone: ubuntu-17.04 → ubuntu-17.09

Hello Brian, or anyone else affected,

Accepted network-manager into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.4.4-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

description: updated
Changed in network-manager (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
removed: verification-done
description: updated
description: updated
Brian Murray (brian-murray) wrote :

I've verified this is fixed with the version of network-manager in zesty-proposed, using both the testcases (with and without nplan installed).

tags: added: verification-done-zesty
removed: verification-needed-zesty
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers