network-manager crashes when using libnl-3-200 3.2.21-1ubuntu1

Bug #1539634 reported by Ryan Harper on 2016-01-29
524
This bug affects 178 people
Affects Status Importance Assigned to Milestone
network-manager (Ubuntu)
High
Unassigned
Trusty
High
Unassigned

Bug Description

[Triage Notes]

This regression is caused by a libnl proposed update tracked in bug 1511735. It has not been released as an update, only proposed as an update for early testing. If you volunteered to test proposed updates, then thank you. Your care and attention will stop this regression from hitting ordinary users, and we appreciate your contribution to Ubuntu.

If you are not aware that you volunteered to test proposed updates, then please be aware that your system is configured to do so. In this case, you probably should turn this off. I'd appreciate if someone could explain how to do this in the comments.

[Impact]

 * NetworkManager depends on libnl (libnl-3-200 libnl-genl-3-200
   libnl-route-3-200) and when libnl is updated to proposed
   3.2.21-1ubuntu1 NM segfaults due to libnl exposing a bug in NM
   validation packets.

   This affects the 0.98 release of NetworkManager and has already
   been fixed in newer releases.

 * Backporting fixes from upstream release is required to prevent
   NetworkManager from faulting which breaks networking on most
   Desktop releases.

 * All patches applied are already accepted upstream and newer Ubuntu
   releases are not affected.

 * Upgrading NetworkManager to 0.9.8.8-0ubuntu7.3 prior to updating
   libnl-3.2.21-1ubuntu1 is required to prevent NetworkManager crashing.

[Test Case]

 * Reproduce crash test:
    1. Download 14.04.03 Desktop iso, amd64
    2. Launched in a VM, run from iso
    3. In terminal, enable proposed
    4. sudo apt-get install libnl-3-200 libnl-genl-3-200 libnl-route-3-200
    5. sudo restart network-manager
    # note the nm-applet disappears as NM has now crashed, no network
       conn.
    6. dmesg to confirm network-manager crash
    7. sudo dpkg --install gir1.2-networkmanager-1.0_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib4_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib-vpn1_0.9.8.8-0ubuntu7.3_amd64.deb libnm-util2_0.9.8.8-0ubuntu7.3_amd64.deb network-manager_0.9.8.8-0ubuntu7.3_amd64.deb
    8. sudo start network-manager
    # nm-applet reappears, network connectivity restored

 * Successful upgrade test:
   1. Download 14.04.03 Desktop iso, amd64
   2. Launched in a VM, run from iso
   3. In a terminal, sudo dpkg --install gir1.2-networkmanager-1.0_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib4_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib-vpn1_0.9.8.8-0ubuntu7.3_amd64.deb libnm-util2_0.9.8.8-0ubuntu7.3_amd64.deb network-manager_0.9.8.8-0ubuntu7.3_amd64.deb
   4. sudo restart network-manager # NM still runs, no crash, net up
   5. enable trusty-proposed in /etc/apt/sources.list
   6. sudo apt-get update
   7. sudo apt-get install libnl-3-200 libnl-genl-3-200 libnl-route-3-200
   8. sudo restart network-manager # NM still runs, no crash, net up.

[Regression Potential]

 * NetworkManager is a highly visible package, so testing NetworkManager
   functionality is critical. The patch does address the crash specificly
   introduced by the updated libnl SRU.

 * We need to ensure that users have installed this NetworkManager
   update before installing libnl3 update (special phasing needed).

[Original Description]
Testing out proposed libnl-3 package[1] exposed a bug in NM 0.98.

After installing updated libnl3 and restarting networkmanager, NM crashes with:

/var/log/syslog shows:
init: network-manager main process (1057) killed by SEGV signal
init: network-manager main process ended, respawning
init: network-manager main process (1065) killed by SEGV signal
init: network-manager respawning too fast, stopped

The bug has been fixed in upstream 0.98 with these patches:

http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=8e4576b9fdb5c888d20a13aa2cc198df790dba54
http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=65981edb9f562c07e78815c98093da67c50bfdcf

Ryan Harper (raharper) wrote :

I applied the two upstream commits into NM 0.98; built the package and it does fix the libnl-3 updated issue. NM no longer crashes when using the updated (proposed) libnl-3.

Ryan Harper (raharper) wrote :
Robie Basak (racb) on 2016-01-29
Changed in network-manager (Ubuntu Trusty):
status: New → Triaged
Ryan Harper (raharper) wrote :

Dev release is not affected (libnl version >= 3.2.25 and network-manager > 0.98 ) Xenial has 1.0.4.

Changed in network-manager (Ubuntu):
status: New → Fix Released
HugoHirsch (ubuntubugs-aiki-it) wrote :

Is it possible to somehow block libnl-updates until the network-manager fix is available as well?

Otherwise people are going to run into networkless systems as reported in #1539513
Since 14.04 is a LTS this is rather risky indeed.

Vadim Peretokin (vperetokin) wrote :

Yep, already have ran into that problem - rebooted and had no internet due to NM repeatedly crashing. This is a severe issue.

Ryan Harper (raharper) wrote :

Hi,

For now, you'll need to downgrade your libnl-3 packages:

% apt-get install libnl-3-200=3.2.21-1 libnl-genl-3-200=3.2.21-1 libnl-route-3-200=3.2.21-1

And you may need to apply a hold until a fix for NetworkManager can be released.

% sudo apt-mark hold libnl-3-200 libnl-genl-3-200 libnl-route-3-200

And after the fix:

% sudo apt-mark auto libnl-3-200 libnl-genl-3-200 libnl-route-3-200

aleandro (aleandrodasilva) wrote :

Confirm the bug in ubuntu 14.04 proposed updates.

 A workaround is to install the program wicd-gtk or to downgrade some libraries as suggested.

It is a bad bug since your connection to the web becomes impossible. To surf in the web and see this bug I used the bluetooth connection of my smartphone.

Samuel (aroqsi) wrote :

Downgrading using the terminal needs internet in our machine. And if we paste the previous libnl with a USB stick and click, Ubuntu Software Center says -Error. A later version is already installed-. I tried desperate many more things. Provably only a new repaired libnl UPDATE which we can open by clicking might really solve this for everyone. I believe there are more people affected but reporting this bug requires internet.

Thank you.

Alex Frenkel (sirshurf) wrote :

Managed to bring Internet back by using:

sudo ifconfig eth0 up
sudo dhclient

But I cannot downgrade:

sudo apt-get install libnl-3-200=3.2.21-1 libnl-genl-3-200=3.2.21-1 libnl-route-3-200=3.2.21-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '3.2.21-1' for 'libnl-3-200' was not found
E: Version '3.2.21-1' for 'libnl-genl-3-200' was not found
E: Version '3.2.21-1' for 'libnl-route-3-200' was not found

Ryan Harper (raharper) on 2016-02-02
description: updated
Ryan Harper (raharper) wrote :

v2
  - Add DEP3 header to patches

Robie Basak (racb) on 2016-02-02
description: updated

Hello Ryan, or anyone else affected,

Accepted network-manager into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/0.9.8.8-0ubuntu7.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in network-manager (Ubuntu Trusty):
status: Triaged → Fix Committed
tags: added: verification-needed

I've run the 'Successful upgrade Test' steps from the SRU text successfully. Verified working. Thanks.

Dmitry Ershov (edvice) wrote :

Fixes bug for me after upgrade:

gir1.2-networkmanager-1.0:amd64 0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3
libnm-glib-vpn1:amd64 0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3
libnm-glib4:amd64 0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3
libnm-util2:amd64 0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3
network-manager:amd64 0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3

verification-done? or anyone else?

dynaguy (dynaguy) wrote :

Fix confirm. Thanks a lot.

I notice I noticed this updated network-manager is only available in trusty-proposed repo now.

Martin Beynon (mebeyn) wrote :

Unfortunately, I have found that it did not work for me.

I first tried a standard apt-get upgrade (in which I noticed a new network-manager package) and rebooted, to find no real change - network-manager still SEGVs.

Then I tried downgrading libnl-3 (as described above) and rebooted. Again, no change - network-manager still SEGVs.

Correct. We pushed the nm fix so anyone using trusty-proposed would be
fixed but the libnl fix needs more time to test. Only users who have
trusty-proposed are affected and this fixes that while we do further
validation on libnl dependent applications for regressions.
On Feb 2, 2016 7:26 PM, "dynaguy" <email address hidden> wrote:

> Fix confirm. Thanks a lot.
>
> I notice I noticed this updated network-manager is only available in
> trusty-proposed repo now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1539634
>
> Title:
> network-manager crashes when using libnl-3-200-3.21.1-1ubuntu1
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1539634/+subscriptions
>

Thanks a lot,
Fixes bug for me after upgrade:

0.9.8.8-0ubuntu7.2 -> 0.9.8.8-0ubuntu7.3
gir1.2-networkmanager-1.0
libnm-glib-vpn1
libnm-glib4
libnm-util2
network-manager

3.2.21-1 : ok + 3.2.21-1ubuntu1: ok
libnl-3-200
libnl-genl-3-200
libnl-route-3-200

Stephen Hazleton (shazleto) wrote :

Fix confirmed here, thanks all.

Martin Beynon (mebeyn) wrote :

Unfortunately, I have found that it did not work for me.

I first tried a standard apt-get upgrade (in which I noticed a new network-manager package) and rebooted, to find no real change - network-manager still SEGVs.

Then I tried downgrading libnl-3 (as described above) and rebooted. Again, no change - network-manager still SEGVs.

Changed in network-manager (Ubuntu):
importance: Undecided → High
Changed in network-manager (Ubuntu Trusty):
importance: Undecided → Critical
importance: Critical → High
tags: added: regression-proposed
Blacky Bones (bones210) wrote :

I have been following this since Saturday night after installing the update. I was able to connect to the Ethernet with the workaround #7.

 I just installed the latest update from the Software Updater and it boots fine. I still get a system error message, but the Internet works.

The internet icon is now active, but when I disconnect from the Ethernet, it shows that it is disconnected, BUT I am still connected.

Don't know if this helps.

I did remove the workaround #7 wicd-gtk

Jan Wester (j-vester) wrote :

Tested the fix. For me it seems to work. This combos work:

ii gir1.2-networkmanager-1.0 0.9.8.8-0ubuntu7.3 amd64 GObject introspection data for NetworkManager

ii libnl-3-200:amd64 3.2.21-1ubuntu1 amd64 library for dealing with netlink sockets
ii libnl-genl-3-200:amd64 3.2.21-1ubuntu1 amd64 library for dealing with netlink sockets - generic netlink
ii libnl-route-3-200:amd64 3.2.21-1ubuntu1 amd64 library for dealing with netlink sockets - route interface

Jan Wester (j-vester) wrote :

Just to be complete. Also this packages were updated:

ii libnm-util2 0.9.8.8-0ubuntu7.3 amd64 network management framework (shared library)

ii libnm-glib4 0.9.8.8-0ubuntu7.3 amd64 network management framework (GLib shared library)

i libnm-glib-vpn1 0.9.8.8-0ubuntu7.3 amd64 network management framework (GLib VPN shared library)

emmanuel okeke (emmanix2002) wrote :

Fix confirmed, it works fine for me and network-manager doesn't crash even after removing the hold on libnl* packages and upgrading them.

minimalprocedure (zairik) wrote :

Fix working for me.

Georgios Tsipotis (gtsipotis) wrote :

Kubuntu 14.04 user here. After updating network manager and libnl* packages, ethernet came back but can not enable wireless. Had to downgrade the 3 libnl* packages again for the wireless settings to reappear.

mirak011 (mirak) wrote :

Fix really works, awesome! Thanks.

Samuel (aroqsi) wrote :

How did you fix it? I assume that I have to go to "https://launchpad.net/ubuntu/+source/network-manager/0.9.8.8-0ubuntu7.3" and to "Downloads" where there are 2 folders and one file. If that is right, what should I do with those and how can I install them?
Thank you!

Galen Thurber (godfree2) wrote :

see Question #284147 : Questions : Ubuntu
after downloading the debs via another distro
and
sudo dpkg -i --force-downgrade libnl-3-200_3.2.21-1_amd64.deb libnl-genl-3-200_3.2.21-1_amd64.deb libnl-route-3-200_3.2.21-1_amd64.deb
reboot
my
 amd64 14.04
now has networkmanager back populated with one eth0

Marsh (marsh) wrote :

Worked for me. Updated nl libs as well. Worked even after reboot.

joe invader (mrinvader) wrote :

The fix in proposed also fixed it here!! thank you from myself and all my folks who had this happen, to all those who triaged this issue and fixed it!

LucaChiampo (luca-chiampo) wrote :

Same same problem

Following suggestion of comment #27 bug #1539513

I restored network with:

:# ip link set dev eth0 up
:# ip addr add 192.168.x.z/24 dev eth0
:# ip route add default via 192.168.x.y dev eth0
:# echo nameserver 192.168.x.y >> /etc/resolve.conf

 and than updated NetworkManarger to proposed version 0.9.8.8-0ubuntu7.3 via synaptic
no downgrade of libnl.

Solved for me

Samuel (aroqsi) wrote :

After 40 hours of trying, I bought a DVD reader and reinstalled Ubuntu.
I encourage to implement: when we accept proposed updates, a warning appears explaining the risks.
Reason: many of us can not understand what is suggested here or elsewhere in Internet related with Terminal -no further than copy&paste- even if we have an extra computer to read online the suggestions -Network Manager was broken- and try to solve this problem for days, we still can not repair this bugs.
PS: move/copy this suggestion if you think of a better place for implementation.

aleandro (aleandrodasilva) wrote :

Hello there. I removed old nm package and reinstalled it by enabling the proposed. I can connect to the web but I encounter problems when I switch to another wireless hotspot. Then I see an error related to libnl and the connection crashes. I will try to remove the libnl too.

aleandro (aleandrodasilva) wrote :

Hello I removed all packages and reinstalled the necessary. It seems to work fine yet.

Ryan Harper (raharper) on 2016-02-17
tags: added: verification-done
removed: verification-needed

The verification of the Stable Release Update for network-manager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package network-manager - 0.9.8.8-0ubuntu7.3

---------------
network-manager (0.9.8.8-0ubuntu7.3) trusty; urgency=medium

  * netlink-fix-crash-in-netlink-monitor-validating-netlink-messages.patch:
  * netlink-simplify-netlink-message-validation.patch
    Apply the above patches prevent NM from crashing when passing in wrong
    pointer due to updated libnl. (LP: #1539634)

 -- Ryan Harper <email address hidden> Fri, 29 Jan 2016 08:14:17 -0600

Changed in network-manager (Ubuntu Trusty):
status: Fix Committed → Fix Released
Tero Gusto (tero-gusto) wrote :

I am getting no internet after booting and the "network-manager main process ... killed by SEGV signal" error in my log file today, after having used internet via my computer this morning.

I can bring Internet back by using:

sudo ifconfig eth0 up
sudo dhclient

John B. (jbuncher) wrote :

Same here, filed Bug #1581617 to report this new issue.

castlehard (castlehard) wrote :

Seems like a "This bug was fixed in the package network-manager - 0.9.8.8-0ubuntu7.3" problem.

Same at my place, no proposal repository activated.

My network-manager still is the "old" 0.9.8.8-0ubuntu7.2. So it seems for me like the update of the network-manager did not make it into the "yesterday's patchset", and now we are a bit offline.

So: update the patchset-list to make both changes happen the same time. I think this would solve all the problems with this bug.

castlehard (castlehard) wrote :

-> manually updated network-manager to the 7.3 Version:
# sudo apt-get install network-manager
and did an unhold of the libnl-* packages, started update-manager (did an update to these libnl things) and rebooted as wanted by the update-manager.

Everything works fine now.

Michel-Ekimia (michel.ekimia) wrote :

This happened today to a customer on 14.04 who did NOT have proposed enable.

castlehard (castlehard) wrote :

As from yesterday it seems this update was moved from purposed to updates, so it will happen to anyone updating 14.04 LTS .
So it's highly important to bundle the network-manager update to this libnl-updates IN ONE STEP. (a job for canonical here ;) ).

All or nothing should be the right way to keep customers or users away from this problem.

Daniel Browne (grimsqueaker13) wrote :

Just happened to me, 14.04 without proposed updates, nothing in interfaces file so I can't even bring it up manually.

This hit me 1 hour ago after i had a pwer down because i forgot to link the power cable. After reboot i had no network at all. I had no symbol in the header bar. "rfkill list all" shows all blocked. manually starting "service network-manager restart" not wirked with an exception. After searching alsmost one hour for the problem i looked in

> cat /var/log/syslog

and search then for the logged error in google with

ubuntu networkmanager general protection

this bug. Now with

> ifconfig wlan0 up

and

> ifconfig eth0 up

and

> dhclient

it worked. Then i did an

> apt-get update and apt-get upgrade

while that the networkmanager comes back to life and the symbol comes back to life.

Erwin Segerer (erwin-segerer) wrote :

As of yesterday I had no internet connection (updates where performed on 14.04 without proposed updates enabled).
Thanks Matthias, your comments did the trick for me too, internet is back (just had to restart the network-manager afterwards):

sudo ifconfig wlan0 up
sudo ifconfig eth0 up
sudo dhclient

sudo apt-get update
sudo apt-get upgrade

sudo service network-manager restart

Vasiliy Pedak (truvazia) wrote :

I have no internet connection, since I've started my laptop after shutdown today morning.

Vasiliy Pedak (truvazia) wrote :

I've managed to make all working by such steps http://askubuntu.com/a/727462 - manual downloading and installing packages. Except adding the freshest libnl* packages on hold. So after manual downgrading and restarting `network-manager` I got network working. Then I did

sudo apt-get update
sudo apt-get upgrade

Everything works now.

David Strausz (strausz-s) wrote :

Received 3.2.21-1ubuntu1.1 versions of libnl* with upgrade about a day ago, even though definitely have "proposed" repository turned off. This cause network-manager to fail after boot, as described.

Recovered by manual downgrade of libnl* as described (http://askubuntu.com/a/727462).

Then upgraded again when internet connection restored. This re upgraded the libnl* packages to the 3.2.21-1ubuntu1.1 versions again, but also upgraded network-manager to 0.9.8.8-0ubuntu7.3 (it was previously vrs. 7.2). Everything works again with these versions.

Tom Dalton (tom-dalton) wrote :

My Ubuntu 14.04 laptop downloaded these 3 3.2.21-1ubuntu1 packages on Friday 13th May, and after a restart my network manager was unable to start (segfaulting presumably for the same reasons as original).

Downgrading to the previously-installed -1 packages worked for me.

Happy to provide any more info requested.

Alex Mandel (wildintellect) wrote :

Also just happened to me on a machine with "proposed" disabled. I fixed by downloading a newer version of network-manager via another computer/usb stick. My system seemed to show a newer version was available, and had I done the upgrade before turning it off it probably would have skipped this issue.

http://packages.ubuntu.com/trusty-updates/network-manager

I also grabbed libnm-glib4_0.9.8.8-0ubuntu7.3_amd64.deb and libnm-util2_0.9.8.8-0ubuntu7.3_amd64.deb as I wasn't sure if they were needed to fix the problem or required by network manager.

Tero Gusto (tero-gusto) wrote :

Todays regular updates via the Ubuntu "Software Updater" has fixed this issue.

summary: - network-manager crashes when using libnl-3-200-3.21.1-1ubuntu1
+ network-manager crashes when using libnl-3-200 3.2.21-1ubuntu1
Tom Dalton (tom-dalton) wrote :

#52 also fixed for me, now got both libnl-3-200 3.2.21-1ubuntu1 and network manager packages 0.9.8.8-0ubuntu7.3 and everything's working again.

Enrico Rosina (e-rosina) wrote :

Hello,

Thanks for the bug fix. Problem solved for me too, on Ubuntu 14.04.4 LTS.

After several days of manual "advanced" reboot, the problem disappeared simply upgrading with this command:

sudo apt-get install network-manager

... which upgraded to this version:

network-manager 0.9.8.8-0ubuntu7.3

Nice job, thank you again
Enrico
http://www.anisor.ch

I look after a relative's 14.04 xubuntu system. They live some way away so it wasn't possible for me to help them directly. They lost all internet connections, so a) couldn't look for solutions, and b) couldn't do any upgrades. They are capable of following detailed instructions, but don't really understand what they are doing. So when I spell out 'sudo' and they still type 'pseudo' I don't know why it hasn't worked. I'm no expert either, but I can usually sort things out after a time.

Suffice to say this was really hard to resolve. Once I found the askubuntu thread I realised that was probably the cause of their problem, and found this report, but that was only after many other checks and tests.

I should stress that they don't have 'proposed' updates enabled (nor even LTS updates), so their system was indeed crippled by an official release, and in a way that made determining a cause and fixing it rather hard. Not great.

I got them to try the solution manually enabling eth0, but this didn't seem to work. Eventually I managed to rather painfully describe the method of downgrading the 3 library files (which were very fortunately still in the archives) and that worked. Getting new .debs to them would have been fraught with difficulty. They have since done a full update and all is now well.

I've gone to a little length to explain this here because I think the ubuntu family of systems is great, and I encourage non-technical people to use it. Bad updates to network software are, however, very serious indeed, since the means of diagnosing and resolving such issues are inherently unavailable. This doesn't do much for ubuntu's reputation, or its wider use amongst less technical people.

Can I hope that 'lessons have been learned'?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments