libnl: fail to bind() netlink sockets

Bug #1511735 reported by Nicolas Dichtel on 2015-10-30
162
This bug affects 27 people
Affects Status Importance Assigned to Milestone
libnl3 (Debian)
Fix Released
Unknown
libnl3 (Ubuntu)
Medium
Unassigned
Trusty
Medium
Adam Conrad

Bug Description

[Triage Notes]

The proposed fix for this bug in libnl3 caused a regression in trusty-proposed, tracked in bug 1539634. The regression is caused by a latent bug in Network Manager. We expect to fix this by landing a fix for the bug in Network Manager in trusty-updates at the same time as this fix.

[Impact]

 * Applications in Trusty using libnl-3-200 which frequently open and
   close netlink sockets can easily fail when attempting to bind the
   local socket. The problem happens when libnl choose a port id
   already used by another application and subsequently libnl fails
   instead of trying another port id.

   The original bug was discovered when attempting to start a virtual
   machine under libvirt, which is a user of this library.

 * Backporting fixes from upstream release fixes a real bug in the
   current version of the library in Trusty. The alternative is for all
   applications to manually manage their local port allocation, or as
   upstream has accepted allowing libnl to attempt to try for other
   local ports.

 * All patches applied are already accepted upstream and newer Ubuntu
   releases are not affected.

[Test Case]

 * On a Trusty 14.04 system
   1. sudo apt-get install libnl-3-200 libnl-3-dev libnl-3-dev \
                           libnl-genl-3-dev libnl-route-3-200 \
                           make gcc build-essential libnl1
   2. download and unpack attachment: lp_1511735_test.tar
   3. Run testcases:

    % if ./example.sh; then echo "libnl OK"; else echo "libnl FAILED"; fi
    gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
    set manually the local port to 6975 (pid: 6974)
    local port has been set by the libnl to 6975 (pid: 6975)
    ERROR: genl_connect(): Object exists (local port: 6975, pid: 6975)
    libnl FAILED
    % python libnl3-test-rh1249158.py a b c d
    ulimit(NOFILE) = (2048, 4096)
    Test: PID=6978
    TEST (a)...
    Traceback (most recent call last):
      File "libnl3-test-rh1249158.py", line 226, in <module>
        locals()["TEST_" + arg]()
      File "libnl3-test-rh1249158.py", line 140, in TEST_a
        sk = nl_get_socket()
      File "libnl3-test-rh1249158.py", line 115, in nl_get_socket
        raise IOError(-err, _nl_geterror())
    OSError: [Errno 6] b'Unspecific failure'

   4. After applying the updated packages:

    % if ./example.sh; then echo "libnl OK"; else echo "libnl FAILED"; fi
    gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
    set manually the local port to 11295 (pid: 11294)
    local port has been set by the libnl to 2894081055 (pid: 11295)
    libnl OK
    % python libnl3-test-rh1249158.py a b c d
    ulimit(NOFILE) = (2048, 4096)
    Test: PID=11296
    TEST (a)...
    ...done
    TEST (b)...
    ...done
    TEST (c)...
    ...done
    TEST (d)...
    ...done

[Regression Potential]

 * There are quite a few high profile packages that depend on this package,
   notably libvirt and network-manager. The complete list is here:

   # on Trusty
    % apt-rdepends -r libnl-3-200 | head -n 33
    libnl-3-200
      Reverse Depends: batctl (>= 2013.4.0-2)
      Reverse Depends: bmon (>= 1:3.1-1)
      Reverse Depends: crda (>= 1.1.2-1ubuntu2)
      Reverse Depends: hostapd (>= 1:2.1-0ubuntu1)
      Reverse Depends: ipvsadm (>= 1:1.26-2ubuntu1)
      Reverse Depends: iw (>= 3.4-1)
      Reverse Depends: keepalived (>= 1:1.2.7-1ubuntu1)
      Reverse Depends: kismet (>= 2013.03.R1b-3)
      Reverse Depends: knemo (>= 0.7.6-2)
      Reverse Depends: libfsobasics3 (>= 0.12.0-4)
      Reverse Depends: libnetcf1 (>= 1:0.2.3-4ubuntu1)
      Reverse Depends: libnl-3-200-dbg (= 3.2.21-1)
      Reverse Depends: libnl-3-dev (= 3.2.21-1)
      Reverse Depends: libnl-cli-3-200 (= 3.2.21-1)
      Reverse Depends: libnl-genl-3-200 (= 3.2.21-1)
      Reverse Depends: libnl-nf-3-200 (= 3.2.21-1)
      Reverse Depends: libnl-route-3-200 (= 3.2.21-1)
      Reverse Depends: libnl-utils (>= 3.2.21-1)
      Reverse Depends: libnss-gw-name (>= 0.3-2)
      Reverse Depends: libvirt-bin (>= 1.2.2-0ubuntu13)
      Reverse Depends: libvirt0 (>= 1.2.2-0ubuntu13)
      Reverse Depends: lowpan-test-tools (>= 0.3-1)
      Reverse Depends: lowpan-tools (>= 0.3-1)
      Reverse Depends: neard (>= 0.11-1)
      Reverse Depends: neard-tools (>= 0.11-1)
      Reverse Depends: network-manager (>= 0.9.8.8-0ubuntu7)
      Reverse Depends: ntrack-module-libnl-0 (>= 016-1.2ubuntu2)
      Reverse Depends: plainbox-provider-resource-generic (>= 0.3-1)
      Reverse Depends: powertop (>= 2.5-1ubuntu1)
      Reverse Depends: quota (>= 4.01-3)
      Reverse Depends: sssd-common (>= 1.11.5-1ubuntu3)
      Reverse Depends: wpasupplicant (>= 2.1-0ubuntu1)

 * This patch does change the default behavoir when asking libnl-3-200
   to generate local ports. Applications (or libraries) may already
   have retry code in-place and it's not clear if those applications
   would break.

[Original Description]
The following upstream patches are needed in order to avoid failures when binding a netlink socket:

1f734a8f892a lib/socket: randomize the generated local port
http://git.infradead.org/users/tgr/libnl.git/commitdiff/1f734a8f892a

4dd5fdd0af2c lib/socket: retry generate local port in nl_connect on ADDRINUSE
http://git.infradead.org/users/tgr/libnl.git/commitdiff/4dd5fdd0af2c

027157898708 lib/socket: don't fail if no more local ports can be assigned in nl_socket_alloc
http://git.infradead.org/users/tgr/libnl.git/commitdiff/027157898708

0fd510b3673f lib/socket: use proper typed constant UINT32_MAX for uint32_t typed port
http://git.infradead.org/users/tgr/libnl.git/commitdiff/0fd510b3673f

Without these patches, an application which opens and closes regularly netlink sockets can easily fails to bind them.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1511735

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
no longer affects: linux (Ubuntu)

Any news?

Robie Basak (racb) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Which Ubuntu releases and libnl3 package versions are affected, please? Which upstream release first included these patches? And please could you provide your failure use cases so that we can decide on importance and the trade-offs in maintenance, regression risk, etc, in deciding whether to cherry-pick these fixes over waiting to update to a new upstream release in a future Ubuntu release?

Once you have answered these questions, please change the bug status back to New. Thanks!

Changed in libnl3 (Ubuntu):
status: New → Incomplete

It affects ubuntu-14.04, libnl version 3.2.21 (http://packages.ubuntu.com/trusty/libnl-3-200)
The first libnl upstream release which includes these patches is 3.2.25.

The problem happens when the libnl choose a port id already used by another application. The libnl fails instead of trying another port id.

Please find enclosed a script to reproduce the problem:
$ ./example.sh
gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
set manually the local port to 12145 (pid: 12144)
local port has been set by the libnl to 12145 (pid: 12145)
genl_connect(): Object exists (local port: 12145, pid: 12145)

The first instance of example sets manually the port id to pid+1.
The second instance of example let the libnl calculate automatically the port id. It chooses by default the pid, which is already used.

To summarize, if an application on the system chooses manually a port id (because it does not use the libnl or because it sets it manually), it may conflict with applications that use libnl and prevent them to work properly.

Changed in libnl3 (Ubuntu):
status: Incomplete → New
Robie Basak (racb) wrote :

Thank you for your response. It looks like this is fixed in Wily then, which is on 3.2.26, so this bug affects 14.04 only?

I'm concerned that this changes library behaviour. How can we be sure that no existing users of libnl will not be regressed by this change in behaviour if we backport this to 14.04?

> To summarize, if an application on the system chooses manually a port id (because it does not use the libnl or because it sets it manually), it may conflict with applications that use libnl and prevent them to work properly.

Thanks, this is useful. But can you take us through to actual user impact of this bug, please? I'm sure you have a concrete case for wanting this, but so far this explanation is hypothetical. In what concrete case on 14.04 does an application manually chose a port id that then causes a conflict and causes a problem for a user?

Robie Basak (racb) wrote :

For now I'm marking this Fix Released as it is fixed in Wily, and opening a task for Trusty (which depending on what we decide may end up being Won't Fix).

Changed in libnl3 (Ubuntu):
status: New → Fix Released

>Thank you for your response. It looks like this is fixed in Wily then, which is on 3.2.26, so this bug affects 14.04 only?

Yes.

>I'm concerned that this changes library behaviour. How can we be sure that no existing users of libnl will not be regressed by this change in behaviour if we backport this to 14.04?

This does not change the library behavior. It fixes a real bug. Without this fix, the user has to manage the port id itself. And the only consequence is to increase the probability to show the bug in another application. It's a domino effect. There is two options:
1/ patching all applications that use the libnl
2/ fixing the libnl

>Thanks, this is useful. But can you take us through to actual user impact of this bug, please? I'm sure you have a concrete case for wanting this, but so far this explanation is hypothetical. In what concrete case on 14.04 does an application manually chose a port id that then causes a conflict and causes a problem for a user?

Yes, I have a concrete case, but it's with a non FOSS application. It's not hypothetical, we hit the bug regularly. And as explained above, trying to fix the impacted applications is a wasted effort because it increases the probability to show the bug in another applications.
FWIW, these patches have been backported by redhat, you can found more info about this bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1249158

tags: added: 6wind
Robie Basak (racb) wrote :

Ryan, please could you take a look at this? I've taken a brief look at the patches and I follow what they're achieving and why, but I'm not confident that I follow all possible edge cases introduced by these patches.

If you're satisfied then please could you prepare an SRU? Thanks!

Changed in libnl3 (Ubuntu Trusty):
assignee: nobody → Ryan Harper (raharper)
Ryan Harper (raharper) wrote :

Initial look at the patches results in the following assessment:

The first 2 patches apply fine to the version included in Ubuntu Trusty

027157898708 lib/socket: don't fail if no more local ports can be assigned in nl_socket_alloc
http://git.infradead.org/users/tgr/libnl.git/commitdiff/027157898708

0fd510b3673f lib/socket: use proper typed constant UINT32_MAX for uint32_t typed port
http://git.infradead.org/users/tgr/libnl.git/commitdiff/0fd510b3673f

But at

4dd5fdd0af2c lib/socket: retry generate local port in nl_connect on ADDRINUSE
http://git.infradead.org/users/tgr/libnl.git/commitdiff/4dd5fdd0af2c

This requires some additional plumbing changes in libnl3 that's not present in the version. In particular,
the capabilities framework (utils.h/utils.c NL_CAPABILITY_*) and the individual linker scripts introduced here:

http://git.infradead.org/users/tgr/libnl.git/commitdiff/e7d57da0ddad9097962ebfec4f5202b756ed93c9

It seems non-trivial (I've not studied all of libnl3 library code yet) and (somewhat risky) to backport the capabilities which may change behavior of the library (destabilizing existing users) and normally beyond the scope of a bug fix that's SRU'able.

We'd need to retain the logic of the fix, but port it to the level of libnl3 we have. I'd be happy to pick this back up if the we get a reworked patch 3 and 4 that don't require backporting additional infrastructure into libnl3.

Additional feedback, the test-case included was somewhat awkward in that it orphans the first process without any cleanup. Test-cases should strive to exit non-zero on failure for easier integration into test frameworks. It would be good to see about integrating the test into the existing tests of the project, but that's a nice to have.

The enclosed tar file contains the updated test script and the 4 patches that can be applied cleanly over the ubuntu libnl3-3.2.21.

Before the patches:
$ if ./example.sh ; then echo success ; else echo failure ; fi
gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
set manually the local port to 15754 (pid: 15753)
local port has been set by the libnl to 15754 (pid: 15754)
genl_connect(): Object exists (local port: 15754, pid: 15754)
failure

After the patches:
$ if ./example.sh ; then echo success ; else echo failure ; fi
gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
set manually the local port to 15768 (pid: 15767)
local port has been set by the libnl to 432029080 (pid: 15768)
success

Ryan Harper (raharper) wrote :

Hi,

Thanks for the quick update. I've applied the 4 patches and built a new test package with the changes. The example test does indeed fail on the current version and passes on the new version as well.

I went looking for any other libnl3 tests we might run to catch any possible regression that might occur when accepting the patches I looked at the RH bug you referenced and found another testcase ( used to expose the issue. I ran that on the updated package and it passes tests a, b and c, but not d.

I examined the patch applied and it included more fixes, specifically:

Subject: [PATCH 09/10] socket: clear port when unable to generate local port

which the 'd' test which attempts to test by allocate more than 1024 local ports. Should a fix for that also be included?

Yes you're right. Here is an updated tar file with all patches (the first four patches are the same).
The following patches are added:
- 9614acf4c435 lib/nl: preserve s_local if nl_connect() fails
  https://github.com/thom311/libnl/commit/9614acf4c435
- f78c3e82398a socket: clear port when unable to generate local port
  https://github.com/thom311/libnl/commit/f78c3e82398a
- 96e1e5bdc2e8 socket: add fallback for nl_connect() by trying to bind to unspecified local port
  https://github.com/thom311/libnl/commit/96e1e5bdc2e8
- eaa75b7c7d3e socket: fix assertion in nl_connect() when all ports are already in use
  https://github.com/thom311/libnl/commit/eaa75b7c7d3e

With this new series:
root@ubuntu1404:~# python ./libnl3-test-rh1249158.py a b c d
ulimit(NOFILE) = (2048, 4096)
Test: PID=25856
TEST (a)...
...done
TEST (b)...
...done
TEST (c)...
...done
TEST (d)...
...done
root@ubuntu1404:~# cd libnl-bp/
root@ubuntu1404:~/libnl-bp# if ./example.sh ; then echo success ; else echo failure ; fi
gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
set manually the local port to 25867 (pid: 25866)
local port has been set by the libnl to 2793432331 (pid: 25867)
success

Ryan Harper (raharper) wrote :

Attaching debdiff with changes needed to resolve the bug.

Ryan Harper (raharper) wrote :

Tarball of the test-case used to confirm failure and fix.

description: updated

Thank you.

Note that the last three patches are not included in libnl 3.2.26, which is the version of ubuntu 16.04 /Xenial (https://launchpad.net/ubuntu/+source/libnl3/3.2.26-1).
Should I make a new series for this version or will the libnl 3.2.27 be included in Xenial?

On Wed, Dec 16, 2015 at 11:29 AM, Nicolas Dichtel <nicolas.dichtel@6wind.com
> wrote:

> Thank you.
>
> Note that the last three patches are not included in libnl 3.2.26, which
> is the version of ubuntu 16.04 /Xenial (
> https://launchpad.net/ubuntu/+source/libnl3/3.2.26-1).
>

Thank you for mentioning this.

> Should I make a new series for this version or will the libnl 3.2.27 be
> included in Xenial?
>

We should file a bug in Debian against libnl3 indicating the issue that's
still present in libnl3 3.2.26 (clearing port when fail to generate local
port) and then rebasing/attaching the 3 patches that fix the issue.

On the Ubuntu side, we can upload libnl 3.2.27 from upstream and then sync
with Debian when Unstable moves to 3.2.27.

> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1511735
>
> Title:
> libnl: fail to bind() netlink sockets
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1511735/+subscriptions
>

The bug report on debian has been created: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808213

Changed in libnl3 (Debian):
status: Unknown → New

Ryan, do you know when the ubuntu-14.04 will have the new libnl3 package?

Ryan Harper (raharper) wrote :

Nicolas,

Thanks for the ping. I'm looking for a sponsor to review the update. If approved, then it'll get uploaded and continue through the SRU process.

Thanks again for following up on this.

Changed in libnl3 (Ubuntu):
importance: Undecided → Medium
Changed in libnl3 (Ubuntu Trusty):
importance: Undecided → Medium
status: New → Triaged
tags: added: patch

Hello Nicolas, or anyone else affected,

Accepted libnl3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libnl3/3.2.21-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libnl3 (Ubuntu Trusty):
status: Triaged → Fix Committed
tags: added: verification-needed
Jakub Velimsky (velimsky) wrote :

Hi,

after automatic update of packages libnl-3-200, libnl-route-3-200, and libnl-genl-3-200 to version 3.2.21-1ubuntu1
in my 64-bit Ubuntu 14.04-LTS the network-manager ceased working.

/var/log/syslog shows:
init: network-manager main process (1057) killed by SEGV signal
init: network-manager main process ended, respawning
init: network-manager main process (1065) killed by SEGV signal
init: network-manager respawning too fast, stopped

No internet connection thereafter, wire or wireless. Had to manually downgrade back to version 3.2.21-1.

If you need more details, please let me know what to report.

Jakub

Ryan Harper (raharper) wrote :

Hi Jakub,

Thanks for testing the new package. It would be helpful to collect any core files and backtraces.

apport-collect -p network-manager 1511735

Should collect the info for this bug report.

Tests are OK for us.

Here is a screen shot of the manuel test:
root@ubuntu1404:~# apt-get install libnl-3-200/trusty-proposed libnl-3-dev/trusty-proposed libnl-genl-3-200/trusty-proposed libnl-genl-3-dev/trusty-proposed libnl-nf-3-200/trusty-proposed libnl-nf-3-dev/trusty-proposed libnl-route-3-200/trusty-proposed libnl-route-3-dev/trusty-proposed
[snip]
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-genl-3-dev amd64 3.2.21-1ubuntu1 [9,488 B]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-genl-3-200 amd64 3.2.21-1ubuntu1 [10.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-nf-3-dev amd64 3.2.21-1ubuntu1 [26.2 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-route-3-dev amd64 3.2.21-1ubuntu1 [103 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-3-dev amd64 3.2.21-1ubuntu1 [69.5 kB]
Get:6 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-nf-3-200 amd64 3.2.21-1ubuntu1 [27.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-route-3-200 amd64 3.2.21-1ubuntu1 [96.1 kB]
Get:8 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-3-200 amd64 3.2.21-1ubuntu1 [45.8 kB]
[snip]
root@ubuntu1404:~# python ./libnl3-test-rh1249158.py a b c d
ulimit(NOFILE) = (2048, 4096)
Test: PID=18863
TEST (a)...
...done
TEST (b)...
...done
TEST (c)...
...done
TEST (d)...
...done
root@ubuntu1404:~# cd libnl-bp/
root@ubuntu1404:~/libnl-bp# if ./example.sh ; then echo success ; else echo failure ; fi
gcc -o example -I/usr/include/libnl3 example.c -lnl-3 -lnl-genl-3
set manually the local port to 18895 (pid: 18894)
local port has been set by the libnl to 4060105167 (pid: 18895)
success
root@ubuntu1404:~/libnl-bp#

We also have an internal test suite for our product (which heavily use the libnl) and no problem was detected.

tags: added: verification-done
removed: verification-needed
Ryan Harper (raharper) wrote :

Thanks Nicolas. I'm working on fixing up NetworkManager; we'll need that fix as well to go out at the sametime.

Ryan Harper (raharper) on 2016-01-29
tags: added: verification-failed
removed: verification-done

Sure. Let me know if you need some help.

Ryan Harper (raharper) wrote :

I've filed https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1539634 to track the updated needed there.
I attached a debdeff in the NM bug after applying the patches from comment #24.

I tested the updated NM with proposed libnl-3 and it works as expected.

1. Download 14.04.03 Desktop iso, amd64
2. Launched in a VM, run from iso
3. In terminal, enable proposed
4. sudo apt-get install libnl-3-200 libnl-genl-3-200 libnl-route-3-200
5. sudo restart network-manager
# note the nm-applet disappears as NM has now crashed
6. dmesg to confirm network-manager crash
7. sudo dpkg --install gir1.2-networkmanager-1.0_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib4_0.9.8.8-0ubuntu7.3_amd64.deb libnm-glib-vpn1_0.9.8.8-0ubuntu7.3_amd64.deb libnm-util2_0.9.8.8-0ubuntu7.3_amd64.deb network-manager_0.9.8.8-0ubuntu7.3_amd64.deb
8. sudo start network-manager
# nm-applet restored, network connectivity restored.

Nick (nbar) wrote :

Dear Ryan,
I am fighting with the same problem. So I also have to apply this patch to get my network working again.
Can you tell me wurde I can find these deb-packages?
Thank you very much in advance!
Nico

Ryan Harper (raharper) wrote :

Hi Nick,

We've not built a new NetworkManager package with the fix. For now, you'll need to downgrade your libnl-3 packages:

% apt-get install libnl-3-200=3.2.21-1 libnl-genl-3-200=3.2.21-1 libnl-route-3-200=3.2.21-1

And you may need to apply a hold until a fix for NetworkManager can be released.

% sudo apt-mark hold libnl-3-200 libnl-genl-3-200 libnl-route-3-200

And after the fix:

% sudo apt-mark auto libnl-3-200 libnl-genl-3-200 libnl-route-3-200

Nick (nbar) wrote :

Hi Ryan,

thank you very much for your quick help. That worked fine!

Best regards, Nick

D. Charles Pyle (dcharlespyle) wrote :

Same problems as above hit me today after an update on my Ubuntu 14.04.3 LTS machine. After applying updates, networking ceased to function.

Downgraded libnl* and now networking is working again.

Luckily I was able to do the downgrade by first using recovery options at boot from GRuB. From there I enabled networking, and then I resumed booting. Networking lasted long enough to do the downgrade (per instructions above), followed by a reboot of the system.

All is working now, and I am waiting for the coming network-manager update to fix the problem.

Changed in libnl3 (Debian):
status: New → Fix Released
Robie Basak (racb) on 2016-02-01
description: updated
JTProg (p-mail-h) on 2016-02-02
Changed in libnl3 (Ubuntu Trusty):
status: Fix Committed → Fix Released
Ryan Harper (raharper) wrote :

I've retagged this as verification-needed since we've had one regression (NetworkManager, a bug there) and we'd like to request other users of libnl to test out the version included in proposed so we can ensure we're flushing out other applications that may be exposed due to the behavior changed in this update.

tags: added: verification-needed
removed: verification-failed
Ryan Harper (raharper) wrote :

This has not been released; this version is only available in trusty-proposed. It's not been verified at this point.

Changed in libnl3 (Ubuntu Trusty):
status: Fix Released → Fix Committed
Galen Thurber (godfree2) wrote :

this is pure sabotage? expunge the traitors!!!

description: updated
Mathew Hodson (mathew-hodson) wrote :

There should probably be a new upload of libnl3 with

Breaks: network-manager (<< 0.9.8.8-0ubuntu7.3)

so users can't put their system into a broken state when both packages are released to -updates.

Ryan Harper (raharper) wrote :
Download full text (3.2 KiB)

I've updated the debdiff to include a Breaks: network-manager (<< 0.9.8.8-0ubuntu7.3) to ensure that users are forced to use the newer network-manager (ie, this libnl won't install unless user has already updated their network-manager) Thanks Mathew for the suggestion.

I've built and tested this in the following way:

1. deploy trusty 14.04
2. apt-get install network-manager
3. download the libnl3 packages (3.2.21-1ubuntu2)
4. dpkg --install libnl-3-200_3.2.21-1ubuntu2_amd64.deb libnl-genl-3-200_3.2.21-1ubuntu2_amd64.deb libnl-route-3-200_3.2.21-1ubuntu2_amd64.deb

This results in the following (successful blocking of upgrade):

dpkg: regarding libnl-3-200_3.2.21-1ubuntu2_amd64.deb containing libnl-3-200:amd64:
 libnl-3-200:amd64 breaks network-manager (<< 0.9.8.8-0ubuntu7.3)
  network-manager (version 0.9.8.8-0ubuntu7.2) is present and installed.

dpkg: error processing archive libnl-3-200_3.2.21-1ubuntu2_amd64.deb (--install):
 installing libnl-3-200:amd64 would break network-manager, and
 deconfiguration is not permitted (--auto-deconfigure might help)
(Reading database ... 37060 files and directories currently installed.)
Preparing to unpack libnl-genl-3-200_3.2.21-1ubuntu2_amd64.deb ...
Unpacking libnl-genl-3-200:amd64 (3.2.21-1ubuntu2) over (3.2.21-1) ...
Preparing to unpack libnl-route-3-200_3.2.21-1ubuntu2_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.21-1ubuntu2) over (3.2.21-1) ...
dpkg: dependency problems prevent configuration of libnl-genl-3-200:amd64:
 libnl-genl-3-200:amd64 depends on libnl-3-200 (= 3.2.21-1ubuntu2); however:
  Version of libnl-3-200:amd64 on system is 3.2.21-1.

dpkg: error processing package libnl-genl-3-200:amd64 (--install):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnl-route-3-200:amd64:
 libnl-route-3-200:amd64 depends on libnl-3-200 (= 3.2.21-1ubuntu2); however:
  Version of libnl-3-200:amd64 on system is 3.2.21-1.

dpkg: error processing package libnl-route-3-200:amd64 (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 libnl-3-200_3.2.21-1ubuntu2_amd64.deb
 libnl-genl-3-200:amd64
 libnl-route-3-200:amd64

After installing network-manager 0.9.8.8-0ubuntu7.3, I can successfully re-run the libnl3 install command:

dpkg --install libnl-3-200_3.2.21-1ubuntu2_amd64.deb libnl-genl-3-200_3.2.21-1ubuntu2_amd64.deb libnl-route-3-200_3.2.21-1ubuntu2_amd64.deb
(Reading database ... 41021 files and directories currently installed.)
Preparing to unpack libnl-3-200_3.2.21-1ubuntu2_amd64.deb ...
Unpacking libnl-3-200:amd64 (3.2.21-1ubuntu2) over (3.2.21-1) ...
Selecting previously unselected package libnl-genl-3-200:amd64.
Preparing to unpack libnl-genl-3-200_3.2.21-1ubuntu2_amd64.deb ...
Unpacking libnl-genl-3-200:amd64 (3.2.21-1ubuntu2) ...
Selecting previously unselected package libnl-route-3-200:amd64.
Preparing to unpack libnl-route-3-200_3.2.21-1ubuntu2_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.21-1ubuntu2) ...
Setting up libnl-3-200:amd64 (3.2.21-1ubuntu2) ...
Setting up libnl-genl-3-200:amd64 (3.2.21-1ubuntu2) ...
Setting up libnl-route-3-200:amd64 (3.2.21-1ubuntu2) ...
Processing tri...

Read more...

Ryan Harper (raharper) wrote :

v3 debdiff
  - Removed unneeded spaces in debian/control Breaks entry
  - Fixed up libnl-3-200.symbols file to tag private symbols as optional (removes dpkg-gensymbols warning)
  - Removed invalid use of Closes() for Debian bugs; instead use LP: #<bug num>

Ryan Harper (raharper) wrote :

I've done some more verification tests by installing as many of the reverse depends on libnl to see if any other package implodes like Network-Manager did.

Sniff Check OK: Tested by install and invoke binaries.
batctl
bmon
ipvsadm
keepalived
libfsobasics3
libnetcf1
libnl-3-200-dbg
libnl-3-dev
libnl-cli-3-200
libnl-genl-3-200
libnl-nf-3-200
libnl-route-3-200
libnl-utils
libnss-gw-name
libvirt-bin
libvirt0
network-manager (with what's in proposed)
powertop
quota

SKIPPED: Packages that depended on wireless devices for testing
crda
hostapd
iw
kismet
knemo
lowpan-test-tools
lowpan-tools
neard
neard-tools
ntrack-module-libnl-0
plainbox-provider-resource-generic
sssd-common
wpasupplicant

Changed in libnl3 (Ubuntu Trusty):
assignee: Ryan Harper (raharper) → nobody
Ryan Harper (raharper) on 2016-02-17
tags: added: verification-done
removed: verification-needed
Pavlic (pavlic148) on 2016-04-01
Changed in libnl3 (Ubuntu Trusty):
status: Fix Committed → Fix Released
Mathew Hodson (mathew-hodson) wrote :

The previous upload is still in -proposed. There also needs to be a new upload from lp_1511735_libnl3_v3.diff

Changed in libnl3 (Ubuntu Trusty):
status: Fix Released → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnl3 - 3.2.21-1ubuntu1

---------------
libnl3 (3.2.21-1ubuntu1) trusty; urgency=low

  * Apply backported upstream patches (Closes: LP: #1511735)
    - lib-socket-use-proper-typed-constant-UINT32_MAX.patch
    - lib-socket-don-t-fail-if-no-more-local-ports-can-be.patch
    - lib-socket-retry-generate-local-port-in-nl_connect.patch
    - lib-socket-randomize-the-generated-local-port.patch
    - lib-nl-preserve-s_local-if-nl_connect-fails.patch
    - socket-clear-port-when-unable-to-generate-local-port.patch
    - socket-add-fallback-for-nl_connect-by-trying-to-bind.patch
    - socket-fix-assertion-in-nl_connect-when-all-ports-ar.patch
  * Updated symbol file libnl-3-200.symbols

 -- Ryan Harper <email address hidden> Thu, 28 Jan 2016 10:19:00 -0600

Changed in libnl3 (Ubuntu Trusty):
status: Triaged → Fix Released

The verification of the Stable Release Update for libnl3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Galen Thurber (godfree2) wrote :

yet another bullshit fix

Daniel Holz (daniel-holz91) wrote :

This update broke Network-Manager on my system. Used these steps to manually downgrade libnl to get network connections working again.

http://askubuntu.com/questions/727127/last-upgrade-crashes-network-manager-no-internet-connection-no-applet

Dave Musicant (musicant) wrote :

I'm in the same boat, this update just broke Network Manager on my system.

Dario Bertini (berdario) wrote :

There's a new bug, I guess that pitti might pay more attention to this over there, since this old one is now considered fixed (incorrently, but alas)

https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1581535

Martin Pitt (pitti) wrote :

I removed the update from trusty-proposed as it causes a regression, in bug 1581535. In the best case this was just a partial upgrade and affected users did not get the corresponding network-manager update from bug 1539634 - in that case please reupload an ubuntu2 with a corresponding Breaks: to n-m. Otherwise we'll have another regression. I asked in bug 1581535 about the versions.

Changed in libnl3 (Ubuntu Trusty):
assignee: nobody → Ryan Harper (raharper)
assignee: Ryan Harper (raharper) → nobody
status: Fix Released → In Progress
Martin Pitt (pitti) on 2016-05-13
Changed in libnl3 (Ubuntu Trusty):
assignee: nobody → Ryan Harper (raharper)
Ryan Harper (raharper) wrote :

Not sure why the proposed pocket didn't have the 1ubuntu2 that included the breaks. When we encountered this, I asked that the 1ubuntu2 of libnl3 be uploaded and staged such that everyone gets the libnl3 with the Breaks n-m first.

ChaosPredictor (kuzi81) wrote :

please release the fix in location that accessible by phone (not like Mega - in the place that I now I can't download this application)

Adam Conrad (adconrad) wrote :

For what it's worth, I reverted libnl3 to a known good state in trusty-updates (so it's safe to upgrade without concern), and staged a corrected version with Breaks in trusty-proposed.

Changed in libnl3 (Ubuntu Trusty):
assignee: Ryan Harper (raharper) → Adam Conrad (adconrad)
Dave Musicant (musicant) wrote :

I should read my post before submitting them. The above fix got me back _online_.

sokolov (daniel-sokolov) wrote :

Thank you, Dave, for posting that link!
https://bugs.launchpad.net/ubuntu/+source/libnl3/+bug/1581535/comments/13
I can confirm that the described procedure works. The tricky thing was to find a USB key. I generally don't use them for reasons of security protocol.

Full Name (dataplane) wrote :

It affect my 14.04.1 system WITHOUT trusty-proposed enabled. FYI: kernel 3.19.0-59-generic.

Changed in libnl3 (Ubuntu Trusty):
status: In Progress → Fix Committed
Fox (fox-nowdqk) wrote :

The instructions I found that worked and were these easiest to follow were at http://askubuntu.com/questions/727127/last-upgrade-crashes-network-manager-no-internet-connection-no-applet specifically the very first answer. I tried step 1 but that did not work, so I did step 2 and then rebooted and then my network connections worked again.

Saji Nediyanchath (saji89) wrote :

Looks like the fix is in place, I had downloaded the following packages manually from packages.ubuntu.com, trusty-updates:

- libnl-3-200_3.2.21-1ubuntu1.1_amd64.deb
- libnl-genl-3-200_3.2.21-1ubuntu1.1_amd64.deb
- libnl-route-3-200_3.2.21-1ubuntu1.1_amd64.deb

Has to install it using sudo dpkg -i, as Ubuntu software center was not allowing it due to the dependency relation between these packages. Anyhow after installing these packages and doing a
sudo service network-manager restart

enabled my network manager, previously with the broken packages, it was resulting in network-manager crashing, if I tried to restart the service, but after these packages were installed the network-manager worked normally.

Though technically a apt-get upgrade and reboot should work, it won't work; as without the network-manager there is no easy way to connect to the internet, so for most users they'll have to download the packages manually and install like I had to do.

Michel-Ekimia (michel.ekimia) wrote :

@Saji : the best way for newbies with ethernet was :

- sudo dhclient in Terminal

- Launch the update manager, install updates and reboot

Martin Pitt (pitti) wrote :

Can you please test libnl3 3.2.21-1ubuntu3 in trusty-proposed instead? This should now correctly enforce the upgrade to the -proposed network-manager as well, and work. Thanks!

tags: added: verification-needed
removed: verification-done
Mathew Hodson (mathew-hodson) wrote :

I downgraded libnl3 to 3.2.21-1 and network-manager to 0.9.8.8-0ubuntu7.1 in order to test the upgrade procedure.

When I hold network-manager, libnl3 doesn't get installed.

$ sudo apt-mark hold network-manager
network-manager set on hold.
$ sudo apt install libnl-3-200
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help resolve the situation:

The following packages have unmet dependencies:
 libnl-3-200 : Breaks: network-manager (< 0.9.8.8-0ubuntu7.3~) but 0.9.8.8-0ubuntu7.1 is to be installed
E: Unable to correct problems, you have held broken packages.

Without the hold, libnl3 properly brings in the new version of network-manager.

$ sudo apt-mark unhold network-manager
Canceled hold on network-manager.
$ sudo apt install libnl-3-200
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libnl-genl-3-200 libnl-route-3-200 network-manager
Recommended packages:
  modemmanager
The following packages will be upgraded:
  libnl-3-200 libnl-genl-3-200 libnl-route-3-200 network-manager
4 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Need to get 10.2 kB/652 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ca.archive.ubuntu.com/ubuntu/ trusty-proposed/main libnl-genl-3-200 amd64 3.2.21-1ubuntu3 [10.2 kB]
Fetched 10.2 kB in 0s (31.1 kB/s)
(Reading database ... 170772 files and directories currently installed.)
Preparing to unpack .../libnl-genl-3-200_3.2.21-1ubuntu3_amd64.deb ...
Unpacking libnl-genl-3-200:amd64 (3.2.21-1ubuntu3) over (3.2.21-1) ...
Preparing to unpack .../libnl-route-3-200_3.2.21-1ubuntu3_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.21-1ubuntu3) over (3.2.21-1) ...
Preparing to unpack .../network-manager_0.9.8.8-0ubuntu7.3_amd64.deb ...
Unpacking network-manager (0.9.8.8-0ubuntu7.3) over (0.9.8.8-0ubuntu7.1) ...
Preparing to unpack .../libnl-3-200_3.2.21-1ubuntu3_amd64.deb ...
Unpacking libnl-3-200:amd64 (3.2.21-1ubuntu3) over (3.2.21-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up libnl-3-200:amd64 (3.2.21-1ubuntu3) ...
Setting up libnl-genl-3-200:amd64 (3.2.21-1ubuntu3) ...
Setting up libnl-route-3-200:amd64 (3.2.21-1ubuntu3) ...
Setting up network-manager (0.9.8.8-0ubuntu7.3) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...

tags: added: verification-done
removed: verification-needed
Martin Pitt (pitti) wrote :
Changed in libnl3 (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.