[ip link] Message truncated error for large number of passthrough VFs

Bug #1720126 reported by Pieter Malan
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
iproute2 (CentOS)
Fix Committed
High
iproute2 (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned
Xenial
Fix Released
High
Unassigned
Zesty
Fix Released
High
Unassigned

Bug Description

[Impact]

When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer.

This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer.

Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch.

[Test Case]

# Set up 60 VF's on an SR-IOV device
ip link show > /dev/null

Observe the following:
Message truncated
Message truncated
Message truncated

[Regression Potential]

1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case.
2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny).
3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load.

[Other Info]

Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04

=====================================================================================================
Ubuntu16 system

stack@cluster04:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

stack@cluster04:~$ uname -r
4.4.0-93-generic

stack@cluster04:~$ apt-cache policy iproute2
iproute2:
  Installed: 4.3.0-1ubuntu3.16.04.1
Version table:
*** 4.3.0-1ubuntu3.16.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
=================================================================================================

Ubuntu14 system:
root@boomslang:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

root@boomslang:~# uname -r
4.4.0-96-generic

root@boomslang:~# apt-cache policy iproute2
iproute2:
  Installed: 3.12.0-2ubuntu1
  Version table:
 *** 3.12.0-2ubuntu1 0
        500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

Description of problem:
Running "ip link show" outputs Message Truncated several times and the resulting interface list is incomplete.

The issue occurs when the number of VF interfaces is over ~50.

This is impacting applications/scripts which rely on parsing the output of ip to ascertain the current network configuration (e.g. ovirt/vdsm)

Version-Release number of selected component (if applicable):
iproute-3.10.0-54.el7_2.1.x86_64

This appears to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1086512

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

This has been fixed upstream in either 4.5 or 4.6.

Rebuilding the SRPM from Fedora 25 (iproute-4.6.0-1.fc25.src.rpm) fixes the problem. But iproute-4.4.0-3.fc24.src.rpm from Fedora 24 does not.

Revision history for this message
In , Hangbin (hangbin-redhat-bugs) wrote :

commit 72b365e8e0fd5efe1d5c05d04c25950736635cfb
Author: Phil Sutter <email address hidden>
Date: Fri Mar 4 19:57:28 2016 +0100

    libnetlink: Double the dump buffer size

    There have been reports about 'ip addr' printing "Message truncated" on
    systems with large numbers of VFs. Although I haven't been able to get
    my hands on hardware suitable to reproduce this, increasing the dump
    buffer has been reported to resolve the issue. For want of a better
    idea, just double the buffer size to 32k.

    Feels like this opportunistic buffer size selection is rather
    workarounding a design flaw in libnetlink or maybe even the netlink
    protocol itself.

    Signed-off-by: Phil Sutter <email address hidden>

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

If there's any interest I could probably provide access to a system exhibiting the issue (Cisco B200 with VIC1340).

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

It seems that increasing the buffer size has simply pushed the problem further up. With ~95 VFs the message truncated warnings return.

On my Cisco UCS platform it's possible to go as high as 114 VFs, while other configurations support up to 223.

Revision history for this message
In , Phil (phil-redhat-bugs) wrote :

Hi Alan,

(In reply to Alan Griffiths from comment #5)
> It seems that increasing the buffer size has simply pushed the problem
> further up. With ~95 VFs the message truncated warnings return.
>
> On my Cisco UCS platform it's possible to go as high as 114 VFs, while other
> configurations support up to 223.

So you see the problem even with iproute-4.6.0-1.fc25?

Thanks, Phil

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

Yes, so the behaviour seems to be the same across the releases now.

I've tested: -

iproute-3.10.0-74 (EL7)
iproute-4.6.0-1 (FC25)
iproute-4.11.0-1 (FC26)

The test system has 2 PF. At 90 x VF all above versions work, at 91 x VF I start seeing message truncated errors.

Revision history for this message
Jan Gutter (jangutter) wrote :

A simple one-line patch seems to solve the issue for me:

Index: iproute2-4.3.0/lib/libnetlink.c
===================================================================
--- iproute2-4.3.0.orig/lib/libnetlink.c
+++ iproute2-4.3.0/lib/libnetlink.c
@@ -202,7 +202,7 @@ int rtnl_dump_filter_l(struct rtnl_handl
   .msg_iov = &iov,
   .msg_iovlen = 1,
  };
- char buf[16384];
+ char buf[65536];
  int dump_intr = 0;

  iov.iov_base = buf;

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in iproute2 (Ubuntu):
status: New → Confirmed
Joshua Powers (powersj)
Changed in iproute2 (Ubuntu):
importance: Undecided → High
Revision history for this message
Jan Gutter (jangutter) wrote :

I'm unfamiliar with where to submit the fix for this bug. Am I submitting the fix to the wrong forum?

Revision history for this message
Jan Gutter (jangutter) wrote :

This bug has already been fixed upstream in the following commit:

https://anonscm.debian.org/git/collab-maint/pkg-iproute.git/commit/?id=72b365e8e0fd5efe1d5c05d04c25950736635cfb

This commit happened between: tags debian/4.3.0-1 and debian/4.6.0-1

Revision history for this message
Jan Gutter (jangutter) wrote :

This fix has been pulled into CentOS 7.3 and later

Revision history for this message
In , Jan (jan-redhat-bugs) wrote :

Just a note:

There are theoretical and practical upper limits to the number of Virtual Functions per PF (256 from some sources)[1], so setting a static maximum buffer size is a justifiable option. That moves the burden over to ensuring that the response per VF doesn't grow in the future.

Libvirt had similar issues (it used libnl) until it enabled message peeking by default. Unfortunately the only way to solve this issue in all cases without performance penalty is to alter the kernel's message truncation handling mechanism to not free the skb if MSG_TRUNC occurred. [1]

It's possible to put in a retry on truncation, resizing the receive buffer to fit the response, BUT since that's two unrelated syscalls, it leaves me feeling a bit queasy.

[1] http://windowsitpro.com/systems-management/q-what-sr-iov
[2] http://man7.org/linux/man-pages/man2/recv.2.html

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "[PATCH] Fix "Message truncated" issue with many VF's" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
description: updated
description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

Is this then fixed in Ubuntu 17.04 which has iproute2 version 4.9.0-1ubuntu1?

Revision history for this message
Jan Gutter (jangutter) wrote :

Yes, I've confirmed it's fixed there.

I build it from the .dsc, and didn't see "Message truncated".

Then, just to make sure, I reverted the patch (changing the buf size to 16384) and the bug was back.

So at least there's been some use in real world of the patch, I hope...

What's the path of least resistance here? Backporting to iproute2's stable branch, or applying the patch downstream at Ubuntu?

Nish Aravamudan (nacc)
Changed in iproute2 (Ubuntu):
status: Confirmed → Fix Released
Changed in iproute2 (Ubuntu Xenial):
status: New → Confirmed
Changed in iproute2 (Ubuntu Zesty):
status: New → Fix Released
Changed in iproute2 (Ubuntu Xenial):
importance: Undecided → High
Changed in iproute2 (Ubuntu):
importance: High → Undecided
Revision history for this message
Nish Aravamudan (nacc) wrote :

@jangutter: I'm not sure why you built from source? I assume you actually tested the version in 17.04 and it worked.

The proper solution is to backport the Debian change to xenial and trusty, most likely. A git repository that can be used with `git-ubuntu` (sudo snap install --classic git-ubuntu`) is available here: https://code.launchpad.net/~usd-import-team/ubuntu/+source/iproute2/+git/iproute2

Feel free to propose MPs against ubuntu/xenial-devel and ubuntu/trusty-devel, or I can do it if you'd prefer.

Revision history for this message
Monique van den Berg (mvandenberg) wrote :
Revision history for this message
Monique van den Berg (mvandenberg) wrote :
Revision history for this message
Monique van den Berg (mvandenberg) wrote :

This is a debdiff for iproute2 applicable to 3.12.0-2ubuntu1. I built this in pbuilder
and it builds successfully, and I installed it, the patch works as intended.

Revision history for this message
Jan Gutter (jangutter) wrote :

@nacc I built from source to verify that the one-liner is directly responsible for fixing and breaking the issue (inherent paranoia). I did test with the binaries and they worked.

Apologies, I'm unfamiliar with the Ubuntu SRU process as you can probably see. What exactly is an "MP" and how would one go about to propose one?

I'm aware of the need of testing bugfixes like these, I'm not familiar with your release pipeline, however.

Revision history for this message
Monique van den Berg (mvandenberg) wrote :

This is a debdiff for iproute2 applicable to 4.3.0-1ubuntu3. I built this in pbuilder
and it builds successfully, and I installed it, the patch works as intended.

Revision history for this message
Nish Aravamudan (nacc) wrote : Re: [Bug 1720126] Re: [ip link] Message truncated error for large number of passthrough VFs

On 17.10.2017 [08:56:17 -0000], Jan Gutter wrote:
> @nacc I built from source to verify that the one-liner is directly
> responsible for fixing and breaking the issue (inherent paranoia). I did
> test with the binaries and they worked.

Ah ok, yeah -- I guess that's reasonable, and is a good preemptive test,
but given that it needs to be backported to a different release, feels a
bit like busywork (the same time could have been spent building it for
xenial :)

> Apologies, I'm unfamiliar with the Ubuntu SRU process as you can
> probably see. What exactly is an "MP" and how would one go about to
> propose one?

MP = Merge Proposal. Roughly like GitHub's Pull Requests (PR), except
less formal with Git (bzr I think is somewhat more first-class in
Launchpad).

> I'm aware of the need of testing bugfixes like these, I'm not familiar
> with your release pipeline, however.

http://www.justgohome.co.uk/blog/2017/07/developing-ubuntu-using-git.html

may help a bit. Roughly:

$ sudo snap install --classic git-ubuntu
$ git ubuntu clone iproute2
$ cd iproute2
$ ... make and commit changes
$ git ubuntu submit

Now, the issue is that middle bit, where you have to know a bit about
source packaging. That is, simply cherry-picking the upstream/Debian fix
is not quite right, as you need to change it into a Quilt patch and then
insert a changelog entry.

We are currently developing a fix for that so you can just do a

$ git ubuntu remote add debian
$ git cherry-pick 72b365e83
$ git ubuntu build-source

And it should spit out a commit that has the quiltify'd and
changelogify'd result that you can use as a base or to submit.

Feel free to find me on IRC if you want some more pointers.

In the meanwhile, I'll try and look at Monique's debdiffs this week.

-Nish

Revision history for this message
Nish Aravamudan (nacc) wrote :

@mvandenberg,

Couple of nits in your debdiffs.

1) changelogs are targetting UNRELEASED, please update to xenial and trusty respectively.

2) Is there a reason your fix is different than the fix upstream/Debian? Does Debian need your version instead?

3) Please use appropriate DEP3 headers (dpkg-source --commit creates a template patch with them): http://dep.debian.net/deps/dep3/, especially Origin (not Origin/Author).

4) The actual author appears to be Phil Sutter, not Jan Gutter, although Jan authored the patch in this bug. Consider using dep3changelog once you have a quilt patch to generate the changelog in the standard format (with an attribution to Phil or Jan as you decide).

5) The trusty version number seems off, it should be 3.12.0-2ubuntu1.1. Please take a look at https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging.

Can you fix these up and reupload the debdiffs?

Thanks,
Nish

Revision history for this message
Monique van den Berg (mvandenberg) wrote :
Revision history for this message
Monique van den Berg (mvandenberg) wrote :
Revision history for this message
Jan Gutter (jangutter) wrote :

@nacc

Thanks so much for the explanation. I also found https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#Merge_Proposals_and_Reviewing that details a bit more of the internal processes. As relative outsiders to the Ubuntu process, I'd appreciate it very much if you could handle that part for Monique's patches. I can be on hand to answer technical questions if required.

Regarding the buffer size choice, it's very arbitrary as Phil said. I'm pretty sure we came to the same conclusion independently (libvirt and libnl had very similar issues) and the workaround is obvious. 32k seems to work for 64 VF's (our test case), but breaks with 128 VF's. Not a lot of machines can handle 128 concurrent VF's. I typed 64k "just because". libvirt+libnl allow message peeking. However, iproute2 uses netlink directly. So, implementing a similar idea would require an entirely new receive codepath with all the fun of finding out where new exception paths occur: something to be done on tip and not suitable for backport without thorough vetting.

I'm sure it'll save a lot of time once the kinks have been worked out of the automation, backports are quite the double-edged sword.

Revision history for this message
Nish Aravamudan (nacc) wrote :

On 19.10.2017 [09:35:19 -0000], Jan Gutter wrote:
> @nacc
>
> Thanks so much for the explanation. I also found
> https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#Merge_Proposals_and_Reviewing
> that details a bit more of the internal processes. As relative outsiders
> to the Ubuntu process, I'd appreciate it very much if you could handle
> that part for Monique's patches. I can be on hand to answer technical
> questions if required.

And to be clear, the MP based workflow for the Git trees is brand new
and experimental :)

I'm happy to integrate the updated debdiffs (I'll reply to those
comments directly).

> Regarding the buffer size choice, it's very arbitrary as Phil said. I'm
> pretty sure we came to the same conclusion independently (libvirt and
> libnl had very similar issues) and the workaround is obvious. 32k seems
> to work for 64 VF's (our test case), but breaks with 128 VF's. Not a lot
> of machines can handle 128 concurrent VF's. I typed 64k "just because".
> libvirt+libnl allow message peeking. However, iproute2 uses netlink
> directly. So, implementing a similar idea would require an entirely new
> receive codepath with all the fun of finding out where new exception
> paths occur: something to be done on tip and not suitable for backport
> without thorough vetting.

Absolutely. My concern is the upstream code is at 32k as is Artful. I'm
hesitant to backport something different (64k) to X and T without also
ensuring Artful gets it (and BB when it opens), and presumably also
fixing it upstream.

So I see two routes forward:

1) File an upstream issue to request they bump to 64k, as you note 32k
is insufficient for 128 VFs. Link to that issue in this bug and we'll
fix AA, X and T with the suggested change (presuming upstream acks it).

2) Backport the upstream change as-is to X and T (AA already has the
necessary fix). This will be faster, of course, but does mean the 128 VF
case is broken. Given that it is less likely to be hit in the field,
perhaps that is ok -- and in the meanwhile, upstream can work on a
proper fix which, when available, we can backport accordingly (or decide
at that point, in any case).

I prefer 2), because I do not like diverging from upstream (or at least
not without an upstream bug report). If you and Monique are ok with 2),
I can update the debdiffs before sponsoring them.

> I'm sure it'll save a lot of time once the kinks have been worked out of
> the automation, backports are quite the double-edged sword.

Definitely :)

Revision history for this message
Nish Aravamudan (nacc) wrote :

I have set up two MPs with the adjustments (it looks like Monique's latest debdiffs followed path 2) from my previous comment already) to the DEP3 headers that I think make the most sense. Please take a look at them and if you approve the changes I will upload them.

I note also there are a number of statically sized buffers in the iproute2 code -- is there any concern about other buffers overflowing?

Revision history for this message
Jan Gutter (jangutter) wrote :

I concur with option 2), unnecessary deviation will just cause confusion.

Regarding the other buffer sizes, the last time I looked they were mostly OK. The issue reared its head in this particular case because the netlink message that previously had a pretty constant per-netdev response size suddenly had the ability to balloon with "no warning". A number of workarounds exist (i.e. you have to explicitly ask for the VF info), but, in this case we actually want the VF info and iproute2 was just unprepared for the size of it.

I guess the core issue is that it's entirely possible for the kernel to add extra netlink attributes to any query response, iproute2 makes the assumption that the queries it's making is not necessarily going to explode with gigabytes of new annotations and 16k will easily fit any current real-world system. A pragmatic approach would probably be to handle the "Message Truncated" path with a dynamically sized buffer as an exceptional case.

Any fix in iproute2 that "properly" addresses this issue has to be carefully vetted. Who knows how many inherent races will get exposed if the ip command doubles in execution time.

Revision history for this message
Jan Gutter (jangutter) wrote :

I had a look at the two proposals and could not spot any obvious mistakes:

- the correct upstream git commit has been cherry-picked
- I don't have any objections to attribution or log messages

Thanks again for shepherding this one through!

Revision history for this message
Nish Aravamudan (nacc) wrote :

On 20.10.2017 [07:09:00 -0000], Jan Gutter wrote:
> I concur with option 2), unnecessary deviation will just cause
> confusion.

Thank you for confirming that!

> Regarding the other buffer sizes, the last time I looked they were
> mostly OK. The issue reared its head in this particular case because the
> netlink message that previously had a pretty constant per-netdev
> response size suddenly had the ability to balloon with "no warning". A
> number of workarounds exist (i.e. you have to explicitly ask for the VF
> info), but, in this case we actually want the VF info and iproute2 was
> just unprepared for the size of it.

Ok, that's good to hear.

> I guess the core issue is that it's entirely possible for the kernel to
> add extra netlink attributes to any query response, iproute2 makes the
> assumption that the queries it's making is not necessarily going to
> explode with gigabytes of new annotations and 16k will easily fit any
> current real-world system. A pragmatic approach would probably be to
> handle the "Message Truncated" path with a dynamically sized buffer as
> an exceptional case.

Yep, I can see how iproute2 itself has to move in lockstep with the
kernel, which also means older iproute2 that can run on newer kernels
needs periodic updates like this one.

> Any fix in iproute2 that "properly" addresses this issue has to be
> carefully vetted. Who knows how many inherent races will get exposed if
> the ip command doubles in execution time.

Yep :) I'm fine with eventually doubling the buffer again statically if
that is the conclusion upstream reaches. My guess is that is the
simplest solution.

Revision history for this message
Nish Aravamudan (nacc) wrote :

On 20.10.2017 [07:18:15 -0000], Jan Gutter wrote:
> I had a look at the two proposals and could not spot any obvious
> mistakes:
>
> - the correct upstream git commit has been cherry-picked
> - I don't have any objections to attribution or log messages
>
> Thanks again for shepherding this one through!

You're welcome, I'll upload them both today.

Thank you and Monique for filing the SRU template properly!

-Nish

Revision history for this message
Nish Aravamudan (nacc) wrote :

On 20.10.2017 [08:24:37 -0700], Nish Aravamudan wrote:
> On 20.10.2017 [07:18:15 -0000], Jan Gutter wrote:
> > I had a look at the two proposals and could not spot any obvious
> > mistakes:
> >
> > - the correct upstream git commit has been cherry-picked
> > - I don't have any objections to attribution or log messages
> >
> > Thanks again for shepherding this one through!
>
> You're welcome, I'll upload them both today.
>
> Thank you and Monique for filing the SRU template properly!

I have sponsored both packages. They will need to be approved by an SRU
team member (probably next week) before they appear in
{trusty,xenial}-proposed and then they will need testing as built in
proposed, with a minimum bake time of 7 days in proposed.

Thank you and Monique for your contribution to Ubuntu!

-Nish

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Pieter, or anyone else affected,

Accepted iproute2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iproute2/4.3.0-1ubuntu3.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in iproute2 (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in iproute2 (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed-trusty
Revision history for this message
Chris J Arges (arges) wrote :

Hello Pieter, or anyone else affected,

Accepted iproute2 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iproute2/3.12.0-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Jan Gutter (jangutter)
tags: added: verification-done-trusty
removed: verification-needed-trusty
tags: added: verification-done-xenial verification-needed-trusty
removed: verification-done-trusty verification-needed-xenial
Revision history for this message
Jan Gutter (jangutter) wrote :

I tested on Trusty: iproute2_3.12.0-2ubuntu1.1_amd64.deb fixed the bug.

Revision history for this message
Monique van den Berg (mvandenberg) wrote :

I tested the iproute2_4.3.0-1ubuntu3.16.04.2_amd64.deb (xenial) package and this fixed the bug for me.

tags: added: verification-done-trusty
removed: verification-needed-trusty
tags: added: verification-needed-trusty verification-needed-xenial
Jan Gutter (jangutter)
tags: added: verification-done
removed: verification-needed
tags: removed: verification-needed-trusty verification-needed-xenial
Changed in iproute2 (CentOS):
importance: Unknown → High
status: Unknown → In Progress
Revision history for this message
Jan Gutter (jangutter) wrote :

@arges

Hi, would you need any more testing from us?

Revision history for this message
Nish Aravamudan (nacc) wrote : Re: [Bug 1720126] Re: [ip link] Message truncated error for large number of passthrough VFs

On Wed, Nov 1, 2017 at 7:15 AM, Jan Gutter <email address hidden> wrote:
> @arges
>
> Hi, would you need any more testing from us?

Nope, your part is done. All SRUs have to bake for 7 days in -proposed
to shake out regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iproute2 - 4.3.0-1ubuntu3.16.04.2

---------------
iproute2 (4.3.0-1ubuntu3.16.04.2) xenial; urgency=medium

  * debian/patches/99-increase-receive-buffer-size.patch: Fix "Message
    truncated" issue with many VF's. Thanks to Phil Sutter
    <email address hidden>. Closes LP: #1720126.

 -- Monique van den Berg <email address hidden> Tue, 17 Oct 2017 11:33:13 +0200

Changed in iproute2 (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for iproute2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
In , Phil (phil-redhat-bugs) wrote :

A real fix for this issue has been accepted upstream and was applied to net-next branch:

commit 2d34851cd341f0e1b3fc17ca3e6e874229f3a1f8
Author: Hangbin Liu <email address hidden>
Date: Thu Oct 26 09:41:46 2017 +0800

    lib/libnetlink: re malloc buff if size is not enough

commit 86bf43c7c2fdc33d7c021b4a1add1c8facbca51c
Author: Hangbin Liu <email address hidden>
Date: Thu Oct 26 09:41:47 2017 +0800

    lib/libnetlink: update rtnl_talk to support malloc buff at run time

Alan, are you able to compile iproute2 by yourself? If so, could you please
give upstream's 'net-next' branch a try? It should solve the issue you are
seeing.

Thanks, Phil

Revision history for this message
Nish Aravamudan (nacc) wrote :

If someone is interested in the proper upstream fixes, please file a new bug and we will need to fix bionic and prepare a distinct SRU for the existing releases.

Revision history for this message
In , Alan (alan-redhat-bugs) wrote :

I can confirm it now works with 112 VF. This is the most my hardware will support.

Thanks,

Alan

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

The trusty upload seems to have an armhf autopkgtest regression - looks really strange and I see it failed once already. I think it might be good to run the autopkgtest suite with no triggers to see if it's an unrelated regression or not. Could anyone take a look at that?

http://autopkgtest.ubuntu.com/packages/a/autopkgtest/trusty/armhf

Revision history for this message
Nish Aravamudan (nacc) wrote : Re: [Bug 1720126] Re: [ip link] Message truncated error for large number of passthrough VFs

I'm guessing this might be a quirk of the armhf runners:

test_tree_garbage (__main__.ChrootRunner)
copied source tree contains only expected files ... : Operation not permitted
cp: cannot create special file ‘/tmp/tmp36ee0r77/chroot//dev/null’:
Operation not permitted
cp: cannot create special file ‘/tmp/tmplo89rot6/chroot//dev/null’:
Operation not permitted
cp: cannot create special file ‘/tmp/tmpsd8_r783/chroot//dev/null’:
Operation not permitted
ERROR
test_tree_norestrictions_nobuild_fail_on_exit (__main__.ChrootRunner)
source tree, no build, no restrictions, test fails with non-zero ... ERROR
test_tree_norestrictions_nobuild_fail_on_stderr (__main__.ChrootRunner)
source tree, no build, no restrictions, test fails with stderr ... ERROR
test_tree_norestrictions_nobuild_fail_on_stderr_and_exit (__main__.ChrootRunner)
source tree, no build, no restrictions, test fails with stderr+exit ... ERROR

I'll see if it might need some restrictions (or if the runner env changed)?

-Nish

Changed in iproute2 (CentOS):
status: In Progress → Fix Committed
Mathew Hodson (mhodson)
Changed in iproute2 (Ubuntu):
importance: Undecided → High
Changed in iproute2 (Ubuntu Trusty):
importance: Undecided → High
Changed in iproute2 (Ubuntu Zesty):
importance: Undecided → High
Revision history for this message
Robie Basak (racb) wrote :

I don't think the Trusty armhf autopkgtest test is likely to be a regression. It doesn't appear to have ever passed before and the error looks associated with the test runner rather than iproute2 itself. Given the nature of the patch, it seems unlikely to me to be caused by this update. I think it's OK to treat this situation as if we have no operational autopkgtest for Trusty armhf. And if that were the case, we'd be releasing the SRU anyway. So I'm releasing it to free up the next iproute2 SRU we have lined up.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iproute2 - 3.12.0-2ubuntu1.1

---------------
iproute2 (3.12.0-2ubuntu1.1) trusty; urgency=medium

  * debian/patches/99-increase-receive-buffer-size.patch: Fix "Message
    truncated" issue with many VF's. Thanks to Phil Sutter
    <email address hidden>. Closes LP: #1720126.

 -- Monique van den Berg <email address hidden> Mon, 16 Oct 2017 09:26:42 +0200

Changed in iproute2 (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.