Activity log for bug #1720126

Date Who What changed Old value New value Message
2017-09-28 12:14:49 Pieter Malan bug added bug
2017-09-29 13:30:07 Jan Gutter bug added subscriber Jan Gutter
2017-09-29 13:31:03 Launchpad Janitor iproute2 (Ubuntu): status New Confirmed
2017-10-02 16:39:04 Joshua Powers bug added subscriber Ubuntu Server Team
2017-10-02 16:39:07 Joshua Powers iproute2 (Ubuntu): importance Undecided High
2017-10-10 09:17:16 Jan Gutter attachment added [PATCH] Fix "Message truncated" issue with many VF's https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4966551/+files/iproute2-fix-netlink-overflow.patch
2017-10-10 11:24:27 Jan Gutter bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=1380803
2017-10-10 11:24:27 Jan Gutter bug task added iproute2 (CentOS)
2017-10-10 12:28:55 Ubuntu Foundations Team Bug Bot tags patch
2017-10-10 12:29:05 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Review Team
2017-10-12 14:35:51 Monique van den Berg description I've ran into an issue when creating a large number of virtual functions on a SR-IOV capable device. ip link show reports a message truncated error: ip link show > /dev/null Message truncated Message truncated Message truncated A likely cause might be that when called in a system where the number of PCIe Virtual Functions are more than 30 for a given Physical Function, the netlink response is larger than 16K, meaning that a message is truncated. The issue is seen with Ubuntu14.04 and Ubuntu16.04. A possible solution for the issue is to increase the size of the receive buffer in libnetlink.c Additional information: ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2: Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2: Installed: 3.12.0-2ubuntu1 Version table: *** 3.12.0-2ubuntu1 0 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. All of the above are mitigated because this workaround is in upstream iproute2 and has been running on different distributions without repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2:   Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500         500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2:   Installed: 3.12.0-2ubuntu1   Version table:  *** 3.12.0-2ubuntu1 0         500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
2017-10-12 14:36:29 Monique van den Berg description [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. All of the above are mitigated because this workaround is in upstream iproute2 and has been running on different distributions without repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2:   Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500         500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2:   Installed: 3.12.0-2ubuntu1   Version table:  *** 3.12.0-2ubuntu1 0         500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. All of the above are mitigated because this workaround is in upstream iproute2 and has been running on different distributions without repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2:   Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500         500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2:   Installed: 3.12.0-2ubuntu1   Version table:  *** 3.12.0-2ubuntu1 0         500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
2017-10-12 14:37:34 Monique van den Berg description [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. All of the above are mitigated because this workaround is in upstream iproute2 and has been running on different distributions without repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2:   Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500         500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2:   Installed: 3.12.0-2ubuntu1   Version table:  *** 3.12.0-2ubuntu1 0         500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2:   Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500         500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2:   Installed: 3.12.0-2ubuntu1   Version table:  *** 3.12.0-2ubuntu1 0         500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
2017-10-12 14:50:03 Monique van den Berg bug added subscriber Ubuntu Sponsors Team
2017-10-12 15:23:28 Monique van den Berg removed subscriber Ubuntu Sponsors Team
2017-10-12 19:32:14 Nish Aravamudan nominated for series Ubuntu Xenial
2017-10-12 19:32:14 Nish Aravamudan bug task added iproute2 (Ubuntu Xenial)
2017-10-12 19:32:14 Nish Aravamudan nominated for series Ubuntu Zesty
2017-10-12 19:32:14 Nish Aravamudan bug task added iproute2 (Ubuntu Zesty)
2017-10-12 19:32:21 Nish Aravamudan iproute2 (Ubuntu): status Confirmed Fix Released
2017-10-12 19:32:25 Nish Aravamudan iproute2 (Ubuntu Xenial): status New Confirmed
2017-10-12 19:32:28 Nish Aravamudan iproute2 (Ubuntu Zesty): status New Fix Released
2017-10-12 19:32:32 Nish Aravamudan iproute2 (Ubuntu Xenial): importance Undecided High
2017-10-12 19:32:34 Nish Aravamudan iproute2 (Ubuntu): importance High Undecided
2017-10-12 19:32:55 Nish Aravamudan nominated for series Ubuntu Trusty
2017-10-12 19:32:55 Nish Aravamudan bug task added iproute2 (Ubuntu Trusty)
2017-10-12 19:53:27 Nish Aravamudan bug added subscriber Nish Aravamudan
2017-10-13 08:19:41 Monique van den Berg bug added subscriber Monique van den Berg
2017-10-13 12:45:01 Monique van den Berg attachment added Changelog https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4969908/+files/changelog
2017-10-13 12:46:26 Monique van den Berg attachment added Patch https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4969909/+files/99-increase-receive-buffer-size.patch
2017-10-13 12:47:50 Monique van den Berg bug added subscriber Ubuntu Sponsors Team
2017-10-13 12:52:07 Monique van den Berg attachment removed Changelog https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4969908/+files/changelog
2017-10-13 12:52:40 Monique van den Berg attachment added Changelog https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4969928/+files/changelog
2017-10-13 13:11:19 Monique van den Berg bug added subscriber Ubuntu Stable Release Updates Team
2017-10-13 14:31:13 Monique van den Berg attachment removed Changelog https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4969928/+files/changelog
2017-10-13 14:32:45 Monique van den Berg attachment added Change https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4970030/+files/changelog
2017-10-16 08:05:13 Monique van den Berg attachment added iproute2_3.12.0-2ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/xenial/+source/iproute2/+bug/1720126/+attachment/4972550/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-16 08:15:50 Monique van den Berg attachment removed iproute2_3.12.0-2ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4972550/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-16 08:17:37 Monique van den Berg attachment added iproute2_3.12.0-2ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4972558/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-16 08:19:07 Monique van den Berg attachment removed iproute2_3.12.0-2ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4972558/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-16 08:22:17 Monique van den Berg attachment added iproute2_3.12.0-2ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4972559/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-17 10:12:45 Monique van den Berg attachment added Xenial https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4973449/+files/iproute2_4.3.0-1ubuntu3.debdiff
2017-10-19 06:53:26 Monique van den Berg attachment added iproute2_3.12.0-2ubuntu1.1.debdiff (Trusty) https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4975629/+files/iproute2_3.12.0-2ubuntu1.debdiff
2017-10-19 06:53:32 Monique van den Berg attachment added iproute2_4.3.0-1ubuntu3.16.0.4.2.debdiff (Xenial) https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+attachment/4975630/+files/iproute2_4.3.0-1ubuntu3.debdiff
2017-10-19 23:57:00 Launchpad Janitor merge proposal linked https://code.launchpad.net/~nacc/ubuntu/+source/iproute2/+git/iproute2/+merge/332530
2017-10-19 23:57:41 Launchpad Janitor merge proposal linked https://code.launchpad.net/~nacc/ubuntu/+source/iproute2/+git/iproute2/+merge/332531
2017-10-25 12:58:29 Chris J Arges iproute2 (Ubuntu Xenial): status Confirmed Fix Committed
2017-10-25 12:58:33 Chris J Arges bug added subscriber SRU Verification
2017-10-25 12:58:39 Chris J Arges tags patch patch verification-needed verification-needed-xenial
2017-10-25 12:58:48 Chris J Arges removed subscriber Ubuntu Sponsors Team
2017-10-25 12:59:26 Chris J Arges iproute2 (Ubuntu Trusty): status New Fix Committed
2017-10-25 12:59:37 Chris J Arges tags patch verification-needed verification-needed-xenial patch verification-needed verification-needed-trusty verification-needed-xenial
2017-10-26 11:58:05 Jan Gutter tags patch verification-needed verification-needed-trusty verification-needed-xenial patch verification-done-trusty verification-needed verification-needed-xenial
2017-10-26 11:58:33 Monique van den Berg tags patch verification-done-trusty verification-needed verification-needed-xenial patch verification-done-xenial verification-needed verification-needed-trusty
2017-10-26 11:59:55 Monique van den Berg tags patch verification-done-xenial verification-needed verification-needed-trusty patch verification-done-trusty verification-done-xenial verification-needed
2017-10-26 12:04:00 Monique van den Berg tags patch verification-done-trusty verification-done-xenial verification-needed patch verification-done-trusty verification-done-xenial verification-needed verification-needed-trusty verification-needed-xenial
2017-10-26 12:04:52 Jan Gutter tags patch verification-done-trusty verification-done-xenial verification-needed verification-needed-trusty verification-needed-xenial patch verification-done verification-done-trusty verification-done-xenial verification-needed-trusty verification-needed-xenial
2017-10-26 12:05:29 Jan Gutter tags patch verification-done verification-done-trusty verification-done-xenial verification-needed-trusty verification-needed-xenial patch verification-done verification-done-trusty verification-done-xenial
2017-10-27 00:06:48 Bug Watch Updater iproute2 (CentOS): status Unknown In Progress
2017-10-27 00:06:48 Bug Watch Updater iproute2 (CentOS): importance Unknown High
2017-10-27 00:06:53 Bug Watch Updater bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=1086512
2017-11-02 16:00:39 Launchpad Janitor iproute2 (Ubuntu Xenial): status Fix Committed Fix Released
2017-11-02 16:00:42 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2017-11-09 09:38:37 Bug Watch Updater iproute2 (CentOS): status In Progress Fix Committed
2017-12-01 20:29:29 Mathew Hodson iproute2 (Ubuntu): importance Undecided High
2017-12-01 20:29:33 Mathew Hodson iproute2 (Ubuntu Trusty): importance Undecided High
2017-12-01 20:29:35 Mathew Hodson iproute2 (Ubuntu Zesty): importance Undecided High
2017-12-01 20:32:25 Mathew Hodson bug watch removed https://bugzilla.redhat.com/show_bug.cgi?id=1086512
2017-12-06 20:53:05 Launchpad Janitor iproute2 (Ubuntu Trusty): status Fix Committed Fix Released