| 2016-11-17 22:29:18 |
bugproxy |
bug |
|
|
added bug |
| 2016-11-17 22:29:20 |
bugproxy |
tags |
|
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin--- |
|
| 2016-11-17 22:29:21 |
bugproxy |
attachment added |
|
keepalived deb package with fix. https://bugs.launchpad.net/bugs/1642763/+attachment/4778870/+files/keepalived_1.2.19-1test_ppc64el.deb |
|
| 2016-11-17 22:29:23 |
bugproxy |
attachment added |
|
Upstream patch ported to keepalived 1.2.19 (found in Ubuntu 16.04) https://bugs.launchpad.net/bugs/1642763/+attachment/4778872/+files/Netlink-error-message-truncated-messages-V1.2.19.patch |
|
| 2016-11-17 22:29:25 |
bugproxy |
ubuntu: assignee |
|
Taco Screen team (taco-screen-team) |
|
| 2016-11-17 22:29:34 |
bugproxy |
affects |
ubuntu |
keepalived (Ubuntu) |
|
| 2016-11-17 22:41:39 |
Jon Grimm |
bug |
|
|
added subscriber Jon Grimm |
| 2016-11-17 22:46:07 |
Jon Grimm |
keepalived (Ubuntu): status |
New |
Triaged |
|
| 2016-11-17 22:46:40 |
Jon Grimm |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2016-11-18 08:22:22 |
Christian Ehrhardt |
keepalived (Ubuntu): importance |
Undecided |
Medium |
|
| 2016-11-29 16:37:05 |
Jon Grimm |
keepalived (Ubuntu): assignee |
Taco Screen team (taco-screen-team) |
Nish Aravamudan (nacc) |
|
| 2016-11-29 16:42:51 |
Mark Brown |
bug |
|
|
added subscriber Mark Brown |
| 2016-11-29 17:01:34 |
Nish Aravamudan |
nominated for series |
|
Ubuntu Yakkety |
|
| 2016-11-29 17:01:34 |
Nish Aravamudan |
bug task added |
|
keepalived (Ubuntu Yakkety) |
|
| 2016-11-29 17:01:34 |
Nish Aravamudan |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-29 17:01:34 |
Nish Aravamudan |
bug task added |
|
keepalived (Ubuntu Xenial) |
|
| 2016-11-29 17:34:03 |
Nish Aravamudan |
keepalived (Ubuntu): status |
Triaged |
In Progress |
|
| 2016-11-29 17:34:05 |
Nish Aravamudan |
keepalived (Ubuntu Xenial): status |
New |
In Progress |
|
| 2016-11-29 17:34:08 |
Nish Aravamudan |
keepalived (Ubuntu Yakkety): status |
New |
In Progress |
|
| 2016-11-29 17:34:10 |
Nish Aravamudan |
keepalived (Ubuntu Xenial): assignee |
|
Nish Aravamudan (nacc) |
|
| 2016-11-29 17:34:11 |
Nish Aravamudan |
keepalived (Ubuntu Yakkety): assignee |
|
Nish Aravamudan (nacc) |
|
| 2016-11-29 18:09:10 |
Nish Aravamudan |
keepalived (Ubuntu): status |
In Progress |
Fix Committed |
|
| 2016-11-29 18:50:34 |
Nish Aravamudan |
description |
== Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 ==
---Problem Description---
Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived).
Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
When we run, the configuration is failing to apply via keepalived.
The logs post the following:
Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!!
However, the device DOES exist. But the keepalived config just doesn't always deploy it.
ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters
This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1
---uname output---
Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power.
Machine Type = S822L
Machine Type = 8286-42A
---Steps to Reproduce---
Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP.
== Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 ==
The problem is fixed in this upstream commit:
https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345
On systems with a page size larger than 4096 keepalived may report:
"Netlink: error: message truncated" messages
This error was reported on a ppc64le in an OpenStack/Nutron environment.
Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg
buffer was too small causing messages to be truncated. The size of the read
buffer for the netlink socket should be based on page size however, it should
not exceed 8192. See the comment in the patch.
I tested the fix by creating 100 veth interfaces and verifying the errors
did not return.
Signed-off-by: David Wilder <dwilder@us.ibm.com>
Signed-off-by: Quentin Armitage <quentin@armitage.org.uk>
... |
[Impact]
* keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages.
* These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does.
[Test Case]
* Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed.
[Regression Potential]
* This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low.
---
== Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 ==
---Problem Description---
Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived).
Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
When we run, the configuration is failing to apply via keepalived.
The logs post the following:
Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!!
However, the device DOES exist. But the keepalived config just doesn't always deploy it.
ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters
This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1
---uname output---
Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power.
Machine Type = S822L
Machine Type = 8286-42A
---Steps to Reproduce---
Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP.
== Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 ==
The problem is fixed in this upstream commit:
https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345
On systems with a page size larger than 4096 keepalived may report:
"Netlink: error: message truncated" messages
This error was reported on a ppc64le in an OpenStack/Nutron environment.
Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg
buffer was too small causing messages to be truncated. The size of the read
buffer for the netlink socket should be based on page size however, it should
not exceed 8192. See the comment in the patch.
I tested the fix by creating 100 veth interfaces and verifying the errors
did not return.
Signed-off-by: David Wilder <dwilder@us.ibm.com>
Signed-off-by: Quentin Armitage <quentin@armitage.org.uk>
... |
|
| 2016-11-29 22:13:31 |
Nish Aravamudan |
description |
[Impact]
* keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages.
* These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does.
[Test Case]
* Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed.
[Regression Potential]
* This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low.
---
== Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 ==
---Problem Description---
Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived).
Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
When we run, the configuration is failing to apply via keepalived.
The logs post the following:
Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!!
However, the device DOES exist. But the keepalived config just doesn't always deploy it.
ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters
This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1
---uname output---
Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power.
Machine Type = S822L
Machine Type = 8286-42A
---Steps to Reproduce---
Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP.
== Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 ==
The problem is fixed in this upstream commit:
https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345
On systems with a page size larger than 4096 keepalived may report:
"Netlink: error: message truncated" messages
This error was reported on a ppc64le in an OpenStack/Nutron environment.
Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg
buffer was too small causing messages to be truncated. The size of the read
buffer for the netlink socket should be based on page size however, it should
not exceed 8192. See the comment in the patch.
I tested the fix by creating 100 veth interfaces and verifying the errors
did not return.
Signed-off-by: David Wilder <dwilder@us.ibm.com>
Signed-off-by: Quentin Armitage <quentin@armitage.org.uk>
... |
[Impact]
* keepalived on ppc64el (due to a large page size) experiences "Netlink: error: message truncated" messages.
* These Netlink truncations result in keepalived thinking that the the underlying device does not exist, even though it does.
[Test Case]
* Creating 100 veth interfaces ppc64el and verify if "Netlink: error: message truncated" errors are emitted. If so, the bug is present. If not, the bug is fixed.
[Regression Potential]
* This is code issue, fixed upstream, in the keepalived code when the system page size exceeds 4096. The upstream fix was backported to all releases and should only properly limit the size of the buffer used for netlink to at most 8192 on systems with a page size greater than 8192. I believe risk of regression is very low.
* Using the tests provided by David Wilder, I ran on both x86_64 and ppc64el LXD containers. Without the backported changes, I saw no issues on x86_64, and the reported issue on ppc64el (as expected, as a page size greater than 4K is required to see the buffer size issue). With the backported changes, both architectures show no issue with the provided testcase.
---
== Comment: #0 - Andrew Thorstensen - 2016-11-17 09:50:25 ==
---Problem Description---
Using Ubuntu 16.04 on ppc64le, we are building a 'neutron network node' using the VRRP configuration (built on keepalived).
Information on this OpenStack configuration can be found here: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
When we run, the configuration is failing to apply via keepalived.
The logs post the following:
Nov 17 02:58:31 p8test-lp1 Keepalived_vrrp[54542]: VRRP is trying to assign VIP to unknown qr-a5f5ba96-52 interface !!! go out and fix your conf !!!
However, the device DOES exist. But the keepalived config just doesn't always deploy it.
ii keepalived 1:1.2.19-1 ppc64el Failover and monitoring daemon for LVS clusters
This configuration sometimes works, but does sometimes fail on Ubuntu 16.04.1
---uname output---
Linux p8test-lp1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:38:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
This is a Power8 system with Ubuntu 16.04.1 installed. Though we see no indication that this is specific to Power.
Machine Type = S822L
Machine Type = 8286-42A
---Steps to Reproduce---
Install openstack. Run the network node in a VRRP HA configuration. Create a router and assign a global IP.
== Comment: #5 - David J. Wilder - 2016-11-17 15:58:04 ==
The problem is fixed in this upstream commit:
https://github.com/acassen/keepalived/commit/9f327bbf3e86def1055a106eda0633638bda0345
On systems with a page size larger than 4096 keepalived may report:
"Netlink: error: message truncated" messages
This error was reported on a ppc64le in an OpenStack/Nutron environment.
Ppc64le is using a 64k pages size. I found that keepalived's netlink recvmsg
buffer was too small causing messages to be truncated. The size of the read
buffer for the netlink socket should be based on page size however, it should
not exceed 8192. See the comment in the patch.
I tested the fix by creating 100 veth interfaces and verifying the errors
did not return.
Signed-off-by: David Wilder <dwilder@us.ibm.com>
Signed-off-by: Quentin Armitage <quentin@armitage.org.uk>
... |
|
| 2016-11-30 14:43:41 |
Ravi Gummadi |
bug |
|
|
added subscriber Ravi Gummadi |
| 2016-12-01 01:09:19 |
Launchpad Janitor |
keepalived (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2016-12-01 18:34:37 |
Brian Murray |
keepalived (Ubuntu Yakkety): status |
In Progress |
Fix Committed |
|
| 2016-12-01 18:34:38 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-12-01 18:34:40 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2016-12-01 18:34:46 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin--- |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin--- verification-needed |
|
| 2016-12-01 18:36:18 |
Brian Murray |
keepalived (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2016-12-01 19:56:46 |
Kyle L. Henderson |
bug |
|
|
added subscriber Kyle L. Henderson |
| 2016-12-02 16:59:40 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin--- verification-needed |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin16041 verification-done |
|
| 2016-12-12 22:06:24 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin16041 verification-done |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin16041 verification-done-xenial verification-needed |
|
| 2016-12-12 22:08:01 |
Launchpad Janitor |
keepalived (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2016-12-15 00:57:30 |
Nish Aravamudan |
tags |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin16041 verification-done-xenial verification-needed |
architecture-ppc64le bugnameltc-148871 severity-critical targetmilestone-inin16041 verification-done-xenial verification-done-yakkety |
|
| 2016-12-15 16:14:33 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2016-12-15 16:14:46 |
Launchpad Janitor |
keepalived (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2016-12-15 16:44:14 |
Nish Aravamudan |
keepalived (Ubuntu): assignee |
Nish Aravamudan (nacc) |
|
|
| 2016-12-15 16:44:16 |
Nish Aravamudan |
keepalived (Ubuntu Xenial): assignee |
Nish Aravamudan (nacc) |
|
|
| 2016-12-15 16:44:18 |
Nish Aravamudan |
keepalived (Ubuntu Yakkety): assignee |
Nish Aravamudan (nacc) |
|
|
