Activity log for bug #1722584

Date Who What changed Old value New value Message
2017-10-10 16:23:57 Trygve Vea bug added bug
2017-10-10 18:35:08 Brian Haley neutron: status New Confirmed
2017-10-10 18:35:13 Brian Haley neutron: importance Undecided Medium
2017-10-10 18:42:26 OpenStack Infra neutron: status Confirmed In Progress
2017-10-10 18:42:26 OpenStack Infra neutron: assignee Brian Haley (brian-haley)
2017-10-10 19:39:50 Brian Haley bug added subscriber Brian Haley
2017-10-11 11:37:27 Tore Anderson bug added subscriber Tore Anderson
2017-12-17 10:52:57 OpenStack Infra neutron: status In Progress Fix Released
2018-01-25 01:00:45 OpenStack Infra tags in-stable-pike
2019-05-07 11:28:22 OpenStack Infra tags in-stable-pike in-stable-pike in-stable-queens
2019-05-08 21:26:57 OpenStack Infra tags in-stable-pike in-stable-queens in-stable-pike in-stable-queens in-stable-rocky
2019-05-21 10:06:07 OpenStack Infra tags in-stable-pike in-stable-queens in-stable-rocky in-stable-pike in-stable-queens in-stable-rocky in-stable-stein
2019-06-10 09:30:46 Edward Hope-Morley description We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network) [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] None expected [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network)
2019-06-10 09:30:57 Edward Hope-Morley description [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] None expected [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network) [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] None expected [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network)
2019-06-10 09:31:08 Edward Hope-Morley bug task added cloud-archive
2019-06-10 09:31:20 Edward Hope-Morley nominated for series cloud-archive/rocky
2019-06-10 09:31:20 Edward Hope-Morley bug task added cloud-archive/rocky
2019-06-10 09:31:20 Edward Hope-Morley nominated for series cloud-archive/queens
2019-06-10 09:31:20 Edward Hope-Morley bug task added cloud-archive/queens
2019-06-10 09:31:20 Edward Hope-Morley nominated for series cloud-archive/stein
2019-06-10 09:31:20 Edward Hope-Morley bug task added cloud-archive/stein
2019-06-10 09:31:30 Edward Hope-Morley nominated for series cloud-archive/train
2019-06-10 09:31:30 Edward Hope-Morley bug task added cloud-archive/train
2019-06-10 09:31:52 Edward Hope-Morley summary Return traffic from metadata service may get dropped by hypervisor due to wrong checksum [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum
2019-06-10 14:22:06 Corey Bryant nominated for series cloud-archive/ocata
2019-06-10 14:22:06 Corey Bryant bug task added cloud-archive/ocata
2019-06-10 14:22:14 Corey Bryant bug task deleted cloud-archive/ocata
2019-06-10 15:28:08 Corey Bryant cloud-archive/train: status New Triaged
2019-06-10 15:28:10 Corey Bryant cloud-archive/stein: status New Triaged
2019-06-10 15:28:13 Corey Bryant cloud-archive/rocky: status New Triaged
2019-06-10 15:28:15 Corey Bryant cloud-archive/queens: status New Triaged
2019-06-10 15:29:34 Corey Bryant bug task added neutron (Ubuntu)
2019-06-10 15:29:49 Corey Bryant nominated for series Ubuntu Eoan
2019-06-10 15:29:49 Corey Bryant bug task added neutron (Ubuntu Eoan)
2019-06-10 15:29:49 Corey Bryant nominated for series Ubuntu Bionic
2019-06-10 15:29:49 Corey Bryant bug task added neutron (Ubuntu Bionic)
2019-06-10 15:29:49 Corey Bryant nominated for series Ubuntu Cosmic
2019-06-10 15:29:49 Corey Bryant bug task added neutron (Ubuntu Cosmic)
2019-06-10 15:29:49 Corey Bryant nominated for series Ubuntu Disco
2019-06-10 15:29:49 Corey Bryant bug task added neutron (Ubuntu Disco)
2019-06-10 15:30:01 Corey Bryant neutron (Ubuntu Bionic): importance Undecided High
2019-06-10 15:30:01 Corey Bryant neutron (Ubuntu Bionic): status New Triaged
2019-06-10 15:30:14 Corey Bryant neutron (Ubuntu Cosmic): importance Undecided High
2019-06-10 15:30:14 Corey Bryant neutron (Ubuntu Cosmic): status New Triaged
2019-06-10 15:30:30 Corey Bryant neutron (Ubuntu Disco): importance Undecided High
2019-06-10 15:30:30 Corey Bryant neutron (Ubuntu Disco): status New Triaged
2019-06-10 15:30:46 Corey Bryant neutron (Ubuntu Eoan): importance Undecided High
2019-06-10 15:30:46 Corey Bryant neutron (Ubuntu Eoan): status New Triaged
2019-06-10 18:28:37 Corey Bryant description [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] None expected [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network) [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] The original issue is no longer fixed once this patch is reverted. [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network)
2019-06-13 10:31:16 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed
2019-06-17 14:00:52 Corey Bryant neutron (Ubuntu Eoan): status Triaged Fix Released
2019-06-17 14:01:06 Corey Bryant cloud-archive/train: status Triaged Fix Released
2019-06-17 19:21:28 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2019-06-19 14:35:54 Edward Hope-Morley description [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] The original issue is no longer fixed once this patch is reverted. [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network) [Impact] Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives. [Test Case] * deploy openstack (>= queens) * create router/network/instance (dvr=false,l3ha=false) * go to router ns on neutron-gateway and check that the following returns nothing sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill' [Regression Potential] Backporting the revert patch will mean that routers created with this patch will no longer have a checksum rule added for metadata tcp packets. The original patch added a rule that turned out not to be the fix for the root issue and was subsequently found to cause problems with kernels < 4.19 since it was never intended for gso tcp packets to have their checksum verified using this type of rule. So, removal of this rule (by addition of the revert patch) is not intended to change behaviour at all. The only potential side-effect is that rules that were already created will not be cleaned up (until node reboot or router recreate) and in an L3HA config you could end up with some router instances having the rule and some not depending on whether they were created before or after the patch was included. [Other Info] This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required. ----------------------------------------------------------------------------- We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages). Instance routes traffic to 169.254.169.254 to it's default gateway. Default gateway is an OpenStack router in a namespace on a networking node. - Traffic gets sent from the guest, - to the router, - iptables routes it to the metadata proxy service, - response packet gets routed back, leaving the namespace - Hypervisor gets the packet in - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge Based on the following bug https://bugs.launchpad.net/openstack-ansible/+bug/1483603, we found that adding the following iptable rule in the router namespace made this work again: 'iptables -t mangle -I POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill' (NOTE: The rule from the 1st comment to the bug did solve access to the metadata service, but the lack of precision introduced other problems with the network)
2019-06-25 23:36:23 Brian Murray neutron (Ubuntu Cosmic): status Triaged Fix Committed
2019-06-25 23:36:28 Brian Murray bug added subscriber SRU Verification
2019-06-25 23:36:33 Brian Murray tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-cosmic
2019-06-26 13:08:28 Corey Bryant cloud-archive/rocky: status Triaged Fix Committed
2019-06-26 13:08:31 Corey Bryant tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-cosmic in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-cosmic verification-rocky-needed
2019-07-03 16:42:08 Brian Murray neutron (Ubuntu Bionic): status Triaged Fix Committed
2019-07-03 16:42:19 Brian Murray tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-cosmic verification-rocky-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed
2019-07-03 18:56:13 Brian Murray neutron (Ubuntu Disco): status Triaged Fix Committed
2019-07-03 18:56:20 Brian Murray tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed
2019-07-04 06:12:30 James Page cloud-archive/stein: status Triaged Fix Committed
2019-07-04 06:12:34 James Page tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed
2019-07-10 14:45:35 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed verification-stein-needed
2019-07-11 08:41:16 Ɓukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2019-07-11 08:51:24 Launchpad Janitor neutron (Ubuntu Disco): status Fix Committed Fix Released
2019-07-11 11:23:27 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed verification-stein-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-done verification-stein-needed
2019-07-11 11:25:02 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-done verification-stein-needed in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed verification-stein-done
2019-07-11 12:50:15 Corey Bryant cloud-archive/stein: status Fix Committed Fix Released
2019-07-11 13:18:39 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-disco verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed verification-stein-done in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-rocky-needed verification-stein-done
2019-07-11 15:22:41 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-rocky-needed verification-stein-done in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-rocky-done verification-stein-done
2019-07-11 15:57:07 Corey Bryant cloud-archive/queens: status Triaged Fix Committed
2019-07-11 15:57:10 Corey Bryant tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-rocky-done verification-stein-done in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-queens-needed verification-rocky-done verification-stein-done
2019-07-12 08:33:30 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-cosmic verification-done-disco verification-needed verification-needed-bionic verification-queens-needed verification-rocky-done verification-stein-done in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-bionic verification-done-cosmic verification-done-disco verification-needed verification-queens-needed verification-rocky-done verification-stein-done
2019-07-12 11:38:19 Edward Hope-Morley tags in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done-bionic verification-done-cosmic verification-done-disco verification-needed verification-queens-needed verification-rocky-done verification-stein-done in-stable-pike in-stable-queens in-stable-rocky in-stable-stein sts sts-sru-needed verification-done verification-done-bionic verification-done-cosmic verification-done-disco verification-queens-done verification-rocky-done verification-stein-done
2019-07-15 07:20:33 Launchpad Janitor neutron (Ubuntu Cosmic): status Fix Committed Fix Released
2019-07-15 08:17:44 Launchpad Janitor neutron (Ubuntu Bionic): status Fix Committed Fix Released
2019-07-15 14:08:00 Corey Bryant cloud-archive/rocky: status Fix Committed Fix Released
2019-07-15 14:14:57 Corey Bryant cloud-archive/queens: status Fix Committed Fix Released