| 2019-06-07 17:15:41 |
David Ames |
bug |
|
|
added bug |
| 2019-06-07 17:17:39 |
Jeff Hillman |
tags |
|
cpe-onsite |
|
| 2019-06-07 17:24:18 |
Jeff Hillman |
bug |
|
|
added subscriber Jeff Hillman |
| 2019-06-07 18:21:39 |
Ryan Beisner |
bug |
|
|
added subscriber Ryan Beisner |
| 2019-06-07 19:33:47 |
Brian Haley |
bug |
|
|
added subscriber Brian Haley |
| 2019-06-10 16:38:51 |
David Ames |
marked as duplicate |
|
1722584 |
|
| 2019-06-13 15:59:41 |
James Page |
bug task added |
|
charm-neutron-openvswitch |
|
| 2019-06-13 15:59:51 |
James Page |
removed duplicate marker |
1722584 |
|
|
| 2019-06-13 15:59:58 |
James Page |
neutron: status |
New |
Incomplete |
|
| 2019-06-14 14:51:05 |
David Ames |
neutron: status |
Incomplete |
New |
|
| 2019-06-14 15:27:24 |
David Ames |
summary |
Checksum drop of metadata traffic on isolated provider networks |
Checksum drop of metadata traffic on isolated provider networks with DPDK |
|
| 2019-06-14 22:31:33 |
David Ames |
summary |
Checksum drop of metadata traffic on isolated provider networks with DPDK |
Checksum drop of metadata traffic on isolated networks with DPDK |
|
| 2019-06-14 23:10:52 |
Miguel Lavalle |
neutron: importance |
Undecided |
Medium |
|
| 2019-07-02 09:10:14 |
OpenStack Infra |
charm-neutron-openvswitch: status |
New |
Fix Committed |
|
| 2019-08-07 09:23:57 |
James Page |
charm-neutron-openvswitch: milestone |
|
19.07 |
|
| 2019-08-12 21:17:47 |
David Ames |
charm-neutron-openvswitch: status |
Fix Committed |
Fix Released |
|
| 2020-03-27 16:06:39 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2020-03-27 16:06:39 |
OpenStack Infra |
neutron: assignee |
|
Alexander Vlasov (avlasov) |
|
| 2020-06-04 17:04:37 |
Andrea Ieri |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2020-06-04 17:04:44 |
Andrea Ieri |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2020-06-20 18:58:16 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2020-07-01 03:10:24 |
OpenStack Infra |
tags |
cpe-onsite |
cpe-onsite in-stable-train |
|
| 2020-07-01 15:47:47 |
OpenStack Infra |
tags |
cpe-onsite in-stable-train |
cpe-onsite in-stable-train in-stable-ussuri |
|
| 2020-07-01 19:59:33 |
OpenStack Infra |
tags |
cpe-onsite in-stable-train in-stable-ussuri |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri |
|
| 2021-01-21 19:06:59 |
Erlon R. Cruz |
attachment added |
|
lp12345_bionic.debdiff https://bugs.launchpad.net/neutron/+bug/1832021/+attachment/5455267/+files/lp12345_bionic.debdiff |
|
| 2021-01-21 19:08:16 |
Erlon R. Cruz |
attachment removed |
lp12345_bionic.debdiff https://bugs.launchpad.net/neutron/+bug/1832021/+attachment/5455267/+files/lp12345_bionic.debdiff |
|
|
| 2021-03-01 19:35:12 |
Erlon R. Cruz |
description |
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
When we added the following in the qdhcp netns, VMs regained access to metadata:
iptables -t mangle -A OUTPUT -o ns-+ -p tcp --sport 80 -j CHECKSUM --checksum-fill
It seems this setting was recently removed from the qrouter netns [0] but it never existed in the qdhcp to begin with.
[0] https://review.opendev.org/#/c/654645/
Related LP Bug #1831935
See https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1831935/comments/10 |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
|
| 2021-03-01 19:36:40 |
Erlon R. Cruz |
bug task added |
|
cloud-archive |
|
| 2021-03-11 12:40:00 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/queens |
|
| 2021-03-11 12:40:00 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/queens |
|
| 2021-03-11 12:40:00 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/stein |
|
| 2021-03-11 12:40:00 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/stein |
|
| 2021-03-11 12:40:07 |
Edward Hope-Morley |
cloud-archive/stein: status |
New |
Fix Released |
|
| 2021-03-11 12:40:13 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/rocky |
|
| 2021-03-11 12:40:13 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/rocky |
|
| 2021-03-11 12:40:45 |
Edward Hope-Morley |
bug task added |
|
neutron (Ubuntu) |
|
| 2021-03-11 13:04:25 |
Erlon R. Cruz |
cloud-archive/queens: assignee |
|
Erlon R. Cruz (sombrafam) |
|
| 2021-03-11 13:04:31 |
Erlon R. Cruz |
cloud-archive/rocky: assignee |
|
Erlon R. Cruz (sombrafam) |
|
| 2021-03-11 13:04:36 |
Erlon R. Cruz |
cloud-archive/stein: assignee |
|
Erlon R. Cruz (sombrafam) |
|
| 2021-03-11 13:04:51 |
Erlon R. Cruz |
charm-neutron-openvswitch: assignee |
|
Erlon R. Cruz (sombrafam) |
|
| 2021-03-11 13:05:01 |
Erlon R. Cruz |
cloud-archive: assignee |
|
Erlon R. Cruz (sombrafam) |
|
| 2021-03-11 13:19:35 |
Erlon R. Cruz |
cloud-archive: importance |
Undecided |
Medium |
|
| 2021-03-11 13:19:44 |
Erlon R. Cruz |
cloud-archive/queens: importance |
Undecided |
Medium |
|
| 2021-03-11 13:19:50 |
Erlon R. Cruz |
cloud-archive/rocky: importance |
Undecided |
Medium |
|
| 2021-03-11 13:19:55 |
Erlon R. Cruz |
cloud-archive/stein: importance |
Undecided |
Medium |
|
| 2021-04-26 18:56:37 |
Erlon R. Cruz |
attachment added |
|
neutron-dpdk-fix.debdiff https://bugs.launchpad.net/neutron/+bug/1832021/+attachment/5492534/+files/neutron-dpdk-fix.debdiff |
|
| 2021-04-26 19:00:19 |
Erlon R. Cruz |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri sts-sru-needed |
|
| 2021-04-26 20:26:56 |
Ubuntu Foundations Team Bug Bot |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri sts-sru-needed |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed |
|
| 2021-04-26 20:27:05 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2021-05-02 00:20:25 |
Mathew Hodson |
neutron (Ubuntu): importance |
Undecided |
Medium |
|
| 2021-05-02 00:21:15 |
Mathew Hodson |
nominated for series |
|
Ubuntu Bionic |
|
| 2021-05-02 00:21:15 |
Mathew Hodson |
bug task added |
|
neutron (Ubuntu Bionic) |
|
| 2021-05-02 00:21:26 |
Mathew Hodson |
neutron (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2021-05-02 00:39:17 |
Mathew Hodson |
nominated for series |
|
Ubuntu Focal |
|
| 2021-05-02 00:39:17 |
Mathew Hodson |
bug task added |
|
neutron (Ubuntu Focal) |
|
| 2021-05-02 00:41:10 |
Mathew Hodson |
neutron (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2021-05-02 00:41:10 |
Mathew Hodson |
neutron (Ubuntu Focal): status |
New |
Fix Released |
|
| 2021-05-02 00:42:34 |
Mathew Hodson |
neutron (Ubuntu): status |
New |
Fix Released |
|
| 2021-05-04 18:38:37 |
Erlon R. Cruz |
attachment added |
|
neutron-dpdk-rocky-fix.debdiff https://bugs.launchpad.net/neutron/+bug/1832021/+attachment/5494768/+files/neutron-dpdk-rocky-fix.debdiff |
|
| 2021-06-02 18:16:58 |
Brian Murray |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2021-06-02 19:22:23 |
Erlon R. Cruz |
attachment removed |
neutron-dpdk-queen-fix.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1832021/+attachment/5492534/+files/neutron-dpdk-fix.debdiff |
|
|
| 2021-06-02 19:23:16 |
Erlon R. Cruz |
attachment added |
|
neutron-dpdk-bionic-fix.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1832021/+attachment/5501964/+files/neutron-dpdk-bionic-fix.debdiff |
|
| 2021-06-02 20:07:06 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-06-02 21:06:24 |
Brian Murray |
neutron (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2021-06-02 21:06:33 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2021-06-02 21:06:42 |
Brian Murray |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-needed verification-needed-bionic |
|
| 2021-06-03 12:23:34 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Fix Committed |
|
| 2021-06-03 12:23:39 |
Corey Bryant |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-needed verification-needed-bionic |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-needed verification-needed-bionic verification-rocky-needed |
|
| 2021-06-03 12:23:41 |
Corey Bryant |
cloud-archive/queens: status |
New |
Fix Committed |
|
| 2021-06-03 12:24:49 |
Corey Bryant |
cloud-archive: status |
New |
Invalid |
|
| 2021-06-10 23:39:32 |
Erlon R. Cruz |
description |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
QUEENS VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/mM22nBsSG2/
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
|
| 2021-06-10 23:40:17 |
Erlon R. Cruz |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-needed verification-needed-bionic verification-rocky-needed |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-bionic-done verification-needed verification-rocky-needed |
|
| 2021-06-10 23:40:33 |
Erlon R. Cruz |
description |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
QUEENS VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/mM22nBsSG2/
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
BIONIC VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/mM22nBsSG2/
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
|
| 2021-06-13 06:25:58 |
Mathew Hodson |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-bionic-done verification-needed verification-rocky-needed |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-rocky-needed |
|
| 2021-06-15 14:30:26 |
Erlon R. Cruz |
description |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
BIONIC VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/mM22nBsSG2/
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_isolated_metadata = True
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-local-dhcp-and-metadata: true' in neutron-openvswitch. A sample, simple 3 node bundle can be found here[1].
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-network-type flat \
--provider-physical-network physnet1 dpdk_net \
--external
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.230.58.100,end=10.230.58.200 \
--subnet-range 10.230.56.0/21 --dhcp --gateway 10.230.56.1 \
--dns-nameserver 10.230.56.2 \
--ip-version 4 --network dpdk_net dpdk_subnet
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_page_size=large
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.py[DEBUG]: Failed reading optional path http://169.254.169.254/openstack/2015-10-15/user_data due to: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
BIONIC VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/mM22nBsSG2/
ROCKY VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https://paste.ubuntu.com/p/qKcdWMxnsJ/
[1] https://gist.github.com/sombrafam/e0741138773e444960eb4aeace6e3e79 |
|
| 2021-06-15 14:30:59 |
Erlon R. Cruz |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-rocky-needed |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-done-rocky |
|
| 2021-06-16 03:55:02 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-06-16 04:00:23 |
Launchpad Janitor |
neutron (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-06-17 19:13:48 |
Corey Bryant |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
| 2021-06-28 13:04:21 |
Edward Hope-Morley |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-done-rocky |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-done-rocky verification-needed-queens |
|
| 2021-07-08 13:37:04 |
Erlon R. Cruz |
tags |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-done-rocky verification-needed-queens |
cpe-onsite in-stable-stein in-stable-train in-stable-ussuri patch sts-sru-needed verification-done-bionic verification-done-rocky verification-queens-failed |
|
| 2021-07-08 18:29:04 |
Corey Bryant |
cloud-archive/queens: status |
Fix Committed |
Won't Fix |
|
| 2021-10-12 12:00:52 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410050 |
|
| 2021-10-12 12:01:14 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~hopem/ubuntu/+source/neutron/+git/neutron/+merge/410051 |
|
