Checksum drop of metadata traffic on isolated networks with DPDK
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Neutron Open vSwitch Charm |
Fix Released
|
Undecided
|
Erlon R. Cruz | ||
Ubuntu Cloud Archive |
Invalid
|
Medium
|
Erlon R. Cruz | ||
Queens |
Won't Fix
|
Medium
|
Erlon R. Cruz | ||
Rocky |
Fix Released
|
Medium
|
Erlon R. Cruz | ||
Stein |
Fix Released
|
Medium
|
Erlon R. Cruz | ||
neutron |
Fix Released
|
Medium
|
Alexander Vlasov | ||
neutron (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Bionic |
Fix Released
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Unassigned |
Bug Description
[Impact]
When an isolated network using provider networks for tenants (meaning without virtual routers: DVR or network node), metadata access occurs in the qdhcp ip netns rather than the qrouter netns.
The following options are set in the dhcp_agent.ini file:
force_metadata = True
enable_
VMs on the provider tenant network are unable to access metadata as packets are dropped due to checksum.
[Test Plan]
1. Create an OpenStack deployment with DPDK options enabled and 'enable-
2. Create an external flat network and subnet:
openstack network show dpdk_net || \
openstack network create --provider-
openstack subnet show dpdk_net || \
openstack subnet create --allocation-pool start=10.
3. Create an instance attached to that network. The instance must have a flavor that uses huge pages.
openstack flavor create --ram 8192 --disk 50 --vcpus 4 m1.dpdk
openstack flavor set m1.dpdk --property hw:mem_
openstack server create --wait --image xenial --flavor m1.dpdk --key-name testkey --network dpdk_net i1
4. Log into the instance host and check the instance console. The instance will hang into the boot and show the following message:
2020-11-20 09:43:26,790 - openstack.
5. Apply the fix in all computes, restart the DHCP agents in all computes and create the instance again.
6. No errors should be shown and the instance quickly boots.
[Where problems could occur]
* This change is only touched if datapath_type and ovs_use_veth. Those settings are mostly used for DPDK environments. The core of the fix is
to toggle off checksum offload done by the DHCP namespace interfaces.
This will have the drawback of adding some overhead on the packet processing for DHCP traffic but given DHCP does not demand too much data, this should be a minor proble.
* Future changes on the syntax of the ethtool command could cause regressions
[Other Info]
* None
BIONIC VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https:/
ROCKY VERIFICATION DONE
1 - Followed through the process above and confirmed that after installing the package the problems are solved.
2 - Testing output can be seen here: https:/
[1] https:/
tags: | added: cpe-onsite |
Changed in neutron: | |
status: | New → Incomplete |
Changed in neutron: | |
status: | Incomplete → New |
summary: |
- Checksum drop of metadata traffic on isolated provider networks + Checksum drop of metadata traffic on isolated provider networks with + DPDK |
Changed in neutron: | |
importance: | Undecided → Medium |
Changed in charm-neutron-openvswitch: | |
milestone: | none → 19.07 |
Changed in charm-neutron-openvswitch: | |
status: | Fix Committed → Fix Released |
Changed in neutron: | |
assignee: | nobody → Alexander Vlasov (avlasov) |
status: | New → In Progress |
description: | updated |
Changed in charm-neutron-openvswitch: | |
assignee: | nobody → Erlon R. Cruz (sombrafam) |
Changed in cloud-archive: | |
assignee: | nobody → Erlon R. Cruz (sombrafam) |
Changed in cloud-archive: | |
importance: | Undecided → Medium |
Changed in neutron (Ubuntu): | |
importance: | Undecided → Medium |
Changed in neutron (Ubuntu Bionic): | |
importance: | Undecided → Medium |
description: | updated |
tags: |
added: verification-bionic-done removed: verification-needed-bionic |
description: | updated |
tags: |
added: verification-done-bionic removed: verification-bionic-done verification-needed |
description: | updated |
tags: |
added: verification-done-rocky removed: verification-rocky-needed |
tags: | added: verification-needed-queens |
David - please see the link that was the reason I reverted this change, https:/ /lore.kernel. org/patchwork/ patch/824819/ - that is basically saying this rule has no effect for TCP, it was only meant for UDP, and was finally changed to log a warning in the kernel.
There is probably something else going on here causing issues, possibly outside of neutron.