Unpredictable behaviour on conflicting flow actions
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Invalid
|
Critical
|
Unassigned | ||
Ussuri |
Fix Released
|
Critical
|
Unassigned | ||
Victoria |
Invalid
|
Critical
|
Unassigned | ||
ovn (Ubuntu) |
Fix Released
|
Critical
|
Unassigned | ||
Focal |
Fix Released
|
Critical
|
Frode Nordahl | ||
Groovy |
Fix Released
|
Critical
|
Frode Nordahl |
Bug Description
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontes
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patches caused a regression chances are very high it would have bubbled up somewhere by now. For extra caution we have had the packages in -proposed for an extended period and the packages has also been consumed in other recent large scale internal networking tests, such as the PS5 project.
[Other Info]
Fixed upstream:
https:/
Other bug trackers:
https:/
Symptoms:
Every other packet does not arrive.
2020-12-
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05
commit 33c15c145988daa
commit 107bb25029350bd
commit e49ce9a33f38f29
commit dadae4f800ccb1f
commit 7cab7bd1268ba67
commit 9d2e8d32fb98655
commit f4e508dd7a6cfbf
commit 6f0b1e02d9ab3a9
commit 23063cf4178c05f
commit 354d3853d40cbce
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me.
Related branches
- Ubuntu Server Developers: Pending requested
-
Diff: 4356 lines (+4230/-0) (has conflicts)20 files modifieddebian/changelog (+16/-0)
debian/patches/ovn-ctl-cluster-db-upgrades.patch (+63/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-01.patch (+63/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-02.patch (+889/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-03.patch (+718/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-04.patch (+154/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-05.patch (+481/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-06.patch (+109/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-07.patch (+233/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-08.patch (+90/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-09.patch (+216/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-10.patch (+77/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-11.patch (+215/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-12.patch (+40/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-13.patch (+416/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-14.patch (+123/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-15.patch (+66/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-16.patch (+132/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-17.patch (+108/-0)
debian/patches/series (+21/-0)
- Ubuntu Server Developers: Pending requested
-
Diff: 4592 lines (+4464/-0)21 files modifieddebian/changelog (+12/-0)
debian/patches/ovn-ctl-cluster-db-upgrades.patch (+63/-0)
debian/patches/ovn-northd-revert-manage-arp-process-locally-dvr.patch (+198/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-01.patch (+66/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-02.patch (+892/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-03.patch (+719/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-04.patch (+157/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-05.patch (+484/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-06.patch (+112/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-07.patch (+236/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-08.patch (+93/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-09.patch (+220/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-10.patch (+80/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-11.patch (+218/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-12.patch (+43/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-13.patch (+419/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-14.patch (+126/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-15.patch (+66/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-16.patch (+135/-0)
debian/patches/ovn-ofctrl-predictable-resolution-conflicting-flow-actions-17.patch (+106/-0)
debian/patches/series (+19/-0)
Changed in ovn (Ubuntu): | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in ovn (Ubuntu): | |
assignee: | nobody → Frode Nordahl (fnordahl) |
Changed in ovn (Ubuntu): | |
importance: | High → Critical |
Changed in ovn (Ubuntu): | |
status: | Triaged → In Progress |
Changed in ovn (Ubuntu Focal): | |
status: | New → In Progress |
Changed in ovn (Ubuntu Groovy): | |
status: | New → In Progress |
importance: | Undecided → Critical |
Changed in ovn (Ubuntu Focal): | |
importance: | Undecided → Critical |
Changed in ovn (Ubuntu Groovy): | |
assignee: | nobody → Frode Nordahl (fnordahl) |
Changed in ovn (Ubuntu Focal): | |
assignee: | nobody → Frode Nordahl (fnordahl) |
Changed in ovn (Ubuntu): | |
status: | In Progress → Triaged |
assignee: | Frode Nordahl (fnordahl) → nobody |
description: | updated |
Changed in ovn (Ubuntu): | |
status: | Triaged → Fix Committed |
Changed in cloud-archive: | |
status: | New → Triaged |
importance: | Undecided → Critical |
Changed in ovn (Ubuntu): | |
status: | Fix Committed → Fix Released |
description: | updated |
description: | updated |
description: | updated |
Frode, this solves the packet loss issue I saw previously. I marked this field crit, as we don't have a workaround, and would be grate to make it through the SRU process intro stable branch for UCA/Ussuri/Bionic.