| 2020-12-05 13:35:19 |
Frode Nordahl |
bug |
|
|
added bug |
| 2020-12-05 13:35:41 |
Frode Nordahl |
ovn (Ubuntu): status |
New |
Triaged |
|
| 2020-12-05 13:35:44 |
Frode Nordahl |
ovn (Ubuntu): importance |
Undecided |
High |
|
| 2020-12-07 09:37:31 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2020-12-07 16:41:03 |
Frode Nordahl |
ovn (Ubuntu): assignee |
|
Frode Nordahl (fnordahl) |
|
| 2020-12-09 08:05:31 |
Márton Kiss |
bug |
|
|
added subscriber Canonical Field Critical |
| 2020-12-10 04:46:11 |
Frode Nordahl |
ovn (Ubuntu): importance |
High |
Critical |
|
| 2020-12-10 14:38:34 |
Frode Nordahl |
ovn (Ubuntu): status |
Triaged |
In Progress |
|
| 2020-12-11 11:26:35 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~fnordahl/ubuntu/+source/ovn/+git/ovn/+merge/394584 |
|
| 2020-12-11 11:53:03 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~fnordahl/ubuntu/+source/ovn/+git/ovn/+merge/395221 |
|
| 2020-12-11 22:14:42 |
Frode Nordahl |
bug task added |
|
cloud-archive |
|
| 2020-12-11 22:15:43 |
Frode Nordahl |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-12-11 22:15:43 |
Frode Nordahl |
bug task added |
|
ovn (Ubuntu Groovy) |
|
| 2020-12-11 22:15:43 |
Frode Nordahl |
nominated for series |
|
Ubuntu Focal |
|
| 2020-12-11 22:15:43 |
Frode Nordahl |
bug task added |
|
ovn (Ubuntu Focal) |
|
| 2020-12-11 22:16:32 |
Frode Nordahl |
ovn (Ubuntu Focal): status |
New |
In Progress |
|
| 2020-12-11 22:16:37 |
Frode Nordahl |
ovn (Ubuntu Groovy): status |
New |
In Progress |
|
| 2020-12-11 22:16:43 |
Frode Nordahl |
ovn (Ubuntu Groovy): importance |
Undecided |
Critical |
|
| 2020-12-11 22:16:46 |
Frode Nordahl |
ovn (Ubuntu Focal): importance |
Undecided |
Critical |
|
| 2020-12-11 22:16:50 |
Frode Nordahl |
ovn (Ubuntu Groovy): assignee |
|
Frode Nordahl (fnordahl) |
|
| 2020-12-11 22:16:54 |
Frode Nordahl |
ovn (Ubuntu Focal): assignee |
|
Frode Nordahl (fnordahl) |
|
| 2020-12-11 22:17:00 |
Frode Nordahl |
ovn (Ubuntu): status |
In Progress |
Triaged |
|
| 2020-12-11 22:17:25 |
Frode Nordahl |
ovn (Ubuntu): assignee |
Frode Nordahl (fnordahl) |
|
|
| 2021-01-06 08:15:57 |
Frode Nordahl |
description |
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior.
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
|
| 2021-01-06 08:17:21 |
Frode Nordahl |
ovn (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2021-01-11 12:12:32 |
James Page |
cloud-archive: status |
New |
Triaged |
|
| 2021-01-11 12:12:34 |
James Page |
cloud-archive: importance |
Undecided |
Critical |
|
| 2021-01-11 12:12:51 |
James Page |
nominated for series |
|
cloud-archive/ussuri |
|
| 2021-01-11 12:12:51 |
James Page |
bug task added |
|
cloud-archive/ussuri |
|
| 2021-01-11 12:12:51 |
James Page |
nominated for series |
|
cloud-archive/victoria |
|
| 2021-01-11 12:12:51 |
James Page |
bug task added |
|
cloud-archive/victoria |
|
| 2021-01-11 12:12:59 |
James Page |
cloud-archive/victoria: status |
Triaged |
In Progress |
|
| 2021-01-11 12:13:01 |
James Page |
cloud-archive/ussuri: status |
New |
In Progress |
|
| 2021-01-11 12:13:03 |
James Page |
cloud-archive/ussuri: importance |
Undecided |
Critical |
|
| 2021-01-12 11:44:29 |
James Page |
ovn (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2021-01-12 11:46:17 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-01-19 19:12:41 |
Brian Murray |
ovn (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2021-01-19 19:12:44 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2021-01-19 19:12:47 |
Brian Murray |
bug |
|
|
added subscriber Brian Murray |
| 2021-01-19 19:12:51 |
Brian Murray |
tags |
|
verification-needed verification-needed-groovy |
|
| 2021-01-19 19:22:41 |
Brian Murray |
ovn (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-01-19 19:22:48 |
Brian Murray |
tags |
verification-needed verification-needed-groovy |
verification-needed verification-needed-focal verification-needed-groovy |
|
| 2021-01-21 22:23:28 |
Corey Bryant |
cloud-archive/ussuri: status |
In Progress |
Fix Committed |
|
| 2021-01-21 22:23:30 |
Corey Bryant |
tags |
verification-needed verification-needed-focal verification-needed-groovy |
verification-needed verification-needed-focal verification-needed-groovy verification-ussuri-needed |
|
| 2021-02-08 14:33:12 |
Frode Nordahl |
description |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patched caused a regression chances are very high it would have bubbled up somewhere by now.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
|
| 2021-02-08 14:37:33 |
Frode Nordahl |
description |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patched caused a regression chances are very high it would have bubbled up somewhere by now.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patched caused a regression chances are very high it would have bubbled up somewhere by now. For extra caution we have had the packages in -proposed for an extended period and the packages has also been consumed in other recent large scale internal networking tests, such as the PS5 project.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
|
| 2021-02-08 14:47:32 |
Frode Nordahl |
description |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patched caused a regression chances are very high it would have bubbled up somewhere by now. For extra caution we have had the packages in -proposed for an extended period and the packages has also been consumed in other recent large scale internal networking tests, such as the PS5 project.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
[Impact]
When CMS configures ACLs with overlapping rules the flow rules OVN programs into Open vSwitch may lead to unpredictable forwarding behavior such as every other packet being dropped.
[Test Case]
How to reproduce with OpenStack as CMS:
- Update the "default" group to accept ICMP, then:
openstack security group create a
openstack security group create b
openstack security group create c
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group b b
openstack security group rule create --ingress --ethertype IPv4 --protocol icmp --remote-group c c
openstack security group rule create --ingress --ethertype IPv6 --protocol icmp --remote-group c c
openstack server add security group
for server in zaza-neutrontests-ins-1 zaza-neutrontests-ins-2; do for group in a b c; do openstack server add security group $server $group;done;done
Look for bad conjunction messages in ovn-controller log and monitor ICMP reachability to the instances.
[Regression potential]
The fixes all apply to a single file and area of the OVN controller operation, except for the patches to its tests. 6 of the patches have been in the wild since the 20.09 release of September 2020. 10 of them have been in the wild since the 20.12 release of December 2020. There has since not been any bugs reported nor further updates touching this area of the code. We have also had the code in the wild through Ubuntu Groovy with OVN 20.06 (the parts that are in 20.06) and Ubuntu Hirsute (all of them). The code paths are executed by anyone using OVN so if any of these patches caused a regression chances are very high it would have bubbled up somewhere by now. For extra caution we have had the packages in -proposed for an extended period and the packages has also been consumed in other recent large scale internal networking tests, such as the PS5 project.
[Other Info]
Fixed upstream:
https://github.com/ovn-org/ovn/commit/986b3d5e4ad6f05245d021ba699c957246294a22
Other bug trackers:
https://bugzilla.redhat.com/1871931
Symptoms:
Every other packet does not arrive.
2020-12-05T10:33:38.304Z|00016|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1af): NXBAC_BAD_CONJUNCTION
OFPT_FLOW_MOD (OF1.3) (xid=0x1af): ***decode error: NXBAC_BAD_CONJUNCTION***
00000000 04 0e 00 b0 00 00 01 af-00 00 00 00 e6 89 28 3a |..............(:|
00000010 00 00 00 00 00 00 00 00-2c 00 00 00 00 00 07 d2 |........,.......|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 53 80 00 0a 02-08 00 80 00 14 01 01 00 |...S............|
00000040 01 1e 04 00 00 00 03 00-01 d3 08 00 00 00 22 00 |..............".|
00000050 00 00 2b 00 01 d9 20 00-00 00 00 00 00 00 00 00 |..+... .........|
00000060 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 01 80-00 04 08 00 00 00 00 00 |................|
00000080 00 00 03 00 00 00 00 00-00 04 00 28 00 00 00 00 |...........(....|
00000090 ff ff 00 10 00 00 23 20-00 0e ff f8 2d 00 00 00 |......# ....-...|
000000a0 ff ff 00 10 00 00 23 20-00 22 01 02 00 00 00 09 |......# ."......|
I have been able to backport this fix to 20.03.1 with minor adaption using these commits from master, however a flaky test may need some more investigation:
commit 986b3d5e4ad6f05245d021ba699c957246294a22
commit 33c15c145988daa6172928dc870f3a0225515f50
commit 107bb25029350bd0f7dfeeb0ef3053adbd504e3e
commit e49ce9a33f38f29c44e3c30afcc189b5f6a9ef8e
commit dadae4f800ccb1f2759378f0bd804dd002e31605
commit 7cab7bd1268ba67429954da4f73de91090acf779
commit 9d2e8d32fb9865513b70408a665184a67564390d
commit f4e508dd7a6cfbfc2e3250a8c11a8d0fdc1dfdd0
commit 6f0b1e02d9ab3a94048c4818f2d382938cad4b71
commit 23063cf4178c05f5d6b3e4ec6d323ccc88df6101
commit 354d3853d40cbce89a434632f67daed7fc992d8b
The list of commits is quite long and this is due to how controller/ofctrl.c has changed from 20.03.1 was cut until now, but the nature of the changes look sane to me. |
|
| 2021-02-16 09:28:51 |
Frode Nordahl |
tags |
verification-needed verification-needed-focal verification-needed-groovy verification-ussuri-needed |
verification-needed verification-needed-focal verification-needed-groovy verification-ussuri-done |
|
| 2021-02-16 09:48:56 |
Frode Nordahl |
tags |
verification-needed verification-needed-focal verification-needed-groovy verification-ussuri-done |
verification-done-focal verification-needed verification-needed-groovy verification-ussuri-done |
|
| 2021-02-16 09:56:33 |
Frode Nordahl |
tags |
verification-done-focal verification-needed verification-needed-groovy verification-ussuri-done |
verification-done verification-done-focal verification-done-groovy verification-ussuri-done |
|
| 2021-02-18 09:19:40 |
Launchpad Janitor |
ovn (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2021-02-18 09:19:48 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-02-18 09:26:48 |
Launchpad Janitor |
ovn (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-03-01 15:45:06 |
Corey Bryant |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
| 2021-03-02 08:59:45 |
James Page |
cloud-archive/victoria: status |
In Progress |
Invalid |
|
