Hello.
There seems to be a bug with port chain update which causes the chaining rule to disappear from ovs. It happens when I update the chain removing some of it's elements. For example, say there is a chain with group1, group2, classifier1. If I update the chain adding a classifier2 (--flow-classifier classifier1 --flow-classifier classifier2) everything is fine, but if I now try to remove classifier2 (--flow-classifier classifier1) the ovs rule in table 5 of br-int on compute node disappears and the chain breaks.
-----------------
detailed steps:
1. Create 3 vms connected to one network
2. Create bidirectional port-pair for vm1. Create port-pair-group for this port-pair.
3. Create flow classifier with logical-source-port = vm2_port, protocol tcp.
4. Create flow classifier with logical-source-port = vm2_port, protocol icmp.
5. Create port chain with group1, classifier1. (port-chain-create --port-pair-group group1 --flow-classifier classifier1 chain)
6. port-chain-update --flow-classifier cl1 --flow-classifier cl2 chain1
7. port-chain-update --flow-classifier cl1 chain1
After this no traffic is routed to chain from vm2.
Ovs rules on compute node before p.7:
root@node-13:~# ovs-ofctl dump-flows br-int -Oopenflow13
2016-10-18T12:43:59Z|00001|ofp_actions|INFO|OFPAT_SET_MPLS_TTL is deprecated in OpenFlow13 (use Set-Field)
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0xadaeaf73ae7b63ce, duration=8.302s, table=0, n_packets=0, n_bytes=0, priority=30,icmp,in_port=181 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=8.201s, table=0, n_packets=0, n_bytes=0, priority=30,tcp,in_port=181 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=8.047s, table=0, n_packets=0, n_bytes=0, priority=30,icmp,in_port=179 actions=group:1
cookie=0xadaeaf73ae7b63ce, duration=7.948s, table=0, n_packets=0, n_bytes=0, priority=30,tcp,in_port=179 actions=group:1
cookie=0xadaeaf73ae7b63ce, duration=12214.067s, table=0, n_packets=18, n_bytes=1880, priority=20,mpls actions=resubmit(,10)
cookie=0xadaeaf73ae7b63ce, duration=4804.825s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=179,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4798.018s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=180,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4793.402s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=181,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4804.632s, table=0, n_packets=41, n_bytes=1722, priority=10,arp,in_port=179 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4797.783s, table=0, n_packets=39, n_bytes=1638, priority=10,arp,in_port=180 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4793.206s, table=0, n_packets=43, n_bytes=1806, priority=10,arp,in_port=181 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4805.054s, table=0, n_packets=76, n_bytes=11905, priority=9,in_port=179 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4798.241s, table=0, n_packets=71, n_bytes=11665, priority=9,in_port=180 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4793.600s, table=0, n_packets=54, n_bytes=10243, priority=9,in_port=181 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=12216.425s, table=0, n_packets=2695606, n_bytes=272444738, priority=0 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=8.748s, table=5, n_packets=0, n_bytes=0, priority=0,ip,dl_dst=fa:16:3e:f3:09:06 actions=push_mpls:0x8847,set_field:511->mpls_label,set_mpls_ttl(255),push_vlan:0x8100,set_field:4098->vlan_vid,resubmit(,10)
cookie=0xadaeaf73ae7b63ce, duration=7.898s, table=10, n_packets=0, n_bytes=0, priority=1,mpls,dl_vlan=2,dl_dst=fa:16:3e:f3:09:06,mpls_label=511 actions=pop_vlan,pop_mpls:0x0800,output:181
cookie=0xadaeaf73ae7b63ce, duration=12213.981s, table=10, n_packets=16, n_bytes=1184, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=12216.339s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=4804.933s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=179,icmp_type=136,nd_target=fe80::f816:3eff:fec8:e209 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4798.141s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=180,icmp_type=136,nd_target=fe80::f816:3eff:fe30:c666 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4793.501s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=181,icmp_type=136,nd_target=fe80::f816:3eff:fef3:906 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4804.728s, table=24, n_packets=41, n_bytes=1722, priority=2,arp,in_port=179,arp_spa=192.168.100.13 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4797.910s, table=24, n_packets=39, n_bytes=1638, priority=2,arp,in_port=180,arp_spa=192.168.100.14 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4793.304s, table=24, n_packets=43, n_bytes=1806, priority=2,arp,in_port=181,arp_spa=192.168.100.15 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=12216.252s, table=24, n_packets=164, n_bytes=6888, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=4805.250s, table=25, n_packets=115, n_bytes=13487, priority=2,in_port=179,dl_src=fa:16:3e:c8:e2:09 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4798.476s, table=25, n_packets=108, n_bytes=13163, priority=2,in_port=180,dl_src=fa:16:3e:30:c6:66 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4793.797s, table=25, n_packets=95, n_bytes=11909, priority=2,in_port=181,dl_src=fa:16:3e:f3:09:06 actions=NORMAL
after p.7:
root@node-13:~# ovs-ofctl dump-flows br-int -Oopenflow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0xadaeaf73ae7b63ce, duration=1.805s, table=0, n_packets=0, n_bytes=0, priority=30,icmp,in_port=181 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=1.511s, table=0, n_packets=0, n_bytes=0, priority=30,icmp,in_port=179 actions=group:1
cookie=0xadaeaf73ae7b63ce, duration=12258.014s, table=0, n_packets=18, n_bytes=1880, priority=20,mpls actions=resubmit(,10)
cookie=0xadaeaf73ae7b63ce, duration=4848.772s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=179,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4841.965s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=180,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4837.349s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=181,icmp_type=136 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4848.579s, table=0, n_packets=41, n_bytes=1722, priority=10,arp,in_port=179 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4841.730s, table=0, n_packets=39, n_bytes=1638, priority=10,arp,in_port=180 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4837.153s, table=0, n_packets=43, n_bytes=1806, priority=10,arp,in_port=181 actions=resubmit(,24)
cookie=0xadaeaf73ae7b63ce, duration=4849.001s, table=0, n_packets=76, n_bytes=11905, priority=9,in_port=179 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4842.188s, table=0, n_packets=71, n_bytes=11665, priority=9,in_port=180 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4837.547s, table=0, n_packets=54, n_bytes=10243, priority=9,in_port=181 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=12260.372s, table=0, n_packets=2695606, n_bytes=272444738, priority=0 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=1.503s, table=10, n_packets=0, n_bytes=0, priority=1,mpls,dl_vlan=2,dl_dst=fa:16:3e:f3:09:06,mpls_label=511 actions=pop_vlan,pop_mpls:0x0800,output:181
cookie=0xadaeaf73ae7b63ce, duration=12257.928s, table=10, n_packets=16, n_bytes=1184, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=12260.286s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=4848.880s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=179,icmp_type=136,nd_target=fe80::f816:3eff:fec8:e209 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4842.088s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=180,icmp_type=136,nd_target=fe80::f816:3eff:fe30:c666 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4837.448s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=181,icmp_type=136,nd_target=fe80::f816:3eff:fef3:906 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4848.675s, table=24, n_packets=41, n_bytes=1722, priority=2,arp,in_port=179,arp_spa=192.168.100.13 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4841.857s, table=24, n_packets=39, n_bytes=1638, priority=2,arp,in_port=180,arp_spa=192.168.100.14 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=4837.251s, table=24, n_packets=43, n_bytes=1806, priority=2,arp,in_port=181,arp_spa=192.168.100.15 actions=resubmit(,25)
cookie=0xadaeaf73ae7b63ce, duration=12260.199s, table=24, n_packets=164, n_bytes=6888, priority=0 actions=drop
cookie=0xadaeaf73ae7b63ce, duration=4849.197s, table=25, n_packets=115, n_bytes=13487, priority=2,in_port=179,dl_src=fa:16:3e:c8:e2:09 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4842.423s, table=25, n_packets=108, n_bytes=13163, priority=2,in_port=180,dl_src=fa:16:3e:30:c6:66 actions=NORMAL
cookie=0xadaeaf73ae7b63ce, duration=4837.744s, table=25, n_packets=95, n_bytes=11909, priority=2,in_port=181,dl_src=fa:16:3e:f3:09:06 actions=NORMAL
There are no rules in table 5, but there should be one for group1.
root@node-13:~# ovs-ofctl dump-groups br-int -Oopenflow13
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=1,type=select,bucket=actions=set_field:fa:16:3e:f3:09:06->eth_dst,resubmit(,5)
neutron-openvswitch-agent.log:
2016-10-18 12:38:42.480 27428 ERROR neutron.agent.linux.utils [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Exit code: 1; Stdin: group_id=1,type=select,bucket=weight=1, mod_dl_dst:fa:16:3e:f3:09:06,resubmit(,5); Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPGMFC_UNKNOWN_GROUP
OFPT_GROUP_MOD (OF1.3) (xid=0x2):
MOD group_id=1,type=select,bucket=actions=set_field:fa:16:3e:f3:09:06->eth_dst,resubmit(,5)
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Exit code: 1; Stdin: group_id=1,type=select,bucket=weight=1, mod_dl_dst:fa:16:3e:f3:09:06,resubmit(,5); Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPGMFC_UNKNOWN_GROUP
OFPT_GROUP_MOD (OF1.3) (xid=0x2):
MOD group_id=1,type=select,bucket=actions=set_field:fa:16:3e:f3:09:06->eth_dst,resubmit(,5)
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib Traceback (most recent call last):
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib File "/usr/local/lib/python2.7/dist-packages/networking_sfc/services/sfc/common/ovs_ext_lib.py", line 128, in run_ofctl
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib process_input=process_input)
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib raise RuntimeError(msg)
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib RuntimeError: Exit code: 1; Stdin: group_id=1,type=select,bucket=weight=1, mod_dl_dst:fa:16:3e:f3:09:06,resubmit(,5); Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPGMFC_UNKNOWN_GROUP
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib OFPT_GROUP_MOD (OF1.3) (xid=0x2):
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib MOD group_id=1,type=select,bucket=actions=set_field:fa:16:3e:f3:09:06->eth_dst,resubmit(,5)
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib
2016-10-18 12:38:42.481 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib
2016-10-18 12:38:42.483 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Unable to execute ['ovs-ofctl', '-O openflow13', 'mod-group', 'br-int', '-'].
2016-10-18 12:38:43.049 27428 ERROR neutron.agent.linux.utils [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=30,dl_type=2048,nw_dst=0.0.0.0/0.0.0.0,nw_proto=1,cookie=12515133325892608974,tp_dst=0/0x0,table=0,tp_src=0/0x0,nw_src=0.0.0.0/0.0.0.0,in_port=179,actions=group:1; Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPBAC_BAD_OUT_GROUP
OFPT_FLOW_MOD (OF1.3) (xid=0x2):
(***truncated to 64 bytes from 88***)
00000000 04 0e 00 58 00 00 00 02-ad ae af 73 ae 7b 63 ce |...X.......s.{c.|
00000010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 1e |................|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 17 80 00 00 04-00 00 00 b3 80 00 0a 02 |................|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=30,dl_type=2048,nw_dst=0.0.0.0/0.0.0.0,nw_proto=1,cookie=12515133325892608974,tp_dst=0/0x0,table=0,tp_src=0/0x0,nw_src=0.0.0.0/0.0.0.0,in_port=179,actions=group:1; Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPBAC_BAD_OUT_GROUP
OFPT_FLOW_MOD (OF1.3) (xid=0x2):
(***truncated to 64 bytes from 88***)
00000000 04 0e 00 58 00 00 00 02-ad ae af 73 ae 7b 63 ce |...X.......s.{c.|
00000010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 1e |................|
00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
00000030 00 01 00 17 80 00 00 04-00 00 00 b3 80 00 0a 02 |................|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib Traceback (most recent call last):
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib File "/usr/local/lib/python2.7/dist-packages/networking_sfc/services/sfc/common/ovs_ext_lib.py", line 128, in run_ofctl
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib process_input=process_input)
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib raise RuntimeError(msg)
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib RuntimeError: Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=30,dl_type=2048,nw_dst=0.0.0.0/0.0.0.0,nw_proto=1,cookie=12515133325892608974,tp_dst=0/0x0,table=0,tp_src=0/0x0,nw_src=0.0.0.0/0.0.0.0,in_port=179,actions=group:1; Stdout: ; Stderr: OFPT_ERROR (OF1.3) (xid=0x2): OFPBAC_BAD_OUT_GROUP
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib OFPT_FLOW_MOD (OF1.3) (xid=0x2):
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib (***truncated to 64 bytes from 88***)
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib 00000000 04 0e 00 58 00 00 00 02-ad ae af 73 ae 7b 63 ce |...X.......s.{c.|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib 00000010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 1e |................|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib 00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib 00000030 00 01 00 17 80 00 00 04-00 00 00 b3 80 00 0a 02 |................|
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib
2016-10-18 12:38:43.050 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib
2016-10-18 12:38:43.051 27428 ERROR networking_sfc.services.sfc.common.ovs_ext_lib [req-ac236a6a-a2e0-476e-9e53-9abf2d7814d4 - - - - -] Unable to execute ['ovs-ofctl', '-O openflow13', 'add-flows', 'br-int', '-'].
-----------------------
Other cases when the same bug happens:
1. Chain has two groups, call update to remove one.
2. Call chain update with the same objects chain already has
Bug does NOT happen when:
1. "expand" chain - add classifiers or groups, do not remove anything
2. Remove all classifiers with --no-flow-classifier option
networking-sfc branch: stable/mitaka (commit bed0bba8b44856d4aff20252f7189fbaffcaa94d, networking-sfc version 2.0.0)
environment: mirantis openstack 9.0 mitaka
I'd have to test again with the stable/mitaka to confirm, but I cannot reproduce with current master (single node devstack). There were quite a few changes though in around the OVS driver.