Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS firewall driver

[Stanislav Tretiakov]

When use Open vSwitch Firewall Driver we noticed that ICMPv6 RA unicast packets are not reaching the VM.

=========================================Troubleshooting flow:===========================================

1) Catch ICMPv6 RA package on physical hypervisor bonding interface:

#tcpdump -XXvpnei bond0 -Q in -c 1 "icmp6[0] = 134 and ether host fa:16:3e:68:e8:19"
dropped privs to tcpdump
tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:56:43.669691 26:28:b0:96:c0:c7 > fa:16:3e:68:e8:19, ethertype 802.1Q (0x8100), length 162: vlan 3254, p 0, ethertype IPv6, (flowlabel 0x7cc6b, hlim 255, next-header ICMPv6 (58) payload length: 104) fe80::2428:b0ff:fe96:c0c7 > fe80::f816:3eff:fe68:e819: [icmp6 sum ok] ICMP6, router advertisement, length 104
 hop limit 64, Flags [managed], pref medium, router lifetime 6s, reachable time 0ms, retrans timer 0ms
   prefix info option (3), length 32 (4): 2a05:fc1:200::/64, Flags [onlink, auto], valid time 2592000s, pref. time 14400s
   rdnss option (25), length 40 (5): lifetime 2s, addr: 2a05:fc1::2 addr: 2a05:fc1::3
   source link-address option (1), length 8 (1): 26:28:b0:96:c0:c7
   advertisement interval option (7), length 8 (1): 2000ms
 0x0000: fa16 3e68 e819 2628 b096 c0c7 8100 0cb6 ..>h..&(........
 0x0010: 86dd 6007 cc6b 0068 3aff fe80 0000 0000 ..`..k.h:.......
 0x0020: 0000 2428 b0ff fe96 c0c7 fe80 0000 0000 ..$(............
 0x0030: 0000 f816 3eff fe68 e819 8600 10d2 4080 ....>..h......@.
 0x0040: 0006 0000 0000 0000 0000 0304 40c0 0027 ............@..'
 0x0050: 8d00 0000 3840 0000 0000 2a05 0fc1 0200 ....8@....*.....
 0x0060: 0000 0000 0000 0000 0000 1905 0000 0000 ................
 0x0070: 0002 2a05 0fc1 0000 0000 0000 0000 0000 ..*.............
 0x0080: 0002 2a05 0fc1 0000 0000 0000 0000 0000 ..*.............
 0x0090: 0003 0101 2628 b096 c0c7 0701 0000 0000 ....&(..........
 0x00a0: 07d0 ..

2) Trace the package using "ofproto/trace":

#ovs-appctl ofproto/trace br-int in_port=1 fa163e68e8192628b096c0c781000cb686dd6007cc6b00683afffe800000000000002428b0fffe96c0c7fe80000000000000f8163efffe68e819860010d2408000060000000000000000030440c000278d0000003840000000002a050fc102000000000000000000000019050000000000022a050fc10000000000000000000000022a050fc100000000000000000000000301012628b096c0c707010000000007d0
Flow: icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

bridge("br-int")
----------------
 0. in_port=1,dl_vlan=3254, priority 3, cookie 0x4ef408376e507615
    set_field:4097->vlan_vid
    goto_table:60
60. dl_vlan=1,dl_dst=fa:16:3e:68:e8:19, priority 90, cookie 0x4ef408376e507615
    set_field:0x9e->reg5
    set_field:0x1->reg6
    pop_vlan
    resubmit(,81)
81. ct_state=-trk,ipv6,reg5=0x9e, priority 90, cookie 0x4ef408376e507615
    ct(table=82,zone=NXM_NX_REG6[0..15])
    drop
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 82.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0
Megaflow: recirc_id=0,ct_state=-trk,eth,icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,dl_dst=fa:16:3e:68:e8:19,nw_frag=no,icmp_type=0x86/0xff
Datapath actions: pop_vlan,ct(zone=1),recirc(0xc30495)

===============================================================================
recirc(0xc30495) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================

Flow: recirc_id=0xc30495,ct_state=new|trk,ct_zone=1,eth,icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

bridge("br-int")
----------------
    thaw
        Resuming from table 82
82. ct_state=+new-est,ipv6,reg5=0x9e, priority 74, cookie 0x4ef408376e507615
    ct(commit,zone=NXM_NX_REG6[0..15])
    drop
     -> Sets the packet to an untracked state, and clears all the conntrack fields.
    output:158
    resubmit(,92)
92. priority 0, cookie 0x4ef408376e507615
    drop

Final flow: recirc_id=0xc30495,eth,icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0
Megaflow: recirc_id=0xc30495,ct_state=+new-est-rel-rpl,eth,ipv6,in_port=1,nw_frag=no
Datapath actions: ct(commit,zone=1),125

3) We've verified that it's all about OVS and the packet is discarded by the rule.

============================================Fixing the issue:============================================

The documentation for the OVS Firewall Driver (https://docs.openstack.org/neutron/latest/contributor/internals/openvswitch_firewall.html#rules-example-with-explanation) lists the rule tables for neighbor solicitation and neighbor advertisement, but there is nothing about router advertisement. We have studied the source code that creates the necessary rules and paid attention to the ICMPV6_ALLOWED_INGRESS_TYPES variable that accepts the list of ICMPv6 types.
We just added the required type there and it solved the problem.

neutron/agent/firewall.py
ICMPV6_ALLOWED_INGRESS_TYPES = (n_const.ICMPV6_TYPE_MLD_QUERY,
+ n_const.ICMPV6_TYPE_RA,
                                n_const.ICMPV6_TYPE_NS,
                                n_const.ICMPV6_TYPE_NA)

It solved the issue.

We got successful trace:
# docker exec openvswitch_vswitchd ovs-appctl ofproto/trace br-ext in_port=1 fa163e68e8192628b096c0c781000cb686dd6007cc6b00683afffe800000000000002428b0fffe96c0c7fe80000000000000f8163efffe68e819860010d2408000060000000000000000030440c000278d0000003840000000002a050fc102000000000000000000000019050000000000022a050fc10000000000000000000000022a050fc100000000000000000000000301012628b096c0c707010000000007d0
Flow: icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0

bridge("br-ext")
----------------
 0. priority 0, cookie 0x919d5b8cf75468f6
    NORMAL
     -> forwarding to learned port

bridge("br-int")
----------------
 0. in_port=1,dl_vlan=3254, priority 3, cookie 0x99db92e60316827
    set_field:4097->vlan_vid
    goto_table:60
60. dl_vlan=1,dl_dst=fa:16:3e:68:e8:19, priority 90, cookie 0x99db92e60316827
    set_field:0x9e->reg5
    set_field:0x1->reg6
    pop_vlan
    resubmit(,81)
81. icmp6,reg5=0x9e,icmp_type=134, priority 100, cookie 0x99db92e60316827
    output:158

Final flow: unchanged
Megaflow: recirc_id=0,eth,icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,nw_frag=no,icmp_type=0x86/0xff
Datapath actions: pop_vlan,125

============================================Affected versions:===========================================

We made tests:
- Ussuri
- Zed
- Master branch

===============================================Conclusions:==============================================

1) I think that correct work has been broken by this one commit: https://opendev.org/openstack/neutron/commit/0dcf3d20c2e5c2592e9674e7277acce4eff98341

2) I suggest returning n_const.ICMPV6_TYPE_RA to the ICMPV6_ALLOWED_INGRESS_TYPES variable and eliminate the duplicated ICMPv6 RA rule from the iptables firewall driver.

[yatin (yatinkarel)]

Thanks @Stanislav for the detailed bug report and referring related commits.

https://opendev.org/openstack/neutron/commit/0dcf3d20c2e5c2592e9674e7277acce4eff98341 was done in favor of https://review.opendev.org/c/openstack/neutron/+/432506 and as per that rules for ICMPV6_TYPE_RA still handled in other way.

Also the issue you reported looks duplicate of https://bugs.launchpad.net/neutron/+bug/1958643 and that was fixed in Yoga+ releases. But you said you hit similar issue in Zed/Master too so something to recheck.

Can you please cross check again the other linked bug and if Yoga+ impacted with that patch included in your tests and share some more details if yoga+ impacted, like:-

- Environment details along with for the impacted port to further triage the missing rules:-
- openstack port show <port id>
- openstack server show <server id>
- openstack network show <network id>
- openstack subnet show <subnet id>

For now marking the bug as INCOMPLETE

[Stanislav Tretiakov (tsv1991)]

Thank you @yatin for the answer.
We did retest and this issue not confirmed for Zed and Master branch. Sorry for the hasty conclusion.
I also looked the bug report https://bugs.launchpad.net/neutron/+bug/1958643 and I confirm that my bug report is duplicate. I sure, the this topic can be closed.

Thank you for your help.

[yatin (yatinkarel)]

Thanks @Stanislav for the confirmation, will close it as Duplicate.