lab_setup with IPv6: OVS-DPDK assessment =============================================================================== Summary: =============================================================================== Very broken =============================================================================== Recommendation: =============================================================================== Do not test w/ IPv6 until Eng investigates and fixes the issues =============================================================================== Topology: =============================================================================== 2x2 virtualbox =============================================================================== Build: =============================================================================== compute-0:~# cat /etc/build.info SW_VERSION="18.10" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="swebster" BUILD_NUMBER="n/a" BUILD_HOST="" BUILD_DATE="2018-10-19 09:54:17 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/swebster/starlingx-0/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/swebster/starlingx-0/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" =============================================================================== Test Cases: =============================================================================== Test Scenario 1: Using a mixed IPv4/IPv6 lab (similar to the old lab_setup.vxlan.conf: PROVIDERNETS=" \ vlan|data0|${DATAMTU}|10-10|shared \ vxlan|data1|${DATAMTU}|600-615|tenant1 \ vxlan|data1|${DATAMTU}|700-731|shared \ vxlan|data1|${DATAMTU}|616-631|tenant2" EXTERNALPNET="vlan|data0|10" INTERNALPNET="vxlan|data1" VXLAN_GROUPS="239.0.1.10 ff05::1:10 239.0.1.11 ff05::1:11" VXLAN_PORTS="4789" DATA1_IPPOOLS="data1v4|192.168.59.0|24|sequential|192.168.59.2-192.168.59.10 data1v6|fd00:0:0:2::|64|random|fd00:0:0:2::2-fd00:0:0:2::a" COMPUTE0_IPADDRS="192.168.59.2/24,fd00:0:0:2::2/64" COMPUTE1_IPADDRS="192.168.59.3/24,fd00:0:0:2::3/64" DATA1_IPROUTES="0.0.0.0/0/192.168.59.1,::/0/fd00:0:0:2::1" 1.1 Creating a network on a segment using an IPv6 vxlan group still results in vxlan traversing over IPv4 - Not sure if this is a legit issue or not. Need to compare with old behaviour - The IP address for the compute node is IPv4 only. I think this is intentional looking at the code. We use the interface's primary address. Test Scenario 2: Using a purely IPv6 vxlan group, endpoint, and IP address pool: PROVIDERNETS=" \ vlan|data0|${DATAMTU}|10-10|shared \ vxlan|data1|${DATAMTU}|600-615|tenant1 \ vxlan|data1|${DATAMTU}|700-731|shared \ vxlan|data1|${DATAMTU}|616-631|tenant2" EXTERNALPNET="vlan|data0|10" INTERNALPNET="vxlan|data1" VXLAN_GROUPS="ff05::1:10" VXLAN_PORTS="4789" DATA1_IPPOOLS="data1v6|fd00:0:0:2::|64|random|fd00:0:0:2::2-fd00:0:0:2::a" COMPUTE0_IPADDRS="fd00:0:0:2::2/64" COMPUTE1_IPADDRS="fd00:0:0:2::3/64" DATA1_IPROUTES="::/0/fd00:0:0:2::1" 2.1 The neutron-openvswitch-agent does not start up. Logs indicate: 2018-10-26 21:24:18.636 16279 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Tunneling can't be enabled with invalid local_ip 'fd00:0:0:2::2'. IP couldn't be found on this host's interfaces. 2.1.1 The IPv6 address is not applied to the bridge correctly. Why? - puppet looks like it is applying the address, and no errors are reported 2018-10-29T17:47:40.750 ^[[0;36mDebug: 2018-10-29 17:47:40 +0000 Exec[ovs-add-address: br-phy1](provider=posix): Executing 'ip addr replace fd00:0:0:2::7/64 dev br-phy1'^[[0m - Looking at the system after it comes up, the IPv6 address is not on br-phy1 - Applying the same command puppet attempts results in the address being applied, and the agent can start up - The address is either not being applied properly by puppet, or something else is removing after puppet does the apply 2.2 When Problem 2.1 is resolved manually, two VMs are launched with network attachments to the tenant1 management network and tenant2 (vxlan) network 2.2.1 Can't DHCP for an IP address when the DHCP server resides on another host 2.2.2 Can't ARP for another host on the same network if it's not on the same host (goes along with 2.1) - Tracing the flows, it looks like broadcast/multicast is dropped at the tunnel bridge. - Perhaps we are missing some arp specific flows? Tracing the flows: compute-0:/home/wrsroot# ovs-appctl ofproto/trace br-int in_port=11,dl_dst=ff:ff:ff:ff:ff:ff,dl_src=fa:16:3e:f0:59:04,dl_type=0x0806,arp_spa=172.18.0.130 Flow: arp,in_port=11,vlan_tci=0x0000,dl_src=fa:16:3e:f0:59:04,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=172.18.0.130,arp_tpa=0.0.0.0,arp_op=0,arp_sha=00:00:00:00:00:00,arp_tha=00:00:00:00:00:00 bridge("br-int") ---------------- 0. arp,in_port=11, priority 10, cookie 0xc2870422c65008c goto_table:24 24. arp,in_port=11,arp_spa=172.18.0.130, priority 2, cookie 0xc2870422c65008c goto_table:25 25. in_port=11,dl_src=fa:16:3e:f0:59:04, priority 2, cookie 0xc2870422c65008c goto_table:60 60. priority 3, cookie 0xc2870422c65008c NORMAL -> no learned MAC for destination, flooding bridge("br-phy0") ----------------- 0. in_port=3, priority 2, cookie 0xb0a1694a46f053d8 drop bridge("br-tun") ---------------- 0. in_port=1, priority 1, cookie 0x2b39e37d0fc65ad2 goto_table:2 2. dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie 0x2b39e37d0fc65ad2 goto_table:22 22. priority 0, cookie 0x2b39e37d0fc65ad2 drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=11,vlan_tci=0x0000,dl_src=fa:16:3e:f0:59:04,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=172.18.0.130,arp_op=0 Datapath actions: push_vlan(vid=5,pcp=0),7,pop_vlan,15 compute-0:/home/wrsroot# ovs-ofctl dump-flows br-tun cookie=0x2b39e37d0fc65ad2, duration=1672.545s, table=0, n_packets=788, n_bytes=43540, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0x2b39e37d0fc65ad2, duration=1672.544s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x2b39e37d0fc65ad2, duration=1672.543s, table=2, n_packets=0, n_bytes=0, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0x2b39e37d0fc65ad2, duration=1672.543s, table=2, n_packets=788, n_bytes=43540, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0x2b39e37d0fc65ad2, duration=1672.542s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x2b39e37d0fc65ad2, duration=1446.631s, table=4, n_packets=0, n_bytes=0, priority=1,tun_id=0x2bc actions=mod_vlan_vid:1,resubmit(,10) cookie=0x2b39e37d0fc65ad2, duration=1382.480s, table=4, n_packets=0, n_bytes=0, priority=1,tun_id=0x268 actions=mod_vlan_vid:3,resubmit(,10) cookie=0x2b39e37d0fc65ad2, duration=1358.423s, table=4, n_packets=0, n_bytes=0, priority=1,tun_id=0x259 actions=mod_vlan_vid:4,resubmit(,10) cookie=0x2b39e37d0fc65ad2, duration=1348.419s, table=4, n_packets=0, n_bytes=0, priority=1,tun_id=0x269 actions=mod_vlan_vid:5,resubmit(,10) cookie=0x2b39e37d0fc65ad2, duration=1672.542s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x2b39e37d0fc65ad2, duration=1672.541s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x2b39e37d0fc65ad2, duration=1672.540s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x2b39e37d0fc65ad2,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0x2b39e37d0fc65ad2, duration=1672.540s, table=20, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,22) cookie=0x2b39e37d0fc65ad2, duration=1672.539s, table=22, n_packets=788, n_bytes=43540, priority=0 actions=drop