default route in default vrf in VPP

[Onong Tayeng]

I have an openstack setup consisting of multiple tenant networks with their distinct vrfs in VPP.

For external network connectivity, default route is set in each tenant’s vrf.

SNAT is configured on the relevant interfaces in VPP.

Do I need to set a default route in the default vrf (vrf 0) in VPP for L3 traffic to flow in/out of the VMs?

What I have observed so far is that traffic does not flow until and unless I add the default route in the default vrf in VPP. I have tested against VPP 17.10, 18.01 and 18.04 RC2.

[Onong Tayeng (onong)]

Neale's reply:

Hi Onong,

For packets to the VMSs I would say that post the out2in translation the packet should be injected into the tenant’s VRF, so no route needs to be added to the outside/default VRF. I don’t know how you are adding you NAT state, but look for the ‘VRF-ID’ configurations.
For packets from the VMs, I assume your default route points to the interface through which the Internet is reachable and so no look-up in the default VRF is needed.

/neale

[Onong Tayeng (onong)]

Hi Neale,

Please find the API trace attached.

Also, here are some other details that I thought might be helpful.

vpp# sh int
              Name Idx State Counter Count
TenGigabitEthernet7/0/0 1 up rx packets 77
                                                     rx bytes 7557
                                                     tx packets 68
                                                     tx bytes 9246
TenGigabitEthernet7/0/0.1013 4 up rx packets 77
                                                     rx bytes 7557
                                                     tx packets 68
                                                     tx bytes 9246
local0 0 down
loop0 3 up rx packets 503
                                                     rx bytes 28324
                                                     tx packets 2
                                                     tx bytes 84
                                                     drops 503
                                                     ip4 357
loop1 6 up rx packets 3
                                                     rx bytes 694
                                                     drops 3
                                                     ip4 2
tap0 2 up rx packets 1981
                                                     rx bytes 141370
                                                     tx packets 1
                                                     tx bytes 42
                                                     drops 1478
                                                     ip4 651
                                                     ip6 5
tapcli-0 5 up rx packets 68
                                                     rx bytes 8974
                                                     tx packets 77
                                                     tx bytes 7249
                                                     drops 3
vpp#

vpp# sh int addr
TenGigabitEthernet7/0/0 (up):
  192.168.11.30/24
TenGigabitEthernet7/0/0.1013 (up):
  l2 bridge bd_id 4 shg 0
local0 (dn):
loop0 (up):
  l2 bridge bd_id 2 bvi shg 0
  10.195.95.141/27
loop1 (up):
  l2 bridge bd_id 4 bvi shg 0
  10.0.0.1/16 table 1
tap0 (up):
  l2 bridge bd_id 2 shg 0
tapcli-0 (up):
  l2 bridge bd_id 4 shg 0
vpp#

 ...

[Onong Tayeng (onong)]

API trace file to reproduce the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/562721

[Onong Tayeng (onong)]

Here's the final say on the matter.

-----------------------------------------------------------------------------------

Hi Matus,

Thank you for the clarification.

Onong,

Given this NAT behaviour, you need to add the default route in to the default table.

/neale
-----------------------------------------------------------------------------------

-----------------------------------------------------------------------------------
Hi Neale,

NAT currently support only one outside FIB table. Be default is used table 0 or you can set it in NAT startup config parameter “outside VRF id” https://wiki.fd.io/view/VPP/NAT#Startup_config

Matus
-----------------------------------------------------------------------------------

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/562721
Committed: https://git.openstack.org/cgit/openstack/networking-vpp/commit/?id=e985ce7bd52deafb2d00953eded1dd37dc509b1d
Submitter: Zuul
Branch: master

commit e985ce7bd52deafb2d00953eded1dd37dc509b1d
Author: Onong Tayeng <email address hidden>
Date: Thu Apr 19 21:21:04 2018 +0530

    Add default route in default vrf in VPP

    The current VPP NAT implementation supports only one outside FIB table
    and by default it uses table 0, ie, the default vrf. So, this is a
    temporary workaround to tide over the limitation.

    Change-Id: Ia5babf5c7ad207468cb4ad262c012fcc7c72c8f5
    Closes-Bug: #1765068

[Michael Qiu (qdy220091330)]

What I see is that tap0 should be the outer interface, but how does the traffic flow go outside? tap0 indeed a virtio interface, it should belongs to a VM. I'm a bit confused. Who could show me the details?

[Onong Tayeng (onong)]

Hi,

The way it is supposed to work is as follows:

1. Create a linux bridge (say, br-ex) on the controller/network node.
2. Plumb the external NIC interface (say, eth0) on the controller/network node under br-ex.
3. Create tapv2 interface (say, tap0) in VPP.
3. Plumb tap0's other end (say, vpp_ext_tap) under br-ex, on the controller/network node.

Please note that tapv2 interfaces are supported from VPP 18.01 onwards only. Here's the command to do that:

create tap host-if-name vpp_ext_tap host-bridge br-ex rx-ring-size 1024 tx-ring-size 1024
set interface state tap0 up

Here is the overall L3 diagram which might provide more clarity:

http://paste.openstack.org/show/722705/

Please do let me know if you have more questions.

Happy to help.

--
Onong