vCenter Contrail 5.0B94: TSN HA setup is not providing a DHCP to BM

[Luke Bockelmann]

incorrect entries were made in TSN-1 so after shutting down TSN-2 agent container, TSN-1 started working:

[root@5b3s35-csn-1 /]# nh --get 33
Id:33 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:2
              Flags:Valid, Multicast, Etree Root,
              Sub NH(label): 27(0) 29(0) 32(0)

Id:27 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Tor, Etree Root,
              Sub NH(label):

Id:29 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Evpn, Etree Root,
              Sub NH(label): 21(6)

Id:21 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:d0 07 ca 2c bd c0 00 50 56 98 39 d9 08 00
              Sip:192.168.0.80 Dip:5.5.5.10

Id:32 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Fabric, Etree Root,
              Sub NH(label): 31(4618)

Id:31 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0
              Flags:Valid, MPLSoGRE, Etree Root,
              Oif:0 Len:14 Data:00 77 56 aa bb 02 00 50 56 98 39 d9 08 00
              Sip:192.168.0.80 Dip:192.168.0.70

Setup can be found here on slide 5: https://docs.google.com/presentation/d/16na-zEHJ1kgTNnMKTH3hCWOKtQss629Ig2eYBoSC_pY/edit#slide=id.g39eacedcc6_1_62

Currently, 5b4s26 is the BM being tested. You can also use 5b4s27 who is currently having the problem.

[Luke Bockelmann (h-luke)]

this is reproducible easily with enabling the csn-2 and running dhclient on 5b4s26

[chhandak (chhandak)]

When we have 2 TSN server in agent.conf ... vrouter is not having QFX VTEP in composite nexthop list of brocast route. WHen there is only one entry things are fine.

Agent conf

[CONTROL-NODE]
servers=192.168.0.10:5269 192.168.0.11:5269 192.168.0.12:5269

[DEFAULT]
collectors=10.87.64.17:8086 10.87.121.104:8086 10.87.121.105:8086
log_file=/var/log/contrail/contrail-vrouter-agent.log
log_level=SYS_NOTICE
log_local=1

xmpp_dns_auth_enable=False
xmpp_auth_enable=False

physical_interface_mac = 00:50:56:98:39:d9
agent_mode = tsn-no-forwarding
tsn_servers = 192.168.0.80,192.168.0.81 ------> 2 entry is casuing the problem.

[root@5b3s35-csn-1 /]# rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index DestMac Flags Label/VNID Nexthop Stats
31264 0:0:5e:0:1:0 DfEc - 3 0
38184 0:50:56:98:39:d9 DfEc - 3 0
50152 0:25:90:7e:f:64 Ec - 1 0
112924 ff:ff:ff:ff:ff:ff LDfEc 6 40 21
115240 2:0:0:0:0:2 DfEc - 12 0
182984 0:50:56:98:d8:41 LDfEc 6 33 0
205564 2:0:0:0:0:1 DfEc - 12 0
216348 0:25:90:7e:10:b0 LEc 6 32 0-->BMS
[root@5b3s35-csn-1 /]# nh --get 40
Id:40 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:2
              Flags:Valid, Multicast, Etree Root,
              Sub NH(label): 36(0) 39(0)

Id:36 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Tor, Etree Root,
              Sub NH(label):----->5.5.5.10 is missing

Id:39 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Fabric, Etree Root,
              Sub NH(label): 23(4618)

Id:23 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0
              Flags:Valid, MPLSoGRE, Etree Root,
              Oif:0 Len:14 Data:00 77 56 aa bb 02 00 50 56 98 39 d9 08 00
              Sip:192.168.0.80 Dip:192.168.0.70

[root@5b3s35-csn-1 /]# nh --get 32
Id:32 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:d0 07 ca 2c bd c0 00 50 56 98 39 d9 08 00
              Sip:192.168.0.80 Dip:5.5.5.10

[Manish Singh (manishs)]

try this:
tsn_servers = 192.168.0.80 192.168.0.81

instead of
tsn_servers = 192.168.0.80,192.168.0.81

[Luke Bockelmann (h-luke)]

no it did not work:

root@5b4s26:~# ifup p4p1
Internet Systems Consortium DHCP Client 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/p4p1/00:25:90:7e:10:b0
Sending on LPF/p4p1/00:25:90:7e:10:b0
Sending on Socket/fallback
DHCPDISCOVER on p4p1 to 255.255.255.255 port 67 interval 3 (xid=0x65e6d761)
DHCPDISCOVER on p4p1 to 255.255.255.255 port 67 interval 3 (xid=0x65e6d761)
DHCPDISCOVER on p4p1 to 255.255.255.255 port 67 interval 4 (xid=0x65e6d761)
DHCPDISCOVER on p4p1 to 255.255.255.255 port 67 interval 6 (xid=0x65e6d761)

[Luke Bockelmann (h-luke)]

DHCPOFFER of 20.20.20.3 from 20.20.20.2
DHCPACK of 20.20.20.3 from 20.20.20.2
bound to 20.20.20.3 -- renewal in 2147483648 seconds.
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 1
Failed to bring up p4p1.
root@5b4s26:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:25:90:7e:10:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.87.64.154/25 brd 10.87.64.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fe7e:10b1/64 scope link
       valid_lft forever preferred_lft forever
3: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:25:90:7e:10:b0 brd ff:ff:ff:ff:ff:ff
    inet 20.20.20.3/24 brd 20.20.20.255 scope global p4p1
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fe7e:10b0/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:7c:0b:1e:28 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
root@5b4s26:~# ping 20.20.20.2
PING 20.20.20.2 (20.20.20.2) 56(84) bytes of data.
64 bytes from 20.20.20.2: icmp_seq=1 ttl=64 time=0.849 ms
64 bytes from 20.20.20.2: icmp_seq=2 ttl=64 time=0.590 ms

[Luke Bockelmann (h-luke)]

please check because I not seeing the entries in the L2 tables of either TSN

[root@5b3s35-csn-1 /]# rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index DestMac Flags Label/VNID Nexthop Stats
31264 0:0:5e:0:1:0 DfEc - 3 0
38184 0:50:56:98:39:d9 DfEc - 3 0
66588 0:50:56:98:80:b4 LDfEc 6 42 0
77728 2:e0:76:a2:4:85 LDfEc 6 40 0
85460 2:de:b7:26:72:74 LDfEc 6 39 0
101884 0:50:56:98:95:fc LDfEc 6 43 0
112924 ff:ff:ff:ff:ff:ff LDfEc 6 44 0
115240 2:0:0:0:0:2 DfEc - 12 0
151536 2:2:31:92:a4:75 LDfEc 6 36 0
182984 0:50:56:98:d8:41 LDfEc 6 41 0
205564 2:0:0:0:0:1 DfEc - 12 0

[root@5b3s35-csn-2 /]# rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index DestMac Flags Label/VNID Nexthop Stats
31264 0:0:5e:0:1:0 DfEc - 3 0
66588 0:50:56:98:80:b4 LDfEc 6 19 0
77728 2:e0:76:a2:4:85 LDfEc 6 26 0
85460 2:de:b7:26:72:74 LDfEc 6 21 0
101884 0:50:56:98:95:fc LDfEc 6 20 0
115240 2:0:0:0:0:2 DfEc - 12 0
151536 2:2:31:92:a4:75 LDfEc 6 27 0
182984 0:50:56:98:d8:41 LDfEc 6 18 0
205564 2:0:0:0:0:1 DfEc - 12 0
226356 0:50:56:98:bd:e4 DfEc - 3 0

[Nagendra E S (esnagendra)]

From: Nagendra E S <email address hidden>
Date: Friday, 27 July 2018 at 17:25
Subject: Re: Bug# 1777562

Hi Luke,

If you are seeing DHCP issue, please do let me know.

Thanks,

[Jeba Paulaiyan (jebap)]

Manishkn has testbed in this state

[Nagendra E S (esnagendra)]

testbed info not available.

[manishkn (manishkn)]

provided the testbed with problem state

[Sivakumar Ganapathy (hotlava51)]

The problem is seen only on few nodes and debug is in progress but it is unlikely to make it to 5.0.1 as RCA is not known yet. Hence tracking it for 5.1.0.

[N Anand Rao (anandrao79)]

Hello Richard,

Me and Ashok are trying to understand the usecases for fat flow enhancements requested as part of https://bugs.launchpad.net/juniperopenstack/+bug/1752618.
We had few questions regarding the same:-

  1. Can you please elaborate more on the usecase for configuring fat flow prefixes?

In particular do you expect us to handle cases like below i.e

Lets say 10.0.0.0/24 subnet configured as fat flow source prefix for protocol TCP and port 22

and 10.0.0.0/16 subnet being configured as fat flow source prefix for protocol UDP and port 53

and both these rules are configured in the same VN?

Do we need to support such usecases? Is it ok to *not* allow overlapping subnet configuration?

  1. NAT implications – We think there won’t be any kind of NATing done on the fat flows as it may not work with FAT flows. Any kind of NAT would be done separately. Correct?

Please help clarify the above so that we can proceed with blueprint and implementation.
We are also ok to discuss these in a call with you. Please let us know.

Thanks & Regards,
Anand

[richard roberts (robric35)]

Sorry for not being responsive this week. I have been hijacked by ps to sipport the cisco vepc integration and this is all my time.

Can we have a conf call on Monday as this is a complex topic, objective is to find the right ratio of colplexity vs use case coverage.

1/ i think things will be simpler than that - but potentially yes. Use case is steering or sg (multiple policy)
2/ let’s talk . I think that depends on which nat

Rdepends

Envoyé de mon iPhone

Le 10 août 2018 à 19:24, Anand Narayanan Rao <<email address hidden><mailto:<email address hidden>>> a écrit :

Hello Richard,

Me and Ashok are trying to understand the usecases for fat flow enhancements requested as part of https://bugs.launchpad.net/juniperopenstack/+bug/1752618.
We had few questions regarding the same:-

  1. Can you please elaborate more on the usecase for configuring fat flow prefixes?

In particular do you expect us to handle cases like below i.e

Lets say 10.0.0.0/24 subnet configured as fat flow source prefix for protocol TCP and port 22

and 10.0.0.0/16 subnet being configured as fat flow source prefix for protocol UDP and port 53

and both these rules are configured in the same VN?

Do we need to support such usecases? Is it ok to *not* allow overlapping subnet configuration?

  1. NAT implications – We think there won’t be any kind of NATing done on the fat flows as it may not work with FAT flows. Any kind of NAT would be done separately. Correct?

Please help clarify the above so that we can proceed with blueprint and implementation.
We are also ok to discuss these in a call with you. Please let us know.

Thanks & Regards,
Anand

[N Anand Rao (anandrao79)]

Sure Richard. I will setup a meeting for Monday.

Regards,
Anand

On Aug 10, 2018 11:29 PM, Richard Roberts <email address hidden> wrote:
Sorry for not being responsive this week. I have been hijacked by ps to sipport the cisco vepc integration and this is all my time.

Can we have a conf call on Monday as this is a complex topic, objective is to find the right ratio of colplexity vs use case coverage.

1/ i think things will be simpler than that - but potentially yes. Use case is steering or sg (multiple policy)
2/ let’s talk . I think that depends on which nat

Rdepends

Envoyé de mon iPhone

Le 10 août 2018 à 19:24, Anand Narayanan Rao <<email address hidden><mailto:<email address hidden>>> a écrit :

Hello Richard,

Me and Ashok are trying to understand the usecases for fat flow enhancements requested as part of https://bugs.launchpad.net/juniperopenstack/+bug/1752618.

We had few questions regarding the same:-

  1. Can you please elaborate more on the usecase for configuring fat flow prefixes?

In particular do you expect us to handle cases like below i.e

Lets say 10.0.0.0/24 subnet configured as fat flow source prefix for protocol TCP and port 22

and 10.0.0.0/16 subnet being configured as fat flow source prefix for protocol UDP and port 53

and both these rules are configured in the same VN?

Do we need to support such usecases? Is it ok to *not* allow overlapping subnet configuration?

  1. NAT implications – We think there won’t be any kind of NATing done on the fat flows as it may not work with FAT flows. Any kind of NAT would be done separately. Correct?

Please help clarify the above so that we can proceed with blueprint and implementation.

We are also ok to discuss these in a call with you. Please let us know.

Thanks & Regards,

Anand

[Sandip Dey (sandipd)]

Could anyone take a look at it?
I have the setup in problem state.

Setup:
Control nodes:nodec4,nodec5,nodec6
compute:10.204.216.181,10.204.216.182,10.204.216.183
TSN:nodea26

Regards
Sandip

logs
----

(vrouter-agent)[root@nodea26 /]$ rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index DestMac Flags Label/VNID Nexthop Stats
31264 0:0:5e:0:1:0 DfEc - 3 0
66528 0:50:56:a4:e6:81 LDfEc 4 16 0
73752 0:25:90:e4:8:e5 LDfEc 4 24 0
110040 0:25:90:aa:9:b7 DfEc - 7 0
112924 ff:ff:ff:ff:ff:ff LDfEc 4 35 108
115240 2:0:0:0:0:2 DfEc - 7 0
144972 0:0:5e:0:1:1 LDfEc 4 25 0
199308 80:71:1f:c0:38:f0 LDfEc 4 25 0
205564 2:0:0:0:0:1 DfEc - 7 0
221688 0:25:90:e4:8:e4 Ec - 1 0

(vrouter-agent)[root@nodea26 /]$ nh --get 24
Id:24 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Ecmp, Etree Root,
              Valid Hash Key Parameters: Proto,SrcIP,SrcPort,DstIp,DstPort
              Sub NH(label): 23(4) 22(4)

Id:23 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:88 a2 5e 88 7f 31 00 25 90 aa 09 b7 08 00
              Sip:192.168.0.26 Dip:172.16.0.22

Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:88 a2 5e 88 7f 31 00 25 90 aa 09 b7 08 00
              Sip:192.168.0.26 Dip:172.16.0.33

(vrouter-agent)[root@nodea26 /]$ nh --get 35
Id:35 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:2
              Flags:Valid, Multicast, Etree Root,
              Sub NH(label): 28(0) 32(0) 31(0)

Id:28 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Tor, Etree Root,
              Sub NH(label): >>>>>>>>>>>>>>>

Id:32 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Evpn, Etree Root,
              Sub NH(label): 23(4) 22(4)

Id:23 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:88 a2 5e 88 7f 31 00 25 90 aa 09 b7 08 00
              Sip:192.168.0.26 Dip:172.16.0.22

Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:8...

[manishkn (manishkn)]

Non center model

:ocata-5.0-263: We are good now.

TSN HA non-vcenter cluster

TSN1:
servers=172.17.90.4:5269

[DEFAULT]
collectors=10.87.74.155:8086
log_file=/var/log/contrail/contrail-vrouter-agent.log
log_level=SYS_NOTICE
log_local=1

xmpp_dns_auth_enable=False
xmpp_auth_enable=False

physical_interface_mac = 52:54:00:da:19:1b
agent_mode = tsn-no-forwarding
tsn_servers = 172.17.90.10 172.17.90.5

[SANDESH]
introspect_ssl_enable=False
sandesh_ssl_enable=False

TSN2:

[CONTROL-NODE]
servers=172.17.90.4:5269

[DEFAULT]
collectors=10.87.74.155:8086
log_file=/var/log/contrail/contrail-vrouter-agent.log
log_level=SYS_NOTICE
log_local=1

xmpp_dns_auth_enable=False
xmpp_auth_enable=False

physical_interface_mac = 52:54:00:a9:8c:5c
agent_mode = tsn-no-forwarding
tsn_servers = 172.17.90.10 172.17.90.5

[SANDESH]
introspect_ssl_enable=False
sandesh_ssl_enable=False

(vrouter-agent)[root@TSN159 /]$ rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index DestMac Flags Label/VNID Nexthop Stats
31264 0:0:5e:0:1:0 DfEc - 3 0
112924 ff:ff:ff:ff:ff:ff LDfEc 6 25 2
115240 2:0:0:0:0:2 DfEc - 7 0
164612 5a:80:4d:ca:45:f2 LEc 6 17 2
202568 52:54:0:da:19:1b DfEc - 7 0
205564 2:0:0:0:0:1 DfEc - 7 0

(vrouter-agent)[root@TSN159 /]$ nh --get 17
Id:17 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:c0 42 d0 44 06 00 52 54 00 da 19 1b 08 00
              Sip:172.17.90.5 Dip:3.3.3.3

(vrouter-agent)[root@TSN159 /]$ !tcp
tcpdump -nei eth1 udp port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:23:44.236180 c0:42:d0:44:10:05 > 52:54:00:da:19:1b, ethertype IPv4 (0x0800), length 110: 3.3.3.3.4773 > 172.17.90.5.4789: VXLAN, flags [I] (0x08), vni 6
5a:80:4d:ca:45:f2 > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.1.1.2 tell 10.1.1.10, length 46
19:23:45.400021 c0:42:d0:44:10:05 > 52:54:00:da:19:1b, ethertype IPv4 (0x0800), length 392: 3.3.3.3.fiveacross > 172.17.90.5.4789: VXLAN, flags [I] (0x08), vni 6
5a:80:4d:ca:45:f2 > Broadcast, ethertype IPv4 (0x0800), length 342: 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 5a:80:4d:ca:45:f2, length 300
19:23:45.400578 52:54:00:da:19:1b > c0:42:d0:44:06:00, ethertype IPv4 (0x0800), length 372: 172.17.90.5.65194 > 3.3.3.3.4789: VXLAN, flags [I] (0x08), vni 6
52:54:00:da:19:1b > 5a:80:4d:ca:45:f2, ethertype IPv4 (0x0800), length 322: 10.1.1.2.bootps > 10.1.1.10.bootpc: BOOTP/DHCP, Reply, length 280
19:23:45.400916 c0:42:d0:44:10:05 > 52:54:00:da:19:1b, ether...

[Sandip Dey (sandipd)]

Queens-5.0-274

CSN HA is working

DHCP resolved for BM
1.Brought down one CSN-DHCP resolved for BM
2.Brought down the other CSN,brought up the first one - DHCP resolved for BM
3.Ping between vm and BM –passed

[Sandip Dey (sandipd)]

Not sure which commit fixed it, marking the bug as invalid