[3.0-2715~kilo]No ECMP seen for a dest. route when multiple port-tuples are added to the same SI

[Ganesha HV]

1]. Single node setup -nodeb12
2]. Created a Service chain between l-vn(10.10.10.0/24) and r-vn(20.20.20.0/24) adding si1 as the service.
3]. In si1, added two port-tuples as below :

{
-service-instance: {
-port_tuples: [
-{
-to: [
"default-domain"
"admin"
"si1"
"si1-port-tuple1-44c0c34b-c03a-42b0-9782-492d9fd3d957"
]
href: "http://nodeb12:8082/port-tuple/bcd766ab-a779-407a-bceb-3b6f1d5133bf"
uuid: "bcd766ab-a779-407a-bceb-3b6f1d5133bf"
}
-{
-to: [
"default-domain"
"admin"
"si1"
"si1-port-tuple0-92c2143d-404f-4db8-894f-434c7071d083"
]
href: "http://nodeb12:8082/port-tuple/ba383fa2-c3cf-4042-9500-df1171907cb8"
uuid: "ba383fa2-c3cf-4042-9500-df1171907cb8"
}
]
-fq_name: [
"default-domain"
"admin"
"si1"
]
uuid: "8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce"
parent_type: "project"
parent_uuid: "8f0a723b-4f89-4e68-b187-f96f9ccd3d75"
parent_href: "http://nodeb12:8082/project/8f0a723b-4f89-4e68-b187-f96f9ccd3d75"
-service_instance_properties: {
management_virtual_network: "default-domain:admin:m-vn"
right_virtual_network: "default-domain:admin:r-vn"
-interface_list: [
-{
virtual_network: "default-domain:admin:m-vn"
}
-{
virtual_network: "default-domain:admin:l-vn"
}
-{
virtual_network: "default-domain:admin:r-vn"
}
]
left_virtual_network: "default-domain:admin:l-vn"
}
-perms2: {
owner: null
owner_access: 7
global_access: 0
share: [ ]
}
href: "http://nodeb12:8082/service-instance/8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce"
-instance_ip_refs: [
-{
-to: [
"8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce-left-v4"
]
href: "http://nodeb12:8082/instance-ip/7fefe397-ba2b-4d5c-a139-d4f96252be4b"
-attr: {
interface_type: "left"
}
uuid: "7fefe397-ba2b-4d5c-a139-d4f96252be4b"
}
-{
-to: [
"8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce-right-v4"
]
href: "http://nodeb12:8082/instance-ip/9ca19eaf-fe51-4ebc-bb9e-1d9186588fe0"
-attr: {
interface_type: "right"
}
uuid: "9ca19eaf-fe51-4ebc-bb9e-1d9186588fe0"
}
-{
-to: [
"8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce-management-v4"
]
href: "http://nodeb12:8082/instance-ip/b3a579e1-4d4d-4c5a-8a6f-1853e7491fe7"
-attr: {
interface_type: "management"
}
uuid: "b3a579e1-4d4d-4c5a-8a6f-1853e7491fe7"
}
]
-id_perms: {
enable: true
-uuid: {
uuid_mslong: 10321739945995882000
uuid_lslong: 10413514900560114000
}
created: "2016-02-23T14:36:47.613526"
description: null
creator: null
user_visible: true
last_modified: "2016-02-23T14:36:48.115261"
-permissions: {
owner: "cloud-admin"
owner_access: 7
other_access: 7
group: "cloud-admin-group"
group_access: 7
}
}
display_name: "si1"
-service_template_refs: [
-{
-to: [
"default-domain"
"pt2"
]
href: "http://nodeb12:8082/service-template/2897e758-07e2-4e50-9250-926ca4c259fc"
attr: null
uuid: "2897e758-07e2-4e50-9250-926ca4c259fc"
}
]
name: "si1"
}
}
4]. I expect the route to dest.VM(20.20.20.3) to have 2 equi-cost paths.

5]. I see in the l-vn VRF that there are 2 paths, but one of them is chosen best.

The setup is in that state.

[Manish Singh (manishs)]

In the setup 20.20.20.3 was published in r-vn by agent.
However in l-vn only one path has been added by SI and thats why ECMP is not formed.
http://10.204.216.5:8083/Snh_ShowRouteReq?x=default-domain:admin:l-vn:service-560b4e6b-d929-46ad-89a8-e50f79425692-default-domain_admin_si1.inet.0

20.20.20.3/32

2016-Feb-23 14:38:04.512240

paths
protocol last_modified local_preference local_as peer_as peer_router_id source as_path next_hop label replicated primary_table secondary_tables communities origin_vn flags tunnel_encap sequence_no origin_vn_path pmsi_tunnel load_balance cluster_list

ServiceChain

 2016-Feb-23 14:38:04.512240

 100

 0

 0

 - - -
      10.204.216.5
     24

false

secondary_tables
bgp.l3vpn.0
default-domain:admin:l-vn:l-vn.inet.0

communities
accept-own-nexthop
encapsulation:gre
encapsulation:udp
secgroup:64512:8000001
originvn:64512:5

default-domain:admin:r-vn

flags
None

tunnel_encap
gre
udp
 -
origin_vn_path
originvn:64512:5

pmsi_tunnel
type -
ar_type -
identifier -
label

0

flags
flags

load_balance
decision_type -
fields
fields

cluster_list

[Manish Singh (manishs)]

CN sent message to agent.

2016-02-23 20:08:04.514 AgentXmppMessage: Received xmpp message from: 10.204.216.5 Port 5269 Size: 6323 Packet: <?xml version="1.0"?> <message <email address hidden>" to="nodeb12/bgp-peer"> <event xmlns="http://jabber.org/protocol/pubsub"> <items node="1/1/default-domain:admin:l-vn:l-vn"> <item id="20.20.20.4/32"> <entry> <nlri> <af>1</af> <safi>1</safi> <address>20.20.20.4/32</address> </nlri> <next-hops> <next-hop> <af>1</af> <address>10.204.216.5</address> <label>24</label> <tunnel-encapsulation-list> <tunnel-encapsulation>gre</tunnel-encapsulation> <tunnel-encapsulation>udp</tunnel-encapsulation> </tunnel-encapsulation-list> <virtual-network>default-domain:admin:r-vn</virtual-network> </next-hop> </next-hops> <version>1</version> <virtual-network>default-domain:admin:r-vn</virtual-network> <sequence-number>0</sequence-number> <security-group-list> <security-group>8000001</security-group> </security-group-list> <community-tag-list> <community-tag>accept-own-nexthop</community-tag> </community-tag-list> <local-preference>100</local-preference> <med>200</med> <load-balance> <load-balance-fields> <load-balance-field-list>l3-source-address</load-balance-field-list> <load-balance-field-list>l3-destination-address</load-balance-field-list> <load-balance-field-list>l4-protocol</load-balance-field-list> <load-balance-field-list>l4-source-port</load-balance-field-list> <load-balance-field-list>l4-destination-port</load-balance-field-list> </load-balance-fields> <load-balance-decision>field-hash</load-balance-decision> </load-balance> </entry> </item> <item id="20.20.20.3/32"> <entry> <nlri> <af>1</af> <safi>1</safi> <address>20.20.20.3/32</address> </nlri> <next-hops> <next-hop> <af>1</af> <address>10.204.216.5</address> <label>24</label> <tunnel-encapsulation-list> <tunnel-encapsulation>gre</tunnel-encapsulation> <tunnel-encapsulation>udp</tunnel-encapsulation> </tunnel-encapsulation-list> <virtual-network>default-domain:admin:r-vn</virtual-network> </next-hop> </next-hops> <version>1</version> <virtual-network>default-domain:admin:r-vn</virtual-network> <sequence-number>0</sequence-number> <security-group-list> <security-group>8000001</security-group> </security-group-list> <community-tag-list> <community-tag>accept-own-nexthop</community-tag> </community-tag-list> <local-preference>100</local-preference> <med>200</med> <load-balance> <load-balance-fields> <load-balance-field-list>l3-source-address</load-balance-field-list> <load-balance-field-list>l3-destination-address</load-balance-field-list> <load-balance-field-list>l4-protocol</load-balance-field-list> <load-balance-field-list>l4-source-port</load-balance-field-list> <load-balance-field-list>l4-destination-port</load-balance-field-list> </load-balance-fields> <load-balance-decision>field-hash</load-balance-decision> </load-balance> </entry> </item>

[Rudra Rugge (rrugge)]

There is another bug filed by Vedu assigned to the agent team wherein the incorrect IP address gets used by the VM. Could this be related to that?

[Rudra Rugge (rrugge)]

Could you also confirm if the service chain IP is attached to both the interfaces (both left should have one service chain ip left and both right should have service chain ip right)

[Ganesha HV (ganeshahv)]

I do see the same service ip getting attached to both the interfaces.

{
-virtual-machine-interface: {
-routing_instance_refs: [
-{
-to: [
"default-domain"
"admin"
"l-vn"
"l-vn"
]
href: "http://nodeb12:8082/routing-instance/5055fb0c-f5bb-425b-bb92-7fca315b3f31"
-attr: {
direction: "both"
protocol: null
ipv6_service_chain_address: null
dst_mac: null
mpls_label: null
vlan_tag: null
src_mac: null
service_chain_address: null
}
uuid: "5055fb0c-f5bb-425b-bb92-7fca315b3f31"
}
]
-virtual_machine_interface_mac_addresses: {
-mac_address: [
"02:84:94:ad:87:ae"
]
}
-virtual_machine_interface_bindings: {
-key_value_pair: [
-{
key: "host_id"
value: "nodeb12"
}
-{
key: "vif_type"
value: "vrouter"
}
-{
key: "vnic_type"
value: "normal"
}
]
}
parent_href: "http://nodeb12:8082/project/8f0a723b-4f89-4e68-b187-f96f9ccd3d75"
parent_type: "project"
href: "http://nodeb12:8082/virtual-machine-interface/8494ad87-aea6-4190-8f45-58f27a51fe06"
-virtual_machine_refs: [
-{
-to: [
"a49ba409-8373-4ebe-ae27-06f394b14fab"
]
href: "http://nodeb12:8082/virtual-machine/a49ba409-8373-4ebe-ae27-06f394b14fab"
attr: null
uuid: "a49ba409-8373-4ebe-ae27-06f394b14fab"
}
]
-id_perms: {
enable: true
-uuid: {
uuid_mslong: 9553451507821986000
uuid_lslong: 10323755519274254000
}
creator: null
created: "2016-02-23T14:35:15.738630"
user_visible: true
last_modified: "2016-02-23T14:38:04.327640"
-permissions: {
owner: "cloud-admin"
owner_access: 7
other_access: 7
group: "cloud-admin-group"
group_access: 7
}
description: null
}
virtual_machine_interface_device_owner: "compute:nova"
-vrf_assign_table: {
-vrf_assign_rule: [
-{
routing_instance: "default-domain:admin:l-vn:l-vn"
-match_condition: {
-src_port: {
end_port: 65535
start_port: 0
}
-src_address: {
security_group: null
-subnet: {
ip_prefix: "10.10.10.5"
ip_prefix_len: 32
}
virtual_network: null
subnet_list: [ ]
network_policy: null
}
ethertype: null
dst_address: null
-dst_port: {
end_port: 65535
start_port: 0
}
protocol: "any"
}
vlan_tag: null
ignore_acl: false
}
-{
routing_instance: "default-domain:admin:l-vn:l-vn"
-match_condition: {
-src_port: {
end_port: 65535
start_port: 0
}
-src_address: {
security_group: null
-subnet: {
ip_prefix: "10.10.10.6"
ip_prefix_len: 32
}
virtual_network: null
subnet_list: [ ]
network_policy: null
}
ethertype: null
dst_address: null
-dst_port: {
end_port: 65535
start_port: 0
}
protocol: "any"
}
vlan_tag: null
ignore_acl: false
}
-{
routing_instance: "default-domain:admin:l-vn:service-560b4e6b-d929-46ad-89a8-e50f79425692-default-domain_admin_si1"
-match_condition: {
-src_port: {
end_port: -1
start_port: -1
}
src_address: null
ethertype: null
dst_address: null
-dst_port: {
end_port: -1
start_port: -1
}
protocol: "any"
}
vlan_tag: null
ignore_acl: true
}
]
}
display_name: "8494ad87-aea6-4190-8f45-58f27a51fe06"
-instance_ip_back_refs: [
-{
-to: [
"8f3e2fcb-410b-4e3e-9084-3ca0ff1f72ce-left-v4"
]
href: "http://nodeb12:8082/instance-ip/7fefe397-ba2b-4d5c-a139-d4f96252be4b"
attr: null
uuid: "7fefe397-ba2b-4d5c-a139-d4f96252be4b"
}
-{
-to: [
"96a0a5f4-c650-43ca-a1ab-f41911194020"
]
href: "http://nodeb12:8082/instance-ip/96a0a5f4-c650-43ca-a1ab-f41911194020"
attr: null
uuid: "96a0a5f4-c650-43ca-a1ab-f41911194020"
}
]
n...

[Hari Prasad Killi (haripk)]

https://bugs.launchpad.net/juniperopenstack/+bug/1548942 is fixed now to avoid giving service instance IP. Please recheck.

[Ganesha HV (ganeshahv)]

This bug isn't similar to https://bugs.launchpad.net/juniperopenstack/+bug/1548942
I was able to see the service chain IP is attached to both the interfaces

[Manish Singh (manishs)]

instance-ip-mode should be set to active-active for ECMP. Same is not seen below.

<node type="instance-ip"> <name>1bfb97d9-f984-4a62-bdb4-114e1fe0e904-left-v4</name> <instance-ip-address>2.2.2.4</instance-ip-address> <instance-ip-family>v4</instance-ip-family> <secondary-ip-tracking-ip> <ip-prefix>5.5.5.0</ip-prefix> <ip-prefix-len>24</ip-prefix-len> </secondary-ip-tracking-ip> <instance-ip-secondary>true</instance-ip-secondary> <service-instance-ip>true</service-instance-ip> <id-perms> <permissions> <owner>cloud-admin</owner> <owner-access>7</owner-access> <group>cloud-admin-group</group> <group-access>7</group-access> <other-access>7</other-access> </permissions> <uuid> <uuid-mslong>13435570936671783831</uuid-mslong> <uuid-lslong>11779353053692278231</uuid-lslong> </uuid> <enable>true</enable> <created>2016-04-06T06:37:20</created> <last-modified>2016-04-06T06:37:21</last-modified> <description></description> <user-visible>true</user-visible> <creator></creator> </id-perms> <perms2> <owner></owner> <owner-access>7</owner-access> <global-access>0</global-access> </perms2> <display-name>1bfb97d9-f984-4a62-bdb4-114e1fe0e904-left-v4</display-name> </node>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19914
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19915
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19915
Committed: http://github.org/Juniper/contrail-controller/commit/45ea31bdab6acc68377bb752ce86cbf786d23ea4
Submitter: Zuul
Branch: R3.0

commit 45ea31bdab6acc68377bb752ce86cbf786d23ea4
Author: Rudra Rugge <email address hidden>
Date: Wed May 4 17:01:48 2016 -0700

Active-standby and active-active for port tuple

Default service chain ip ha mode as active-active.
If allowed address pair is configured then copy the address mode
from AAP to service chain ip.

Change-Id: I8627f146a10342aa9a75bc529b5810b8cdaa50bd
Closes-bug: #1548920

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19914
Committed: http://github.org/Juniper/contrail-controller/commit/59db0796f0cb513fb677d8d33282891b2c2532aa
Submitter: Zuul
Branch: master

commit 59db0796f0cb513fb677d8d33282891b2c2532aa
Author: Rudra Rugge <email address hidden>
Date: Wed May 4 17:01:48 2016 -0700

Active-standby and active-active for port tuple

Default service chain ip ha mode as active-active.
If allowed address pair is configured then copy the address mode
from AAP to service chain ip.

Change-Id: I8627f146a10342aa9a75bc529b5810b8cdaa50bd
Closes-bug: #1548920

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20163
Committed: http://github.org/Juniper/contrail-controller/commit/1c236b5ed2b43e190326d0f875a7b8e65f9639c3
Submitter: Zuul
Branch: master

commit 1c236b5ed2b43e190326d0f875a7b8e65f9639c3
Author: Rudra Rugge <email address hidden>
Date: Thu May 12 09:56:17 2016 -0700

Port tuple list modification during iip udpate

Copy the list of instance-ip uuids from vmi so that
any modification in the iterations does not cause
set change size issues.

Also fix issue with reseting of service-chain-ip
flag.

Change-Id: I9332bc371cd4d7534d488e76186e930c4bfe1ccb
Closes-Bug: 1581118
Closes-bug: 1548920

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20164
Committed: http://github.org/Juniper/contrail-controller/commit/0c1364610cd856e784d6a5244f2cc42fa3dd9fcc
Submitter: Zuul
Branch: R3.0

commit 0c1364610cd856e784d6a5244f2cc42fa3dd9fcc
Author: Rudra Rugge <email address hidden>
Date: Thu May 12 09:56:17 2016 -0700

Port tuple list modification during iip udpate

Copy the list of instance-ip uuids from vmi so that
any modification in the iterations does not cause
set change size issues.

Also fix issue with reseting of service-chain-ip
flag.

Change-Id: I9332bc371cd4d7534d488e76186e930c4bfe1ccb
Closes-Bug: 1581118
Closes-bug: 1548920