Observed BGP peering issues when Md5 was enabled

[Sanju Abraham]

On a setup with 2.20 build 64, Ubuntu 14.04.1, I saw issue with BGP peering with Md5 enabled.

Here are set of issues:

1-> fab provisioned with Md5 and key enabled and it completed successfuly
2-> Gateway (MX) Md5 configuration was using key-chains.
3-> BGP peering info for one of the control node has local_identifier set to "0.0.0.0" though IFMAP has the local identifier populated correctly.

** Note for (2) above Nischal suggested the solution is to use authentication-key in the protocol bgp group configuration instead of the key-chain bgp-auth. Using key-chain bgp-auth does not work with the TCP options used on standard linux distros.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12118
Submitter: Nischal Sheth (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12119
Submitter: Nischal Sheth (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12120
Submitter: Nischal Sheth (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12121
Submitter: Nischal Sheth (<email address hidden>)

[Nischal Sheth (nsheth)]

I suspect that the issue is orthogonal to MD5 configuration.
Have submitted a fix based on traces from the affected control-node
and examination of code.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12119
Committed: http://github.org/Juniper/contrail-controller/commit/120322ce39882b7dc0cb4a540eb3def3d7ec5101
Submitter: Zuul
Branch: master

commit 120322ce39882b7dc0cb4a540eb3def3d7ec5101
Author: Nischal Sheth <email address hidden>
Date: Tue Jun 30 16:48:53 2015 -0700

Fix corner case in IFMapDependencyTracker

Existing code prevents the same node from getting added to the change
list multiple times. This can cause problems if there's some mutual
dependency between the nodes.

In the case of this bug, a bgp-peering happened to be on the change
list, followed by the bgp-router object for the local control node.
The bgp-peering should have been added to the change list again as
part of the dependency evaluation for the bgp-router. However, it
was not added since it was already on the list. This caused an issue
because the local router id was not available when the bgp-peering
was evaluated. The local router id became available when bgp-router
was evaluated, but the bgp-peering was never evaluated again.

Change-Id: I54404ee8505d31aff49d153afd8b93488611c638
Closes-Bug: 1470256

[Sanju Abraham (asanju)]

Verified by taking the diff for this fix and building contrail-control.

Copied the binary on to the setup that had issue and restarted contrail-control so that the fix can kick in.

I can now see the peering established.

Fix holds good for this case.

[Sanju Abraham (asanju)]

As a workaround run the below query on each of the contrail-control nodes in the cluster.

curl -s http://localhost:8083/Snh_ShowBgpNeighborConfigReq | xmllint --format - | grep local_identifier | awk '{print $3}' | grep 0.0.0.0 | wc -l

if the above query returns a value greater than 0, then you need to do the following:

1-> Delete the Contrail-Control node from API / webUI
2-> Re-Add using API (/opt/contrail/utils/provision_control.py) or via the webUI

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12121
Committed: http://github.org/Juniper/contrail-controller/commit/fcdc384380b825a91ffaa251bd581459ada26a4a
Submitter: Zuul
Branch: R2.0

commit fcdc384380b825a91ffaa251bd581459ada26a4a
Author: Nischal Sheth <email address hidden>
Date: Tue Jun 30 16:48:53 2015 -0700

Fix corner case in IFMapDependencyTracker

Existing code prevents the same node from getting added to the change
list multiple times. This can cause problems if there's some mutual
dependency between the nodes.

In the case of this bug, a bgp-peering happened to be on the change
list, followed by the bgp-router object for the local control node.
The bgp-peering should have been added to the change list again as
part of the dependency evaluation for the bgp-router. However, it
was not added since it was already on the list. This caused an issue
because the local router id was not available when the bgp-peering
was evaluated. The local router id became available when bgp-router
was evaluated, but the bgp-peering was never evaluated again.

Change-Id: I54404ee8505d31aff49d153afd8b93488611c638
Closes-Bug: 1470256

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12118
Committed: http://github.org/Juniper/contrail-controller/commit/45ef3d6f3f3cf808ae3af778b96632d92a5fb193
Submitter: Zuul
Branch: R2.20

commit 45ef3d6f3f3cf808ae3af778b96632d92a5fb193
Author: Nischal Sheth <email address hidden>
Date: Tue Jun 30 16:48:53 2015 -0700

Fix corner case in IFMapDependencyTracker

Existing code prevents the same node from getting added to the change
list multiple times. This can cause problems if there's some mutual
dependency between the nodes.

In the case of this bug, a bgp-peering happened to be on the change
list, followed by the bgp-router object for the local control node.
The bgp-peering should have been added to the change list again as
part of the dependency evaluation for the bgp-router. However, it
was not added since it was already on the list. This caused an issue
because the local router id was not available when the bgp-peering
was evaluated. The local router id became available when bgp-router
was evaluated, but the bgp-peering was never evaluated again.

Change-Id: I54404ee8505d31aff49d153afd8b93488611c638
Closes-Bug: 1470256

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12120
Committed: http://github.org/Juniper/contrail-controller/commit/ed4053fcc8ad542b8a9ec6e9b42b84a675cf5cc8
Submitter: Zuul
Branch: R2.1

commit ed4053fcc8ad542b8a9ec6e9b42b84a675cf5cc8
Author: Nischal Sheth <email address hidden>
Date: Tue Jun 30 16:48:53 2015 -0700

Fix corner case in IFMapDependencyTracker

Existing code prevents the same node from getting added to the change
list multiple times. This can cause problems if there's some mutual
dependency between the nodes.

In the case of this bug, a bgp-peering happened to be on the change
list, followed by the bgp-router object for the local control node.
The bgp-peering should have been added to the change list again as
part of the dependency evaluation for the bgp-router. However, it
was not added since it was already on the list. This caused an issue
because the local router id was not available when the bgp-peering
was evaluated. The local router id became available when bgp-router
was evaluated, but the bgp-peering was never evaluated again.

Change-Id: I54404ee8505d31aff49d153afd8b93488611c638
Closes-Bug: 1470256