BGPaaS: BGP session from VM can fail sometimes

[amit surana]

contrail 3.0-2697.

BGP peering to g/w IP and dns IP has been configured in the BGP VM. Both connections were up and routes were being sent. At some point, connection to g/w IP was torn down and subsequent SYN segments from the VM were dropped (Flow Queue Limit Exceeded). TCP connection to dns IP remained up and functional.

gcore is placed here: 10.84.5.112:/cs-shared/bugs/1533924/

flows are also dumped in the above directory. 1.1.3.3 is the VM IP, 1.1.3.1 is g/w and 1.1.3.2 is the dns. 172.16.80.1/.3 are the two control nodes.

[amit surana (asurana-t)]

Debugged this further and found that the bug happens if the BGP VM goes away without actually FIN/RSTing the TCP flow. Now, if the VM tries to establish a new TCP connection to the same destination, the flow gets stuck in Hold state and packets are silently discarded.

The right way to handle this would be to forward the new SYN to the CN and updating the VM->vRouter side of the flow with the new source port. Once the SYN hits the CN, nne of the two scenarios would play out:

1. If syn.seq is in window, CN tcp stack will respond back with a RST. This RST should be forwarded to the VM.
or
2. if syn.seq is out of the window, the CN tcp stack will respond back with an ACK (in an attempt to resynchronize the client). The BGP VM on seeing this ACK will RST the connection (doesn't expect ACK with wrong seq in SYN-sent state), which should be forwarded back to the CN. The RST.seq will be the same as ACK.ack, and so the CN tcp state will be cleaned up.

Either way, the idea is to end up synchronizing the source and destination hosts on the state of the TCP flow; in this case, it should be to abort the old flow and start a new one. After scenario 1/2 plays out, the BGP VM will be able to successfully setup a new peering session with the CN on the next attempt.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17238
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17615
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17615
Committed: http://github.org/Juniper/contrail-controller/commit/1b55ea89ad1bc93c0405a67c5a26adb036a1a8b2
Submitter: Zuul
Branch: R3.0

commit 1b55ea89ad1bc93c0405a67c5a26adb036a1a8b2
Author: Manish Singh <email address hidden>
Date: Tue Feb 23 12:30:39 2016 +0530

BGP session in BGPAAS fails sometimes.

Problem:
Say there is a forward flow (F1) with Nat'd reverse flow (F2).
For some reason session from VM to BGP server does not gets created and
client(VM) keeps sending sync messages. When the source port changes in sync
message after couple of re-tries, a new flow (F3) is to be created. The reverse
flow for F3 is still F2, as key is same for F2. Because of this update,F3 and F2
should get linked but it does not happen.
This happens because agent while creating F3 tries to add F2 again instead of
issuing a change. Due to this flow index manager tries to evict flow F2(because
its add for F2 and index is
present).This F2 does not get deleted as F3 is holding reference and in turn F2
is dangling in deleted state. Update never goes to vrouter.

Solution:
Evict rflow and let it get re-added.

Other than this change also handles partition enqueue to 0 for all port nat
flows (currently linklocal and bgp as service).

Closes-bug: 1533924

Conflicts:
 src/vnsw/agent/pkt/flow_proto.cc
 src/vnsw/agent/pkt/flow_proto.h
 src/vnsw/agent/pkt/flow_table.cc

Change-Id: I84ed4555cdd91902e85519546a7b540a814c7259

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17238
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17238
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17238
Committed: http://github.org/Juniper/contrail-controller/commit/ac2c838e522f1aaf64086a3c85fa07d7369c0e2c
Submitter: Zuul
Branch: master

commit ac2c838e522f1aaf64086a3c85fa07d7369c0e2c
Author: Manish Singh <email address hidden>
Date: Tue Feb 23 12:30:39 2016 +0530

BGP session in BGPAAS fails sometimes.

Problem:
Say there is a forward flow (F1) with Nat'd reverse flow (F2).
For some reason session from VM to BGP server does not gets created and
client(VM) keeps sending sync messages. When the source port changes in sync
message after couple of re-tries, a new flow (F3) is to be created. The reverse
flow for F3 is still F2, as key is same for F2. Because of this update,F3 and F2
should get linked but it does not happen.
This happens because agent while creating F3 tries to add F2 again instead of
issuing a change. Due to this flow index manager tries to evict flow F2(because
its add for F2 and index is
present).This F2 does not get deleted as F3 is holding reference and in turn F2
is dangling in deleted state. Update never goes to vrouter.

Solution:
Evict rflow and let it get re-added.

Other than this change also handles partition enqueue to 0 for all port nat
flows (currently linklocal and bgp as service).

Conflicts:
 src/vnsw/agent/pkt/flow_proto.cc
 src/vnsw/agent/pkt/flow_proto.h
 src/vnsw/agent/pkt/flow_table.cc

(cherry picked from commit 7ed1372ced5e52d94b676da6e0e4ebbfe7cb75e9)
Closes-bug: 1533924
Change-Id: I84ed4555cdd91902e85519546a7b540a814c7259