Sequence number for route are incremented even when allowed-address pair is not configured

[Praveen]

In current design, when agent sees an IP on two different hosts, agent internally treats the addresses to be in active-backup mode. As a result it updates the preference for route when it sees traffic on the interface. This design is used for fast switchover in case of VM migration.

In one of the setup, there was a bug and an instance was spawned on two compute nodes. This resulted in agent treating them as ECMP and treat them as active-backup pairs. Since both instances are active VRouter is seeing traffic from both the instances and active-backup state is spinning between the two compute nodes. Since allowed-address pair is not configured in this case, ideally, agent should not have treated them as active-backup and also avoid incrementing the sequence numbers.

The mail describing the issue is copied below.

----

nova is running the same VM in both boa-001-06 and 07:

in bka-001-06 pid 8791 is executing instance -uuid 6313840c-9068-4b13-8e84-f69925ffd0be
in bka-001-07 pid 9126 is executing instance -uuid 6313840c-9068-4b13-8e84-f69925ffd0be

root@bka-001-02:~# (source openrc; nova show 6313840c-9068-4b13-8e84-f69925ffd0be)
+--------------------------------------+--------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | bka-001-07 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | bka-001-07.bka.cloud. |
| OS-EXT-SRV-ATTR:instance_name | instance-0000b975 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |

[…]
| example-net network | 10.0.0.10, 37.44.0.127 |

This means that both the private IP address (10.0.0.10) and the floating-ip (34.44.0.127) are being advertised by both compute-nodes.
Contrail is then triggering the code that manage the allowed-address pair feature (and vm migration). These feature works by allow us to detect the system that most recently sent traffic claiming to have the specific address by incrementing a sequence number on the route when they see they are not the preferred system.

Both compute nodes are increase the sequence number on the route and claiming the route. They are able to do this quite fast.

The floating-ip address is then re-originated into 800+ stale snatdebug instances and 30+ functional snat instances. These updates are then pushed to all compute nodes… when the compute nodes receive the route updates they re-examined the flows… The flows for this specific VM are constantly being re-examined. That is why you see the agent being a bit slow.

BGP as the message bus seems to be pushing several thousand updates per second… the agent is able to process them and continuously re-examine flows. But that affects its response time to new flows.

I’d recommend that you terminate that instance. I believe that will stop the route update storm…

Praveen, can you please file a bug regarding the sequence number behavior… ? There are two independent issues: this VM didn’t enable allowed-address pair so we shouldn’t really be increasing the sequence number. And when we increase the sequence number there should be an exponential back-off on how fast we update it.

  Pedro.

[Hari Prasad Killi (haripk)]

Backoff : https://review.opencontrail.org/#/c/11413/

[Naveen N (naveenn)]

Fixed with review
 https://review.opencontrail.org/13214
and
https://review.opencontrail.org/13134