"Vrouter operation failed" errors when old tcp flows are cleared

[Vedamurthy Joshi]

R2.20 111 Ubuntu 14.04 Juno multi-node

On this setup, 20.1.1.4 did ssh to 10.204.219.67 using router snat.

The tcp sessions were closed and the user removed the router gateway. Then we see these errors.

2015-11-30 Mon 15:09:51:648.804 IST nodek1 [Thread 139891291694848, Pid 16272]: VrResponseMsg Error: Invalid argument
2015-11-30 Mon 15:09:51:648.955 IST nodek1 [Thread 139891291694848, Pid 16272]: VRouter operation failed. Error <22:Invalid argument>. Object <Flow : 3044372 with Source IP: 20.1.1.4 Source port: 39290 Destination IP: 10.204.219.67 Destination port: 22 Protocol 6>. Operation <Deletion>. Message number :3457
2015-11-30 Mon 15:09:51:649.008 IST nodek1 [Thread 139891253909248, Pid 16272]: VRouter [SYS_ERR]: VRouterError: VRouter operation failed. Error < 22 : Invalid argument >. Object < Flow : 3044372 with Source IP: 20.1.1.4 Source port: 39290 Destination IP: 10.204.219.67 Destination port: 22 Protocol 6 >. Operation < Deletion >. Message number : 3457 controller/src/ksync/ksync_object.cc 436
2015-11-30 Mon 15:09:51:649.041 IST nodek1 [Thread 139891291694848, Pid 16272]: VrResponseMsg Error: Invalid argument
2015-11-30 Mon 15:09:51:649.262 IST nodek1 [Thread 139891291694848, Pid 16272]: VRouter operation failed. Error <22:Invalid argument>. Object <Flow : 1897768 with Source IP: 10.204.219.67 Source port: 22 Destination IP: 20.1.1.4 Destination port: 39290 Protocol 6>. Operation <Deletion>. Message number :3459
2015-11-30 Mon 15:09:51:649.338 IST nodek1 [Thread 139891291694848, Pid 16272]: VrResponseMsg Error: Invalid argument
2015-11-30 Mon 15:09:51:649.501 IST nodek1 [Thread 139891291694848, Pid 16272]: VRouter operation failed. Error <22:Invalid argument>. Object <Flow : 3315392 with Source IP: 20.1.1.4 Source port: 39292 Destination IP: 10.204.219.67 Destination port: 22 Protocol 6>. Operation <Deletion>. Message number :3461
2015-11-30 Mon 15:09:51:649.694 IST nodek1 [Thread 139891291694848, Pid 16272]: VrResponseMsg Error: Invalid argument
2015-11-30 Mon 15:09:51:649.805 IST nodek1 [Thread 139891291694848, Pid 16272]: VRouter operation failed. Error <22:Invalid argument>. Object <Flow : 1289844 with Source IP: 10.204.219.67 Source port: 22 Destination IP: 20.1.1.4 Destination port: 39292 Protocol 6>. Operation <Deletion>. Message number :3465
2015-11-30 Mon 15:09:51:649.362 IST nodek1 [Thread 139891253909248, Pid 16272]: VRouter [SYS_ERR]: VRouterError: VRouter operation failed. Error < 22 : Invalid argument >. Object < Flow : 1897768 with Source IP: 10.204.219.67 Source port: 22 Destination IP: 20.1.1.4 Destination port: 39290 Protocol 6 >. Operation < Deletion >. Message number : 3459 controller/src/ksync/ksync_object.cc 436
2015-11-30 Mon 15:09:51:650.234 IST nodek1 [Thread 139891253909248, Pid 16272]: VRouter [SYS_ERR]: VRouterError: VRouter operation failed. Error < 22 : Invalid argument >. Object < Flow : 3315392 with Source IP: 20.1.1.4 Source port: 39292 Destination IP: 10.204.219.67 Destination port: 22 Protocol 6 >. Operation < Deletion >. Message number : 3461 controller/src/ksync/ksync_object.cc 436
2015-11-30 Mon 15:09:51:650.534 IST nodek1 [Thread 139891253909248, Pid 16272]: VRouter [SYS_ERR]: VRouterError: VRouter operation failed. Error < 22 : Invalid argument >. Object < Flow : 1289844 with Source IP: 10.204.219.67 Source port: 22 Destination IP: 20.1.1.4 Destination port: 39292 Protocol 6 >. Operation < Deletion >. Message number : 3465 controller/src/ksync/ksync_object.cc 436

root@nodek1:/var/log/contrail#
root@nodek1:/var/log/contrail# fg

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15731
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15731
Committed: http://github.org/Juniper/contrail-controller/commit/2dee9b425afb9bdb655e7e7ed9ef55d3ef0cc8c5
Submitter: Zuul
Branch: R2.20

commit 2dee9b425afb9bdb655e7e7ed9ef55d3ef0cc8c5
Author: Hari <email address hidden>
Date: Thu Dec 10 11:43:25 2015 +0530

Changing the error level for the ksync syslog messages.

Messages are sent when flow eviction happens and vrouter rejects
agent operations on earlier flow. Changing them to debug level.
Also, change the error changes to relevant messages.

Change-Id: Icda0b1c81422dcf908a14771190e5c4a66ed5e2a
partial-bug: 1521115

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15873
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15873
Committed: http://github.org/Juniper/contrail-controller/commit/5cae3672ed7039b2185d76b02130ed840d1dd699
Submitter: Zuul
Branch: R2.20

commit 5cae3672ed7039b2185d76b02130ed840d1dd699
Author: Hari <email address hidden>
Date: Thu Dec 17 16:06:50 2015 +0530

Ignore error response when flow is already in deleted state.

Change-Id: I2889fc9b40513080e3cc911fc7198bf62dd3a2dc
related-bug: 1521115

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16261
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16261
Committed: http://github.org/Juniper/contrail-controller/commit/9efd53f86f7af606035821d400b16721a28754e5
Submitter: Zuul
Branch: master

commit 9efd53f86f7af606035821d400b16721a28754e5
Author: Hari <email address hidden>
Date: Thu Dec 10 11:43:25 2015 +0530

Changing the error level for the ksync syslog messages.

Messages are sent when flow eviction happens and vrouter rejects
agent operations on earlier flow. Changing them to debug level.
Also, change the error changes to relevant messages.

partial-bug: 1521115
(cherry picked from commit 2dee9b425afb9bdb655e7e7ed9ef55d3ef0cc8c5)

Conflicts:
 src/ksync/ksync_sock.cc

Ignore error response when flow is already in deleted state.

related-bug: 1521115
(cherry picked from commit 5cae3672ed7039b2185d76b02130ed840d1dd699)

Conflicts:
 src/vnsw/agent/vrouter/ksync/flowtable_ksync.h

Change-Id: Id9891282d136f42323b3a319d39b3c539cb12c04