Activity log for bug #1580685

Date Who What changed Old value New value Message
2016-05-11 16:36:39 Nischal Sheth bug added bug
2016-05-11 16:36:54 Nischal Sheth nominated for series juniperopenstack/r2.21.x
2016-05-11 16:36:54 Nischal Sheth bug task added juniperopenstack/r2.21.x
2016-05-11 16:36:54 Nischal Sheth nominated for series juniperopenstack/r3.0
2016-05-11 16:36:54 Nischal Sheth bug task added juniperopenstack/r3.0
2016-05-11 16:36:54 Nischal Sheth nominated for series juniperopenstack/trunk
2016-05-11 16:36:54 Nischal Sheth bug task added juniperopenstack/trunk
2016-05-11 16:36:54 Nischal Sheth nominated for series juniperopenstack/r2.20
2016-05-11 16:36:54 Nischal Sheth bug task added juniperopenstack/r2.20
2016-05-11 16:36:54 Nischal Sheth nominated for series juniperopenstack/r2.22.x
2016-05-11 16:36:54 Nischal Sheth bug task added juniperopenstack/r2.22.x
2016-05-11 16:37:02 Nischal Sheth juniperopenstack/r2.20: importance Undecided High
2016-05-11 16:37:03 Nischal Sheth juniperopenstack/r2.21.x: importance Undecided High
2016-05-11 16:37:05 Nischal Sheth juniperopenstack/r2.22.x: importance Undecided High
2016-05-11 16:37:07 Nischal Sheth juniperopenstack/r3.0: importance Undecided High
2016-05-11 16:37:15 Nischal Sheth juniperopenstack/r2.20: assignee Tapan Karwa (tkarwa)
2016-05-11 16:37:21 Nischal Sheth juniperopenstack/r2.21.x: assignee Tapan Karwa (tkarwa)
2016-05-11 16:37:29 Nischal Sheth juniperopenstack/r3.0: assignee Tapan Karwa (tkarwa)
2016-05-11 16:37:35 Nischal Sheth juniperopenstack/r2.22.x: assignee Tapan Karwa (tkarwa)
2016-05-11 16:56:31 Nischal Sheth description TBD Existing implementation attempts to reconnect to the same IFMap server 2 times in the event of connection close/failure. If it fails to reconnect, it uses the other IFMap server obtained from discovery. If the IFMap Server is restarted or crashes and comes back up quickly, CN may succeed in connecting to it during the 2 attempts. The IFMap Server will likely not have received the entire config database from API server at that point. Hence CN will receive an incomplete config database and perform an audit, thus deleting valid config objects temporarily. This may cause the CN to stop publishing itself to discovery and/or cause vRouter agents to get unnecessary deletes for config objects, thus causing traffic disruption for tenants. Note that the above behavior happens because IFMap Server doesn't have the ability to refuse incoming connections from CN till it's config database is fully populated. API server sets IFMap Server to oper down in discovery but that doesn't help in above scenario since CN doesn't consult discovery before reconnecting to same IFMap Server. Proposed fix is to not try to reconnect to same IFMap Server. This should fix the problem as long as the other IFMap Server has the complete config database. This should be the same if multiple IFMap Servers are not being restarted at the same time or one after the other.
2016-05-11 16:56:51 Nischal Sheth summary Do not connect to same IFMap server after connection flap Do not connect to same IFMap server after connection goes down
2016-05-11 17:20:56 Nischal Sheth description Existing implementation attempts to reconnect to the same IFMap server 2 times in the event of connection close/failure. If it fails to reconnect, it uses the other IFMap server obtained from discovery. If the IFMap Server is restarted or crashes and comes back up quickly, CN may succeed in connecting to it during the 2 attempts. The IFMap Server will likely not have received the entire config database from API server at that point. Hence CN will receive an incomplete config database and perform an audit, thus deleting valid config objects temporarily. This may cause the CN to stop publishing itself to discovery and/or cause vRouter agents to get unnecessary deletes for config objects, thus causing traffic disruption for tenants. Note that the above behavior happens because IFMap Server doesn't have the ability to refuse incoming connections from CN till it's config database is fully populated. API server sets IFMap Server to oper down in discovery but that doesn't help in above scenario since CN doesn't consult discovery before reconnecting to same IFMap Server. Proposed fix is to not try to reconnect to same IFMap Server. This should fix the problem as long as the other IFMap Server has the complete config database. This should be the same if multiple IFMap Servers are not being restarted at the same time or one after the other. Existing implementation attempts to reconnect to the same IFMap server 2 times in the event of connection close/failure. If it fails to reconnect, it uses the other IFMap server obtained from discovery. If the IFMap Server is restarted or crashes and comes back up quickly, CN may succeed in connecting to it during the 2 attempts. The IFMap Server will likely not have received the entire config database from API server at that point. Hence CN will receive an incomplete config database and perform an audit, thus deleting valid config objects temporarily. This may cause the CN to stop publishing itself to discovery and/or cause vRouter agents to get unnecessary deletes for config objects, thus causing traffic disruption for tenants. Note that the above behavior happens because IFMap Server doesn't have the ability to refuse incoming connections from CN till it's config database is fully populated. API server sets IFMap Server to oper down in discovery but that doesn't help in above scenario since CN doesn't consult discovery before reconnecting to same IFMap Server. Proposed fix is to not try to reconnect to same IFMap Server. This should fix the problem as long as the other IFMap Server has the complete config database. This should be the case if multiple IFMap Servers are not being restarted at the same time or one after the other.
2016-05-11 23:12:33 OpenContrail Admin juniperopenstack/r2.21.x: status New In Progress
2016-05-11 23:12:36 OpenContrail Admin juniperopenstack/r2.20: status New In Progress
2016-05-11 23:15:35 OpenContrail Admin juniperopenstack/r2.22.x: status New In Progress
2016-05-11 23:15:38 OpenContrail Admin juniperopenstack/r3.0: status New In Progress
2016-05-11 23:18:19 OpenContrail Admin juniperopenstack/trunk: status New In Progress
2016-05-13 16:23:09 OpenContrail Admin juniperopenstack/trunk: status In Progress Fix Committed
2016-05-13 16:23:11 OpenContrail Admin juniperopenstack/trunk: milestone r3.1.0.0-fcs
2016-05-13 16:41:26 OpenContrail Admin juniperopenstack/r3.0: status In Progress Fix Committed
2016-05-13 16:41:28 OpenContrail Admin juniperopenstack/r3.0: milestone r3.0.2.0
2016-05-13 18:44:21 Jeba Paulaiyan juniperopenstack/r2.22.x: importance High Medium
2016-05-13 23:51:04 OpenContrail Admin juniperopenstack/r2.22.x: status In Progress Fix Committed
2016-05-13 23:51:05 OpenContrail Admin juniperopenstack/r2.22.x: milestone r2.22.2
2016-05-16 04:34:54 OpenContrail Admin juniperopenstack/r2.20: status In Progress Fix Committed
2016-05-16 04:34:56 OpenContrail Admin juniperopenstack/r2.20: milestone r2.23
2016-05-16 04:51:44 OpenContrail Admin juniperopenstack/r2.21.x: status In Progress Fix Committed
2016-05-16 04:51:45 OpenContrail Admin juniperopenstack/r2.21.x: milestone r2.21.2
2016-07-07 18:24:45 Nischal Sheth juniperopenstack/r2.22.x: importance Medium High