If multiple updates are received for the same object, schema transformer should not reprocess everything again

[Sachin Bansal]

Currently, if schema transformer (or any config daemon) receives an update on an object, regardless of anything of interested being changed on an object or not, it runs dependency tracker again on the object, and reprocesses everything. If an object is connected to a large number of objects, this could result in unnecessary reprocessing multiple times.

As an example, lets say, VN object changed at t0, t1, t2, t3, t4 time points. Schema transformer started would run dependency tracker at t0, and start processing. Lets say, that processing finishes at t5 (after t4). In the current code, we will run dependency tracker 4 more times (once each for t1, t2, t3 and t4), even though running only once would have sufficed.

As a solution, we should track the fields schema transformer is interested in, and when an update is received, it should read those (and only those) fields for the object, compare against previously cached values and if no change is detected, then terminate dependency tracker and subsequent reprocessing.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29898
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29898
Committed: http://github.org/Juniper/contrail-controller/commit/6cc4359925a6f062afb31626a0dd5996698ae626
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6cc4359925a6f062afb31626a0dd5996698ae626
Author: Sachin Bansal <email address hidden>
Date: Wed Mar 22 16:58:57 2017 -0700

Add support for diff calculation in dependency tracker

It is possible that processing an object can take a long time in schema
transformer. During this time, many updates to the same objects could
be received from rabbitmq. All those changes would be processed with the
one update only. We should have a way to ignore the updates where no
change is detected.

With this change, we are adding support for specifying ref/prop fields
for each object type and on update, we compare these fields from
previously cached values. If no change is detected, we terminate the
dependency tracker process.

Closes-Bug: 1674514

Change-Id: I0b1244e5473ded2f085874f2f6bcdade74a5fe63

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30031
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30031
Committed: http://github.org/Juniper/contrail-controller/commit/8e01e933769118d51c0b256b707c48d42831beb6
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 8e01e933769118d51c0b256b707c48d42831beb6
Author: Sachin Bansal <email address hidden>
Date: Wed Mar 22 16:58:57 2017 -0700

Add support for diff calculation in dependency tracker

It is possible that processing an object can take a long time in schema
transformer. During this time, many updates to the same objects could
be received from rabbitmq. All those changes would be processed with the
one update only. We should have a way to ignore the updates where no
change is detected.

With this change, we are adding support for specifying ref/prop fields
for each object type and on update, we compare these fields from
previously cached values. If no change is detected, we terminate the
dependency tracker process.

Closes-Bug: 1674514

Change-Id: I0b1244e5473ded2f085874f2f6bcdade74a5fe63
(cherry picked from commit 6cc4359925a6f062afb31626a0dd5996698ae626)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30226
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30226
Committed: http://github.org/Juniper/contrail-controller/commit/46c36f7079f5fc1a3087a9cd7c8a7bb3a1a4addd
Submitter: Zuul (<email address hidden>)
Branch: master

commit 46c36f7079f5fc1a3087a9cd7c8a7bb3a1a4addd
Author: Sachin Bansal <email address hidden>
Date: Thu Apr 6 16:54:02 2017 -0700

Fixed logic to find out if anything changed during update

Change-Id: I786a21876b4c198a71d6a7f6d2e7d0fc30c28029
Closes-Bug: 1674514