Comment 40 for bug 1755649

Reviewed: https://review.opencontrail.org/43549
Committed: http://github.com/Juniper/contrail-sandesh/commit/8e9f02f85446e92236031c2f7bce2b22fca6e977
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 8e9f02f85446e92236031c2f7bce2b22fca6e977
Author: zcui <email address hidden>
Date: Mon Jun 4 14:04:43 2018 -0700

contrail-collector crash immediately after provisioning

root cause:
Race condition problem:
To state_machine_,
(1) alloced by sandesh_connection.
(2) used by generator
When problem happen, generator receive Resource update message,
and enqueue resouece update to state_machine_, at same time,
update stats immedietly. This action will try to get mutex
sometime, it will lead CPU yield. We call this as thread 1.
At same time, connection close is triggered, and destructor
function will be triggered. And destructure will call termial
and all memory will be released related to this connection.
We call this as thread 2.
When thread 2 finished and thread 1 go ahead, crash will happen.

Solution:
Designer of state_machine should consider this problem. So state
Machine destructure is separated two steps:
(1) call terminal to free memory alloced by its substruct.
(2) start a timer to free state machine self.
Between step1 and step2, deleted_ is used to check state machine
can be used or not.
We add a shutdown fucntion for stats structure to pass this state.

Closes-Bug: 1755649

Change-Id: I599461f0a37adc21d2b68a5ca20d66ccaf4f6e51