From dcorch log we can see that dcorch got the following sync requests from its audit during the period of time in concern. Based on the algorithm, these logging points will trigger sync status update requests to dcmanager to update identity endpoint status. 

[root@controller-0 log(keystone_admin)]# zgrep "subcloud1\/identity: Got .* sync request" /home/sysadmin/andy_test/dcorch.log.17.gz
2019-12-04 23:02:28.244 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 0 sync request(s)
2019-12-04 23:02:29.586 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 1 sync request(s)
2019-12-04 23:02:29.737 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 6 sync request(s)
2019-12-04 23:02:30.402 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 0 sync request(s)

2019-12-04 23:09:07.830 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 1 sync request(s)
2019-12-04 23:09:07.992 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 0 sync request(s)
2019-12-04 23:09:08.142 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 1 sync request(s)
2019-12-04 23:09:08.411 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 2 sync request(s)
2019-12-04 23:09:08.775 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 2 sync request(s)
2019-12-04 23:09:09.119 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 0 sync request(s)
2019-12-04 23:09:09.457 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 1 sync request(s)
2019-12-04 23:09:10.027 2661230 INFO dcorch.engine.sync_thread [-] subcloud1/identity: Got 0 sync request(s)

Then from dcmanager log, we can see that the first 4 in dcorch match the first 3 Updatings in dcmanager. But the rest 8 requests in dcorch doesn't match the rest of 5Updatings in dcmanager. The sync states sequence in dcmanager should be:

out-of-sync -> in-sync -> out-of-sync -> in-sync -> out-of-sync -> in-sync

And there are 2 "out-of-sync" updatings in dcmanager which shouldn't happend at all.

[root@controller-0 log(keystone_admin)]#
[root@controller-0 log(keystone_admin)]# grep "Updating subcloud:subcloud1 endpoint:identity sync" /home/sysadmin/andy_test/dcmanager.log
2019-12-04 23:02:28.260 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync
2019-12-04 23:02:29.601 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync
2019-12-04 23:02:30.419 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync

2019-12-04 23:09:09.303 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync
2019-12-04 23:09:09.346 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync
2019-12-04 23:09:10.202 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync
2019-12-04 23:09:11.709 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync
2019-12-04 23:09:12.114 728803 INFO dcmanager.manager.subcloud_manager [req-fbcd0a36-5137-41a0-8bb1-91062b72f8b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync