From dcorch log we can see that dcorch got the following sync requests from its audit during the period of time in concern. Based on the algorithm, these logging points will trigger sync status update requests to dcmanager to update identity endpoint status.
Then from dcmanager log, we can see that the first 4 in dcorch match the first 3 Updatings in dcmanager. But the rest 8 requests in dcorch doesn't match the rest of 5Updatings in dcmanager. The sync states sequence in dcmanager should be:
From dcorch log we can see that dcorch got the following sync requests from its audit during the period of time in concern. Based on the algorithm, these logging points will trigger sync status update requests to dcmanager to update identity endpoint status.
[root@controller-0 log(keystone_ admin)] # zgrep "subcloud1\ /identity: Got .* sync request" /home/sysadmin/ andy_test/ dcorch. log.17. gz engine. sync_thread [-] subcloud1/identity: Got 0 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 1 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 6 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 0 sync request(s)
2019-12-04 23:02:28.244 2661230 INFO dcorch.
2019-12-04 23:02:29.586 2661230 INFO dcorch.
2019-12-04 23:02:29.737 2661230 INFO dcorch.
2019-12-04 23:02:30.402 2661230 INFO dcorch.
2019-12-04 23:09:07.830 2661230 INFO dcorch. engine. sync_thread [-] subcloud1/identity: Got 1 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 0 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 1 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 2 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 2 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 0 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 1 sync request(s) engine. sync_thread [-] subcloud1/identity: Got 0 sync request(s)
2019-12-04 23:09:07.992 2661230 INFO dcorch.
2019-12-04 23:09:08.142 2661230 INFO dcorch.
2019-12-04 23:09:08.411 2661230 INFO dcorch.
2019-12-04 23:09:08.775 2661230 INFO dcorch.
2019-12-04 23:09:09.119 2661230 INFO dcorch.
2019-12-04 23:09:09.457 2661230 INFO dcorch.
2019-12-04 23:09:10.027 2661230 INFO dcorch.
Then from dcmanager log, we can see that the first 4 in dcorch match the first 3 Updatings in dcmanager. But the rest 8 requests in dcorch doesn't match the rest of 5Updatings in dcmanager. The sync states sequence in dcmanager should be:
out-of-sync -> in-sync -> out-of-sync -> in-sync -> out-of-sync -> in-sync
And there are 2 "out-of-sync" updatings in dcmanager which shouldn't happend at all.
[root@controller-0 log(keystone_ admin)] # admin)] # grep "Updating subcloud:subcloud1 endpoint:identity sync" /home/sysadmin/ andy_test/ dcmanager. log manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync
[root@controller-0 log(keystone_
2019-12-04 23:02:28.260 728803 INFO dcmanager.
2019-12-04 23:02:29.601 728803 INFO dcmanager.
2019-12-04 23:02:30.419 728803 INFO dcmanager.
2019-12-04 23:09:09.303 728803 INFO dcmanager. manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:in-sync manager. subcloud_ manager [req-fbcd0a36- 5137-41a0- 8bb1-91062b72f8 b5 - - - - -] Updating subcloud:subcloud1 endpoint:identity sync:out-of-sync
2019-12-04 23:09:09.346 728803 INFO dcmanager.
2019-12-04 23:09:10.202 728803 INFO dcmanager.
2019-12-04 23:09:11.709 728803 INFO dcmanager.
2019-12-04 23:09:12.114 728803 INFO dcmanager.