Comment 17 for bug 1525214

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Ok, I took another look at one of the recent failures ( https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.filling_root/45/consoleText )

What I see is exactly the same behaviour Alex and I described before:

after making a snapshot in the middle of the case one of the controller nodes is 1-2 minutes behind others, which causes Keystone Ferten tokens to be treated as invalid.

According to the job console log time sync is done once for boostrap nodes, but not after a snapshot of an environment is taken later.

fuel-devops can't do a snapshot atomically, so it suspends nodes one by one first, which must cause this slight clock skew. Unlike cases when we revert a snapshot later, which enforces a time sync, save_load_environment action only makes a snapshot and resume the environment, time sync is skipped.