What I see is exactly the same behaviour Alex and I described before:
after making a snapshot in the middle of the case one of the controller nodes is 1-2 minutes behind others, which causes Keystone Ferten tokens to be treated as invalid.
According to the job console log time sync is done once for boostrap nodes, but not after a snapshot of an environment is taken later.
fuel-devops can't do a snapshot atomically, so it suspends nodes one by one first, which must cause this slight clock skew. Unlike cases when we revert a snapshot later, which enforces a time sync, save_load_environment action only makes a snapshot and resume the environment, time sync is skipped.
Ok, I took another look at one of the recent failures ( https:/ /product- ci.infra. mirantis. net/view/ 8.0_swarm/ job/8.0. system_ test.ubuntu. filling_ root/45/ consoleText )
What I see is exactly the same behaviour Alex and I described before:
after making a snapshot in the middle of the case one of the controller nodes is 1-2 minutes behind others, which causes Keystone Ferten tokens to be treated as invalid.
According to the job console log time sync is done once for boostrap nodes, but not after a snapshot of an environment is taken later.
fuel-devops can't do a snapshot atomically, so it suspends nodes one by one first, which must cause this slight clock skew. Unlike cases when we revert a snapshot later, which enforces a time sync, save_load_ environment action only makes a snapshot and resume the environment, time sync is skipped.