Detailed bug description:
While devops revert the environment, he tries to sync time via ssh and ntpdate on each nodes in the environment.
The ntp serrver was choosed as first ntp server entry in /etc/ntp.conf
On the isolated node, we haven't access to public internet network, and ntpdate returns error code
Steps to reproduce:
1. Install fuel master 8.0 with ha openstack cluster
2. Install octane
3. Create backup
4. Reinstall fuel master node with 9.x release
5. Install octane
6. Restore
7. Start upgrade procedure via octane
8. ...
9. octane upgrade-node --isolated
10. Create snapshot
11. Revert snapshot
Expected results:
Reverting snapshot was successfully
Actual result:
Traceback (most recent call last):
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/helpers/decorators.py", line 121, in wrapper
result = func(*args, **kwargs)
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/tests/tests_upgrade/test_os_upgrade.py", line 116, in upgrade_db
self.env.revert_snapshot("upgrade_first_cic")
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/models/environment.py", line 307, in revert_snapshot
self.sync_time()
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/models/environment.py", line 139, in sync_time
new_time = sync_time(self.d_env, nodes_names, skip_sync)
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
return func(*args, **kwargs)
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 43, in sync_time
g_ntp.do_sync_time(g_ntp.pacemaker_ntps)
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 327, in do_sync_time
ntp.set_actual_time()
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 116, in set_actual_time
self._node_name))
File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 111, in wait
raise TimeoutError(timeout_msg)
TimeoutError: Failed to set actual time on node u'slave-03'
Reproducibility:
100%
Workaround:
Before run upgrade-node --siolated, changed in fuel-qa test ntp server to the fuel master ip /etc/ntp.donf on target node
After upgrade, revert the changes in /etc/ntp.conf
OR
before running the command, make check, if node is in isolated mode, run ntpdate command with fuel master ip argument master
Impact:
Upgrade procedure
Description of the environment:
fuel-octane-9.0.0-1.mos1312.noarch
Additional information:
TimeoutError: Failed to set actual time on node u'slave-03'
Fix proposed to branch: stable/7.0
Review: https:/