Deployment tests fails with sync time on admin node

[Stanislaw Bogatkin]

System tests for changes on CI failed due to connectivity problems. Examples:

https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2929/
https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2932/
https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2934/
https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2936/

Logs from console:
2015-11-20 13:13:04,386 - INFO fuel_web_client.py:816 -- Get nailgun node by slave-02 devops node
2015-11-20 13:13:04,594 - INFO fuel_web_client.py:816 -- Get nailgun node by slave-03 devops node
2015-11-20 13:13:04,786 - INFO fuel_web_client.py:816 -- Get nailgun node by slave-04 devops node
2015-11-20 13:13:05,033 - INFO fuel_web_client.py:816 -- Get nailgun node by slave-05 devops node
2015-11-20 13:13:05,237 - INFO environment.py:474 -- Synchronizing time on Fuel admin node
2015-11-20 13:15:45,750 - INFO environment.py:474 -- Synchronizing time on Fuel admin node
2015-11-20 13:18:26,306 - INFO environment.py:474 -- Synchronizing time on Fuel admin node
2015-11-20 13:20:07,113 - INFO fuel_web_client.py:1032 -- Wait for task 600 seconds:
...
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/tests/test_neutron.py", line 182, in deploy_neutron_vlan_ha
    self.env.revert_snapshot("ready_with_5_slaves")
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/models/environment.py", line 311, in revert_snapshot
    self.sync_time(nailgun_nodes)
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/helpers/decorators.py", line 364, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/models/environment.py", line 476, in sync_time
    g_ntp.do_sync_time()
  File "/home/jenkins/workspace/fuel-main/env_ubuntu-master/fuelweb_test/helpers/ntp.py", line 83, in do_sync_time
    " \n{0}".format(self.report_not_synchronized()))
AssertionError: Time on nodes was not set:
[('admin', ['Wed Nov 18 17:28:50 UTC 2015\n'])]
...

sys_test.log:
...
2015-11-20 13:18:26,639 - DEBUG __init__.py:54 -- Calling: set_actual_time with args: ([<class 'fuelweb_test.helpers.ntp.NtpInitscript'>(0x7fa8c30f6ed0) admin_ip:None, node_name:admin, sync:False, conn:False],) {}
2015-11-20 13:18:26,639 - DEBUG helpers.py:328 -- Executing command: 'ntpdate -p 4 -t 0.2 -bu 0.fuel.pool.ntp.org'
2015-11-20 13:19:26,640 - DEBUG helpers.py:328 -- Executing command: 'ntpdate -p 4 -t 0.2 -bu 0.fuel.pool.ntp.org'
2015-11-20 13:20:06,685 - DEBUG __init__.py:59 -- Done: set_actual_time with result: False
...
2015-11-20 13:21:56,126 - INFO decorators.py:298 -- Failed in method 'deploy_neutron_vlan_ha'. Make snapshot: error_deploy_neutron_vlan_ha
...

[Igor Belikov (ibelikov)]

Issue seems to occure only on ci-slave13.fuel-infra.org. I removed this slave out of deployment tests pool for further investigation.

[Igor Belikov (ibelikov)]

ci-slave13 was redeployed and the issue is fixed.