[fuel-qa] devops fails to revert environment with one isolated node and external ntp server configured

Bug #1620616 reported by sryabin on 2016-09-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Undecided
Vladimir Khlyunev

Bug Description

Detailed bug description:
 While devops revert the environment, he tries to sync time via ssh and ntpdate on each nodes in the environment.
 The ntp serrver was choosed as first ntp server entry in /etc/ntp.conf
 On the isolated node, we haven't access to public internet network, and ntpdate returns error code

Steps to reproduce:
 1. Install fuel master 8.0 with ha openstack cluster
 2. Install octane
 3. Create backup
 4. Reinstall fuel master node with 9.x release
 5. Install octane
 6. Restore
 7. Start upgrade procedure via octane
 8. ...
 9. octane upgrade-node --isolated
 10. Create snapshot
 11. Revert snapshot
Expected results:
 Reverting snapshot was successfully
Actual result:
 Traceback (most recent call last):
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/tests/tests_upgrade/test_os_upgrade.py", line 116, in upgrade_db
    self.env.revert_snapshot("upgrade_first_cic")
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/models/environment.py", line 307, in revert_snapshot
    self.sync_time()
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-mitaka/fuelweb_test/models/environment.py", line 139, in sync_time
    new_time = sync_time(self.d_env, nodes_names, skip_sync)
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 43, in sync_time
    g_ntp.do_sync_time(g_ntp.pacemaker_ntps)
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 327, in do_sync_time
    ntp.set_actual_time()
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 116, in set_actual_time
    self._node_name))
  File "/var/lib/jenkins/workdir/72283b89d87d22edaac581db5f7d3568-44/fuel-qa-venv/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 111, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Failed to set actual time on node u'slave-03'

Reproducibility:
 100%
Workaround:
 Before run upgrade-node --siolated, changed in fuel-qa test ntp server to the fuel master ip /etc/ntp.donf on target node
After upgrade, revert the changes in /etc/ntp.conf

 OR

 before running the command, make check, if node is in isolated mode, run ntpdate command with fuel master ip argument master
Impact:
 Upgrade procedure
Description of the environment:
 fuel-octane-9.0.0-1.mos1312.noarch
Additional information:
 TimeoutError: Failed to set actual time on node u'slave-03'

Reviewed: https://review.openstack.org/366744
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=b3fddcce5c3ef74988c6dcad938b488d77ed8ece
Submitter: Jenkins
Branch: stable/7.0

commit b3fddcce5c3ef74988c6dcad938b488d77ed8ece
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Sep 7 16:03:38 2016 +0300

    Use master node as NTP and DNS server for upgrade scenarios

    We are expecting issues in time sync procedure after reverting the
    snapshots when nodes are in isolation. Switching ntp and dns servers
    removes snapshot-reverting artifacts.

    Change-Id: Ia1e6302ab145bb1b5d5a9eec22b84505840427a2
    Closes-bug:1620616

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/368694

Changed in fuel:
assignee: nobody → Vladimir Khlyunev (vkhlyunev)
status: New → In Progress

Reviewed: https://review.openstack.org/368693
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=3e74524ba8f03ec3cefb6468467e09767be8f500
Submitter: Jenkins
Branch: stable/8.0

commit 3e74524ba8f03ec3cefb6468467e09767be8f500
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Sep 7 16:03:38 2016 +0300

    Use master node as NTP and DNS server for upgrade scenarios

    We are expecting issues in time sync procedure after reverting the
    snapshots when nodes are in isolation. Switching ntp and dns servers
    removes snapshot-reverting artifacts.

    Change-Id: Ia1e6302ab145bb1b5d5a9eec22b84505840427a2
    Closes-bug:1620616
    (cherry picked from commit b3fddcce5c3ef74988c6dcad938b488d77ed8ece)

Reviewed: https://review.openstack.org/368694
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=38e763bb586e7cee1f78f1d2fe44b5628807b046
Submitter: Jenkins
Branch: stable/mitaka

commit 38e763bb586e7cee1f78f1d2fe44b5628807b046
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Sep 7 16:03:38 2016 +0300

    Use master node as NTP and DNS server for upgrade scenarios

    We are expecting issues in time sync procedure after reverting the
    snapshots when nodes are in isolation. Switching ntp and dns servers
    removes snapshot-reverting artifacts.

    Change-Id: Ia1e6302ab145bb1b5d5a9eec22b84505840427a2
    Closes-bug:1620616
    (cherry picked from commit b3fddcce5c3ef74988c6dcad938b488d77ed8ece)

tags: added: in-stable-mitaka
Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/368696
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c8578691b8389b1ff450e0d4a178c2eb292756ba
Submitter: Jenkins
Branch: master

commit c8578691b8389b1ff450e0d4a178c2eb292756ba
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Sep 7 16:03:38 2016 +0300

    Use master node as NTP and DNS server for upgrade scenarios

    We are expecting issues in time sync procedure after reverting the
    snapshots when nodes are in isolation. Switching ntp and dns servers
    removes snapshot-reverting artifacts.

    Change-Id: Ia1e6302ab145bb1b5d5a9eec22b84505840427a2
    Closes-bug:1620616
    (cherry picked from commit b3fddcce5c3ef74988c6dcad938b488d77ed8ece)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers