[System tests] Upgrade fuel after rollback failed with false negative result

Bug #1498353 reported by Tatyanka on 2015-09-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Vasily Gorin
7.0.x
High
Vadim Rovachev

Bug Description

https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.rollback_neutron_gre/68/testReport/(root)/upgrade_fuel_after_rollback/upgrade_fuel_after_rollback/

Upgrade script fails with next message

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/tests/tests_upgrade/test_upgrade.py", line 529, in upgrade_fuel_after_rollback
    exit_code=255)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/__init__.py", line 58, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/helpers/checkers.py", line 296, in run_script
    " /var/log/fuel_upgrade.log")['stdout'])))

But there is no upgrade log in snapshot, and if we investigate system-tests.log we can see that we fail before upgrade execition according to we can not find upgrdae.sh

015-09-22 02:11:09,945 - DEBUG __init__.py:55 -- Calling: run_script with args: (<devops.helpers.helpers.SSHClient object at 0x7f3361e035d0>, '/var', 'upgrade.sh') {'password': 'admin', 'rollback': True, 'exit_code': 255}
2015-09-22 02:11:09,945 - DEBUG helpers.py:330 -- Executing command: 'chmod 755 /var/upgrade.sh'
2015-09-22 02:11:10,013 - DEBUG checkers.py:276 -- Result of cmod is {'exit_code': 1, 'stderr': ["chmod: cannot access `/var/upgrade.sh': No such file or directory\n"], 'stdout': []}
2015-09-22 02:11:10,013 - DEBUG helpers.py:330 -- Executing command: 'UPGRADERS='host-system docker openstack raise-error' /var/upgrade.sh --password admin'
2015-09-22 02:11:10,078 - DEBUG checkers.py:288 -- Try to read status code from chain...

We should assert each command to avoid to false negative messages
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Upgrade script fails with next message

It might be the same problem that we faced before - https://bugs.launchpad.net/fuel/+bug/1448904 .
I propose to add logging of stderr from untar command as part of this bug and also add assert of upgrade.sh file after unpacking.

Changed in fuel:
status: New → Confirmed
Vasily Gorin (vgorin) on 2015-10-02
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Vasily Gorin (vgorin)

Fix proposed to branch: master
Review: https://review.openstack.org/230496

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/230496
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=d468fe1f6be350e24a327cdcd9198dc14283e3e0
Submitter: Jenkins
Branch: master

commit d468fe1f6be350e24a327cdcd9198dc14283e3e0
Author: vgorin <email address hidden>
Date: Fri Oct 2 18:27:06 2015 +0300

    Add checking of existing script file on node in run_upgrade_script.

    Move untar method from checkers to fuel_actions.AdminActions
    Use decorator retry for untar method.
    Delete context manager from upgrade_master.

    Change-Id: Ia5ff27bb54b4ce61caf1576c33d8a3bb838586e3
    Closes-Bug: #1498353

Changed in fuel:
status: In Progress → Fix Committed
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-qa
Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: non-release

Reviewed: https://review.openstack.org/300387
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=99f739a1cc082757e2259835d60f3eefe04d2d1f
Submitter: Jenkins
Branch: stable/7.0

commit 99f739a1cc082757e2259835d60f3eefe04d2d1f
Author: vgorin <email address hidden>
Date: Fri Oct 2 18:27:06 2015 +0300

    Add checking of existing script file on node in run_upgrade_script.

    Move untar method from checkers to fuel_actions.AdminActions
    Use decorator retry for untar method.
    Delete context manager from upgrade_master.

    Change-Id: Ia5ff27bb54b4ce61caf1576c33d8a3bb838586e3
    Closes-Bug: #1498353

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers