[System tests] Upgrade fuel after rollback failed with false negative result

Bug #1498353 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vasily Gorin
7.0.x
Fix Committed
High
Vadim Rovachev

Bug Description

https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.rollback_neutron_gre/68/testReport/(root)/upgrade_fuel_after_rollback/upgrade_fuel_after_rollback/

Upgrade script fails with next message

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/tests/tests_upgrade/test_upgrade.py", line 529, in upgrade_fuel_after_rollback
    exit_code=255)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/__init__.py", line 58, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.rollback_neutron_gre/UPGRADE/fuelweb_test/helpers/checkers.py", line 296, in run_script
    " /var/log/fuel_upgrade.log")['stdout'])))

But there is no upgrade log in snapshot, and if we investigate system-tests.log we can see that we fail before upgrade execition according to we can not find upgrdae.sh

015-09-22 02:11:09,945 - DEBUG __init__.py:55 -- Calling: run_script with args: (<devops.helpers.helpers.SSHClient object at 0x7f3361e035d0>, '/var', 'upgrade.sh') {'password': 'admin', 'rollback': True, 'exit_code': 255}
2015-09-22 02:11:09,945 - DEBUG helpers.py:330 -- Executing command: 'chmod 755 /var/upgrade.sh'
2015-09-22 02:11:10,013 - DEBUG checkers.py:276 -- Result of cmod is {'exit_code': 1, 'stderr': ["chmod: cannot access `/var/upgrade.sh': No such file or directory\n"], 'stdout': []}
2015-09-22 02:11:10,013 - DEBUG helpers.py:330 -- Executing command: 'UPGRADERS='host-system docker openstack raise-error' /var/upgrade.sh --password admin'
2015-09-22 02:11:10,078 - DEBUG checkers.py:288 -- Try to read status code from chain...

We should assert each command to avoid to false negative messages
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Upgrade script fails with next message

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

It might be the same problem that we faced before - https://bugs.launchpad.net/fuel/+bug/1448904 .
I propose to add logging of stderr from untar command as part of this bug and also add assert of upgrade.sh file after unpacking.

Changed in fuel:
status: New → Confirmed
Vasily Gorin (vgorin)
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Vasily Gorin (vgorin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/230496

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/230496
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=d468fe1f6be350e24a327cdcd9198dc14283e3e0
Submitter: Jenkins
Branch: master

commit d468fe1f6be350e24a327cdcd9198dc14283e3e0
Author: vgorin <email address hidden>
Date: Fri Oct 2 18:27:06 2015 +0300

    Add checking of existing script file on node in run_upgrade_script.

    Move untar method from checkers to fuel_actions.AdminActions
    Use decorator retry for untar method.
    Delete context manager from upgrade_master.

    Change-Id: Ia5ff27bb54b4ce61caf1576c33d8a3bb838586e3
    Closes-Bug: #1498353

Changed in fuel:
status: In Progress → Fix Committed
Dmitry Pyzhov (dpyzhov)
tags: added: area-qa
Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/300387

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/7.0)

Reviewed: https://review.openstack.org/300387
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=99f739a1cc082757e2259835d60f3eefe04d2d1f
Submitter: Jenkins
Branch: stable/7.0

commit 99f739a1cc082757e2259835d60f3eefe04d2d1f
Author: vgorin <email address hidden>
Date: Fri Oct 2 18:27:06 2015 +0300

    Add checking of existing script file on node in run_upgrade_script.

    Move untar method from checkers to fuel_actions.AdminActions
    Use decorator retry for untar method.
    Delete context manager from upgrade_master.

    Change-Id: Ia5ff27bb54b4ce61caf1576c33d8a3bb838586e3
    Closes-Bug: #1498353

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.