Missing file on slave after ceph migration

[Alexander Kurenyshev]

The same situation as described here [1] but under 9.1 SWARM test, so it seems to be regression.

Test `Check VM backed with ceph migration in ha mode with 1 controller` failed [2] with an error:

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.ceph_ha_one_controller/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.ceph_ha_one_controller/fuelweb_test/tests/test_ceph.py", line 807, in migrate_vm_backed_with_ceph
    "File is absent in /mnt")
AssertionError: File is absent in /mnt

Test source with failed step [3]

[1] https://bugs.launchpad.net/fuel/+bug/1588808
[2] https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ceph_ha_one_controller/40/console
[3] https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/tests/test_ceph.py#L806

[Alexei Sheplyakov (asheplyakov)]

Most likely the bug has nothing to do with ceph (for the record there've been no ceph related changes for a few months [1])

On the other hand the check itself [2] is not very reliable:
- the exit code is not checked at all, so it can be sudo which actually failed
- the script which is supposed to create the file can fail too

[1] https://review.fuel-infra.org/gitweb?p=packages%2Ftrusty%2Fceph.git;a=commitdiff;h=a6bef4391f17c638c4bcc8db58394331234e5186

[2] https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/tests/test_ceph.py#L801-L805

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ceph_ha_one_controller/82/testReport/(root)/migrate_vm_backed_with_ceph/migrate_vm_backed_with_ceph/

[Alexandra (aallakhverdieva)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ceph_ha_one_controller/85/testReport/(root)/migrate_vm_backed_with_ceph/

[Alexander Kurenyshev (akurenyshev)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ceph_ha_one_controller/128/console

[Alexei Sheplyakov (asheplyakov)]

The code [1] does not distinguish between different error conditions:

a) connection to the remote host failed
b) running sudo on the remote host failed
c) the volume which contains the file hasn't been attached [yet]
d) the filesystem on that volume has been corrupted (due to improper umount)
e) the file hasn't been created (due to the above listed reasons)

Please improve the check so it blames the correct component, or provide some logs indicated that the problem is caused by ceph.

[1] https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/tests/test_ceph.py#L801-L805

[Dmitry Pyzhov (dpyzhov)]

Assigning to fuel-qa according to @Alexei's comment. Please don't move bugs to unassigned state if possible

[Dmitry Belyaninov (dbelyaninov)]

10 swarm
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ceph_ha_one_controller/160/testReport/(root)/migrate_vm_backed_with_ceph/migrate_vm_backed_with_ceph/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423113

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423113
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=437155a3cbc5ce0ddb326cf096df54cfb63860a8
Submitter: Jenkins
Branch: stable/mitaka

commit 437155a3cbc5ce0ddb326cf096df54cfb63860a8
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 20 07:54:03 2017 +0000

    Modification of mounted volume check

    Remote ssh execution returns object with bash outputs.
    Elements in "stdout" of "ls" command contains '\n'
    endings. So test assert was changed to fix false fail.

    Change-Id: I241750d970a6634cb60c0ab905f0e4b7030834ec
    Closes-Bug: 1618022

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423963

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/423964

[Dmitry Belyaninov (dbelyaninov)]

9.x test job passed since #180 run

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423963
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=caa2b64823ef503b948bd50820a288b67608b00f
Submitter: Jenkins
Branch: stable/newton

commit caa2b64823ef503b948bd50820a288b67608b00f
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 20 07:54:03 2017 +0000

    Modification of mounted volume check

    Remote ssh execution returns object with bash outputs.
    Elements in "stdout" of "ls" command contains '\n'
    endings. So test assert was changed to fix false fail.

    Change-Id: I241750d970a6634cb60c0ab905f0e4b7030834ec
    Closes-Bug: 1618022
    (cherry picked from commit 437155a3cbc5ce0ddb326cf096df54cfb63860a8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423964
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=68b3117cafd36a19022b6f918ae98c564d670a56
Submitter: Jenkins
Branch: master

commit 68b3117cafd36a19022b6f918ae98c564d670a56
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 20 07:54:03 2017 +0000

    Modification of mounted volume check

    Remote ssh execution returns object with bash outputs.
    Elements in "stdout" of "ls" command contains '\n'
    endings. So test assert was changed to fix false fail.

    Change-Id: I241750d970a6634cb60c0ab905f0e4b7030834ec
    Closes-Bug: 1618022
    (cherry picked from commit 437155a3cbc5ce0ddb326cf096df54cfb63860a8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-qa 11.0.0.0rc1 release candidate.