Multipath device test catches BadGateway

[Sergey Novikov]

Detailed bug description:

Test https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.deploy_multipath/173/testReport/(root)/bootstrap_multipath/bootstrap_multipath/ fails on bootstrap stage

http://paste.openstack.org/show/595061/

There aren't any errors in nailgun's logs.

Additional information:
snapshot # 766

[Sergey Novikov (snovikov)]

we need to collect more info about the issue. I've assigned the bug to Fuel QA team for this purpose

[Dmitry Belyaninov (dbelyaninov)]

https://bugs.launchpad.net/fuel/+bug/1652788

may be some regression after merge ^^

@Vladimir Kozhukalov could you please take a look

[Vladimir Kozhukalov (kozhukalov)]

Can not reproduce on failed job environment. Looks like the issue appears because we send requests to nailgun when it is still not working after restart. According to logs, there was more than enough time to start, so maybe it is because it could not start at all.

2017-01-17 16:20:19 - INFO fuel_actions.py:439 -- Update Fuel node was successful
2017-01-17 16:25:24 - INFO ssh_manager.py:137 -- SSHManager: Check for current connection fails. Try to reconnect
2017-01-17 16:25:35 - INFO base_test_case.py:255 -- Enabled sending of statistics to fuel-collect-systest.infra.mirantis.net:80
2017-01-17 16:25:37 - ERROR log_helpers.py:229 -- Failed:
'get_releases'(
    'self'=[<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7f8f50b4b3d0), url:None],
)
Traceback (most recent call last):
  File "/home/jenkins/workspace/9.x.custom.ubuntu.system_test/core/helpers/log_helpers.py", line 215, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.x.custom.ubuntu.system_test/fuelweb_test/models/nailgun_client.py", line 198, in get_releases
    return self._get(url="/releases/").json()
  File "/home/jenkins/workspace/9.x.custom.ubuntu.system_test/fuelweb_test/models/nailgun_client.py", line 46, in _get
    return self.session.get(url=url, connect_retries=1, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 727, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 624, in request
    raise exceptions.from_response(resp, method, url)
BadGateway: Bad Gateway (HTTP 502)

[Vladimir Kozhukalov (kozhukalov)]

The root cause of the issue is that after updating the Fuel admin node it is rebooted. The test waits for ssh is available again [1] and continues doing things like sending requests to nailgun, but availabilty of ssh does not mean nailgun is available. We need to improve test framework.

[1] https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/tests/base_test_case.py#L274

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421833

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421833
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=640ac619fd580586f312c9e8658687536da4188e
Submitter: Jenkins
Branch: stable/mitaka

commit 640ac619fd580586f312c9e8658687536da4188e
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 18 14:07:02 2017 +0300

    Wait for fuel is available after fuel reboot

    Some test cases reboot the Fuel master node,
    wait until ssh becomes available and continue
    doing stuff like sending requests to nailgun.
    But ssh availability does not mean api is available.

    Change-Id: I07e82da829d814191da3b0256688aaf74aa92a5a
    Closes-Bug: #1656872

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421920

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422081

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421920
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=27244b223b2fa406cd296473433bb66d247ca8c5
Submitter: Jenkins
Branch: stable/mitaka

commit 27244b223b2fa406cd296473433bb66d247ca8c5
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 18 17:07:16 2017 +0300

    Wait until nailgun is ready after restart

    There are plenty of test cases where we modify
    nailgun settings and then restart nailgun itself.
    We need to wait for nailgun to become again available
    before sending other requests to it.

    Change-Id: I84ec0b4cb460dc6a8b563178ed1ff7e4736288e7
    Partial-Bug: #1656872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422081
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7c1608c2f4340b00779f2a0ea887d1f5294efead
Submitter: Jenkins
Branch: stable/mitaka

commit 7c1608c2f4340b00779f2a0ea887d1f5294efead
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jan 18 19:19:04 2017 +0300

    Nailgun status check should include http responses

    Fuel-utils could be used for automation of some tasks
    and it is crucial service status checks to be adequate.
    It is not enough to check that nailgun is started
    we also need to check that it accepts requests and
    responds.

    Change-Id: I587925a468283cf48e22dc8afef36d7674af9903
    Partial-Bug: #1656872

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423224

[Vladimir Kozhukalov (kozhukalov)]

The root cause of the issue is that we create keystone session in test case then restart nailgun and then immediately use this same keystone session to access nailgun after restart. If sleep(10) after restart or recreate keystone session then everything goes well. Probably the problem appears due to the way how we use keystone (with keystonemiddleware), can not say anything that would make sense. Need additional research.

Regarding this patch https://review.openstack.org/#/c/422081/ we need to revert it because fuel client does not support ssl and `fuel-utils check_all` and `fuel-utils check_service nailgun` will fail if force_ssl is true in astute.yaml.

[Vladimir Kozhukalov (kozhukalov)]

Revert request https://review.openstack.org/#/c/423303/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423224
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c6a26a04c3b6ac097b6cfda325f4b52d372e61db
Submitter: Jenkins
Branch: stable/mitaka

commit c6a26a04c3b6ac097b6cfda325f4b52d372e61db
Author: Vladimir Khlyunev <email address hidden>
Date: Fri Jan 20 15:43:50 2017 +0400

    Sync setup_multipath setup part with base Fuel setup workflow

    For some reasons multipath test is failing getting releases
    without the check before it. Put it and make multipath test
    closer to all other cases.

    Change-Id: I73bfda7e2b4469214b86b9150967bfe249281da2
    Partial-bug:1656872

[Ekaterina Shutova (eshutova)]

Verified on 9.2 ​snapshot ​#796. Passed run: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.deploy_multipath/180/testReport/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/430987

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430988

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/430988

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/430987