[Murano] Time limit exceeded while waiting for Check that needed ports are opened to finish

Bug #1495699 reported by Maksym Strukov on 2015-09-14
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Timur Nurlygayanov
8.0.x
High
Timur Nurlygayanov

Bug Description

Scenario:
1. Create new environment
2. Choose Neutron, TUN
3. Choose Murano
4. Choose Ceilometer
5. Add 3 controller
6. Add 1 compute
7. Add 1 cinder+mongo
8. Add 2 mongo
9. Move Management network to eth1
10. Move Storage network to eth2 and untag it
11. Change disk configuration for both Mongo nodes. Change 'MongoDB' volume for vdc
12. Verify networks
13. Deploy the environment
14. Verify networks
15. Configure Murano in agree with https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#preparing-murano-for-testing
16. Run OSTF tests

Expected: Murano tests passed

Actual:
Murano tests failed with message:
"Time limit exceeded while waiting for Check that needed ports are opened to finish. Please refer to OpenStack logs for more details."

There is no info in logs to see what port are closed. So I added:
LOG.info("MAXMAX: checking {0} to {1}".format(ip, port))
to /usr/lib/python2.6/site-packages/fuel_health/muranomanager.py:356 in ostf docker.
Result:
2015-09-14 21:10:52 INFO (test_mixins) STEP:7, verify action: 'Check that needed ports are opened'
2015-09-14 21:10:52 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
2015-09-14 21:10:57 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
...
2015-09-14 21:15:48 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
2015-09-14 21:15:52 INFO (test_mixins) Timeout 300s exceeded for Check that needed ports are opened

In the same time I can open 10.109.1.177:80 (float ip) in browser on host pc and see Apache demo page, but can't ssh. I also can't access 10.109.1.177:80/22 from master node and ostf container.

Env:
7.0-288 (RC2)

{"build_id": "288", "build_number": "288", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "288", "build_number": "288", "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

Changed in fuel:
assignee: nobody → MOS QA Team (mos-qa)
milestone: none → 8.0
Maksym Strukov (unbelll) wrote :
summary: - [Murano] debian-8-m-agent.qcow2 doesn't has ssh port opened
+ [Murano] Time limit exceeded while waiting for Check that needed ports
+ are opened to finish
description: updated
Maksym Strukov (unbelll) on 2015-09-14
description: updated
Victor Ryzhenkin (vryzhenkin) wrote :

The root cause of this bug is the absence of public network on master node.

Changed in fuel:
status: New → Triaged
assignee: MOS QA Team (mos-qa) → Victor Ryzhenkin (vryzhenkin)
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-qa
Changed in fuel:
milestone: 8.0 → 9.0
Changed in fuel:
assignee: Victor Ryzhenkin (vryzhenkin) → Timur Nurlygayanov (tnurlygayanov)

Fix proposed to branch: master
Review: https://review.openstack.org/275872

Changed in fuel:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/275872
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=3b158d74b77d243f7296fada310455bc6fa44421
Submitter: Jenkins
Branch: master

commit 3b158d74b77d243f7296fada310455bc6fa44421
Author: Timur Nurlygayanov <email address hidden>
Date: Wed Feb 3 21:02:14 2016 +0300

    Fixed issue with ports checks

    OSTF tests don't have direct access to VMs ports,
    we need to check availability of these ports from
    OpenStack controller nodes instead of Fuel master node.

    Change-Id: I90c4980cefb8fc650d1c2d2f4f92e98815bda98f
    Closes-Bug: #1495699

Changed in fuel:
status: In Progress → Fix Committed

Moved to Maintenance Updates for MOS 8.0 accordingly the discussion in mos-dev with PM team.

The fix is on review for master and stable/8.0 branches, the issue affects OSTF Murano tests only for deployments where a public network of OpenStack cluster doesn't accessible from master Fuel node.
I think it is not a critical issue for MOS 8.0 release, I suggest to move it to MOS 8.0 MU1.
Reason: it is reproduced not in 100% of cases, it affects only OSTF tests, we have the fix on review and this fix can be applied on Fuel master node manually if it is a blocker issue for someone.

tags: added: release-notes
tags: added: area-ostf
removed: area-qa ostf

Ok, the issue was fixed, marked as Fix Released.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.

Changed in fuel:
status: Fix Committed → Fix Released
tags: added: release-notes-done
removed: release-notes

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/22326

Reviewed: https://review.fuel-infra.org/22326
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: dcd0ac5631e06625c76d2d62c17bc1fef67a7072
Author: Evgeny Konstantinov <email address hidden>
Date: Wed Jun 22 10:51:45 2016

Add resolved issues to relnotes 9.0

Change-Id: I87df13ac06921547312dd2165097d080528ec864
Related-Bug: #1587960
Related-Bug: #1544446
Related-Bug: #1478310
Related-Bug: #1543050
Related-Bug: #1495699

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers