[Murano] Time limit exceeded while waiting for Check that needed ports are opened to finish

Bug #1495699 reported by Maksym Strukov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Timur Nurlygayanov
8.0.x
In Progress
High
Timur Nurlygayanov

Bug Description

Scenario:
1. Create new environment
2. Choose Neutron, TUN
3. Choose Murano
4. Choose Ceilometer
5. Add 3 controller
6. Add 1 compute
7. Add 1 cinder+mongo
8. Add 2 mongo
9. Move Management network to eth1
10. Move Storage network to eth2 and untag it
11. Change disk configuration for both Mongo nodes. Change 'MongoDB' volume for vdc
12. Verify networks
13. Deploy the environment
14. Verify networks
15. Configure Murano in agree with https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#preparing-murano-for-testing
16. Run OSTF tests

Expected: Murano tests passed

Actual:
Murano tests failed with message:
"Time limit exceeded while waiting for Check that needed ports are opened to finish. Please refer to OpenStack logs for more details."

There is no info in logs to see what port are closed. So I added:
LOG.info("MAXMAX: checking {0} to {1}".format(ip, port))
to /usr/lib/python2.6/site-packages/fuel_health/muranomanager.py:356 in ostf docker.
Result:
2015-09-14 21:10:52 INFO (test_mixins) STEP:7, verify action: 'Check that needed ports are opened'
2015-09-14 21:10:52 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
2015-09-14 21:10:57 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
...
2015-09-14 21:15:48 INFO (muranomanager) MAXMAX: checking 10.109.1.177 to 22
2015-09-14 21:15:52 INFO (test_mixins) Timeout 300s exceeded for Check that needed ports are opened

In the same time I can open 10.109.1.177:80 (float ip) in browser on host pc and see Apache demo page, but can't ssh. I also can't access 10.109.1.177:80/22 from master node and ostf container.

Env:
7.0-288 (RC2)

{"build_id": "288", "build_number": "288", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "288", "build_number": "288", "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25", "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac", "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

Changed in fuel:
assignee: nobody → MOS QA Team (mos-qa)
milestone: none → 8.0
Revision history for this message
Maksym Strukov (unbelll) wrote :
summary: - [Murano] debian-8-m-agent.qcow2 doesn't has ssh port opened
+ [Murano] Time limit exceeded while waiting for Check that needed ports
+ are opened to finish
description: updated
Maksym Strukov (unbelll)
description: updated
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

The root cause of this bug is the absence of public network on master node.

Changed in fuel:
status: New → Triaged
assignee: MOS QA Team (mos-qa) → Victor Ryzhenkin (vryzhenkin)
Dmitry Pyzhov (dpyzhov)
tags: added: area-qa
Changed in fuel:
milestone: 8.0 → 9.0
Changed in fuel:
assignee: Victor Ryzhenkin (vryzhenkin) → Timur Nurlygayanov (tnurlygayanov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (master)

Fix proposed to branch: master
Review: https://review.openstack.org/275872

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275889

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ostf (master)

Reviewed: https://review.openstack.org/275872
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=3b158d74b77d243f7296fada310455bc6fa44421
Submitter: Jenkins
Branch: master

commit 3b158d74b77d243f7296fada310455bc6fa44421
Author: Timur Nurlygayanov <email address hidden>
Date: Wed Feb 3 21:02:14 2016 +0300

    Fixed issue with ports checks

    OSTF tests don't have direct access to VMs ports,
    we need to check availability of these ports from
    OpenStack controller nodes instead of Fuel master node.

    Change-Id: I90c4980cefb8fc650d1c2d2f4f92e98815bda98f
    Closes-Bug: #1495699

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Moved to Maintenance Updates for MOS 8.0 accordingly the discussion in mos-dev with PM team.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The fix is on review for master and stable/8.0 branches, the issue affects OSTF Murano tests only for deployments where a public network of OpenStack cluster doesn't accessible from master Fuel node.
I think it is not a critical issue for MOS 8.0 release, I suggest to move it to MOS 8.0 MU1.
Reason: it is reproduced not in 100% of cases, it affects only OSTF tests, we have the fix on review and this fix can be applied on Fuel master node manually if it is a blocker issue for someone.

tags: added: release-notes
tags: added: area-ostf
removed: area-qa ostf
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Ok, the issue was fixed, marked as Fix Released.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.

Changed in fuel:
status: Fix Committed → Fix Released
tags: added: release-notes-done
removed: release-notes
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to mos/mos-docs (master)

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/22326

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to mos/mos-docs (master)

Reviewed: https://review.fuel-infra.org/22326
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: dcd0ac5631e06625c76d2d62c17bc1fef67a7072
Author: Evgeny Konstantinov <email address hidden>
Date: Wed Jun 22 10:51:45 2016

Add resolved issues to relnotes 9.0

Change-Id: I87df13ac06921547312dd2165097d080528ec864
Related-Bug: #1587960
Related-Bug: #1544446
Related-Bug: #1478310
Related-Bug: #1543050
Related-Bug: #1495699

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-ostf (stable/8.0)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/8.0
Review: https://review.opendev.org/275889
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.