FuelMasterMigrate failed Network verification not avaliable because nodes ["1", "2"] not avaliable via mcollective

Bug #1576276 reported by Egor Kotko
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Alexander Zatserklyany
Mitaka
High
Vadim Rovachev

Bug Description

Steps to reproduce:

Scenario:
    0. Setup master node
    1. Configuration releases
    2. Bootstrap slave and make snapshot
    3. Revert bootstrapped nodes
    4. Create Fuel Environment
    5. Add nodes to environment
    6. Run network checker
    7. Deploy environment
    8. Run network checker
    9. Migrate Fuel Master to a compute
    10. Check periodically the status of Fuel Master migration process
    11. Run network checker
    12. Run health checker

Failed with:
Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not avaliable because nodes ["1", "2"] not avaliable via mcollective.
Inspect Astute logs for the details'

Revision history for this message
Egor Kotko (ykotko) wrote :
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
status: New → Confirmed
tags: added: area-python
removed: area-qa
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Georgy Kibardin (gkibardin)
status: Confirmed → In Progress
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Georgy Kibardin (gkibardin) → Fuel Customer Enablement Team (fuel-customer-enablement)
status: In Progress → Confirmed
Peter Zhurba (pzhurba)
Changed in fuel:
assignee: Fuel Customer Enablement Team (fuel-customer-enablement) → Peter Zhurba (pzhurba)
Revision history for this message
Peter Zhurba (pzhurba) wrote :

According provided snapshot there is no issues
According tail of nailgun.test.domain.local/var/log/fuel-migrate.log everything finished OK, cloned fuel could see both nodes.
Also according node-1/var/log/atop/atop_current there was near 1 minute between finishing data-sync phase and snapshot creating. It lets us suspect that test was performed too early

Revision history for this message
Peter Zhurba (pzhurba) wrote :

Yo may use after first reboot following check:

until ssh -tt ${admin_net_ip} fuel node; do ;sleep 20 do

Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Fuel QA Team (fuel-qa)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/313593

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Peter Zhurba (pzhurba)
status: Confirmed → In Progress
Revision history for this message
Peter Zhurba (pzhurba) wrote :

I've look at all "red" jobs logs.

There are 3 main causes of failure:

1 Qemu related problems (for example hung during coping) .
2 After migration nailgun begins answering (i. e. to work). But not all nodes have time to update their status before test starts
3 Busy indicator /tmp/notready is present only last couple minutes of whole migration time.

So it was added new flag /tmp/migration-done which appears exactly when migration process is finished and nailgun on cone starts working.

But after that, additionally in tests code is necessary to add checking of node status. Inside script itself it is impossible because cluster configuration and current state may be different. During testing configuration of cluster is known

It allows avoid 2 and 3 cases.

According 1 there is only common recommendation - update software on Jenkins slave host, increase memory and cpus on compute node

Szymon Banka (sbanka)
Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Alexander Zatserklyany (zatserklyany)
Changed in fuel:
assignee: Alexander Zatserklyany (zatserklyany) → Peter Zhurba (pzhurba)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/317340

Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Alexander Zatserklyany (zatserklyany)
Changed in fuel:
assignee: Alexander Zatserklyany (zatserklyany) → Peter Zhurba (pzhurba)
Peter Zhurba (pzhurba)
Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Alexander Zatserklyany (zatserklyany)
Changed in fuel:
assignee: Alexander Zatserklyany (zatserklyany) → Peter Zhurba (pzhurba)
Peter Zhurba (pzhurba)
Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Alexander Zatserklyany (zatserklyany)
Dmitry Pyzhov (dpyzhov)
tags: added: area-qa
removed: area-python
Changed in fuel:
assignee: Alexander Zatserklyany (zatserklyany) → Peter Zhurba (pzhurba)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/313593
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ecf9c0d908ece88f21202e59c17109ec098321a7
Submitter: Jenkins
Branch: master

commit ecf9c0d908ece88f21202e59c17109ec098321a7
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df

Peter Zhurba (pzhurba)
Changed in fuel:
assignee: Peter Zhurba (pzhurba) → Alexander Zatserklyany (zatserklyany)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/341244

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/341244
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=23766969e4df642dea41d371598d442f49efaf8c
Submitter: Jenkins
Branch: stable/mitaka

commit 23766969e4df642dea41d371598d442f49efaf8c
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df
    (cherry picked from commit ecf9c0d908ece88f21202e59c17109ec098321a7)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/344647

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/344649

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/317340
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=573ded4401abd7e0f1090e31333d65ab160d4bc7
Submitter: Jenkins
Branch: master

commit 573ded4401abd7e0f1090e31333d65ab160d4bc7
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Changed in fuel:
status: In Progress → Fix Committed
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/8.0)

Reviewed: https://review.openstack.org/344649
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Submitter: Jenkins
Branch: stable/8.0

commit 2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/344647
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7a88c7476dced6292dd9013ef5e7cd031f19c446
Submitter: Jenkins
Branch: stable/mitaka

commit 7a88c7476dced6292dd9013ef5e7cd031f19c446
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers