FuelMasterMigrate failed Network verification not avaliable because nodes ["1", "2"] not avaliable via mcollective

[Egor Kotko]

Steps to reproduce:

Scenario:
    0. Setup master node
    1. Configuration releases
    2. Bootstrap slave and make snapshot
    3. Revert bootstrapped nodes
    4. Create Fuel Environment
    5. Add nodes to environment
    6. Run network checker
    7. Deploy environment
    8. Run network checker
    9. Migrate Fuel Master to a compute
    10. Check periodically the status of Fuel Master migration process
    11. Run network checker
    12. Run health checker

Failed with:
Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not avaliable because nodes ["1", "2"] not avaliable via mcollective.
Inspect Astute logs for the details'

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Peter Zhurba (pzhurba)]

According provided snapshot there is no issues
According tail of nailgun.test.domain.local/var/log/fuel-migrate.log everything finished OK, cloned fuel could see both nodes.
Also according node-1/var/log/atop/atop_current there was near 1 minute between finishing data-sync phase and snapshot creating. It lets us suspect that test was performed too early

[Peter Zhurba (pzhurba)]

Yo may use after first reboot following check:

until ssh -tt ${admin_net_ip} fuel node; do ;sleep 20 do

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313593

[Peter Zhurba (pzhurba)]

I've look at all "red" jobs logs.

There are 3 main causes of failure:

1 Qemu related problems (for example hung during coping) .
2 After migration nailgun begins answering (i. e. to work). But not all nodes have time to update their status before test starts
3 Busy indicator /tmp/notready is present only last couple minutes of whole migration time.

So it was added new flag /tmp/migration-done which appears exactly when migration process is finished and nailgun on cone starts working.

But after that, additionally in tests code is necessary to add checking of node status. Inside script itself it is impossible because cluster configuration and current state may be different. During testing configuration of cluster is known

It allows avoid 2 and 3 cases.

According 1 there is only common recommendation - update software on Jenkins slave host, increase memory and cpus on compute node

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/317340

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313593
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ecf9c0d908ece88f21202e59c17109ec098321a7
Submitter: Jenkins
Branch: master

commit ecf9c0d908ece88f21202e59c17109ec098321a7
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/341244

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341244
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=23766969e4df642dea41d371598d442f49efaf8c
Submitter: Jenkins
Branch: stable/mitaka

commit 23766969e4df642dea41d371598d442f49efaf8c
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df
    (cherry picked from commit ecf9c0d908ece88f21202e59c17109ec098321a7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/344647

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/344649

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/317340
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=573ded4401abd7e0f1090e31333d65ab160d4bc7
Submitter: Jenkins
Branch: master

commit 573ded4401abd7e0f1090e31333d65ab160d4bc7
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

[Alexander Zatserklyany (zatserklyany)]

Verified: https://custom-ci.infra.mirantis.net/job/10.0.custom.system_test/686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344649
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Submitter: Jenkins
Branch: stable/8.0

commit 2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344647
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7a88c7476dced6292dd9013ef5e7cd031f19c446
Submitter: Jenkins
Branch: stable/mitaka

commit 7a88c7476dced6292dd9013ef5e7cd031f19c446
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488