BVT network verify fails after deployment with Network verification not available because nodes ["3"] not available via mcollective.

[Tatyanka]

 Deploy ceph HA with RadosGW for objects

    Scenario:

            Create cluster with Neutron
            Add 3 nodes with controller role
            Add 3 nodes with compute and ceph-osd role
            Deploy the cluster
            Check ceph status
           Run verify network
            Run OSTF tests
            Check the radosqw daemon is started

Actual:
Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not avaliable because nodes ["3"] not avaliable via mcollective.
Inspect Astute logs for the details'

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "473"
  build_id: "473"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Tatyanka (tatyana-leontovich)]

https://172.18.160.103/job/8.0.ubuntu.bvt_2/425/
https://172.18.160.103/job/8.0.ubuntu.bvt_2/424/
http://172.18.66.221:8080/job/8.0.custom_system_test/1621/console
http://172.18.66.221:8080/job/8.0.custom_system_test/1622/

[Nastya Urlapova (aurlapova)]

BVT impacted issue should has Critical priority.

[Tatyanka (tatyana-leontovich)]

+ occurrence https://172.18.160.103/view/8.0_swarm/job/8.0.system_test.ubuntu.plugins.thread_db_separate_services/116/testReport/%28root%29/separate_db_service/separate_db_service/

https://172.18.160.103/view/8.0_swarm/job/8.0.system_test.ubuntu.ha_neutron_tun/119/testReport/%28root%29/ceph_rados_gw/ceph_rados_gw/

https://172.18.160.103/view/8.0_swarm/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/120/testReport/junit/%28root%29/separate_rabbit_service/separate_rabbit_service/

[Alexander Kislitsky (akislitsky)]

On the problem node we have multiple respawning of mcollective:

2016-01-25T13:03:03.529895+00:00 notice: nailgun-agent: mcollective start/running, process 2226
2016-01-25T13:03:03.530112+00:00 notice: nailgun-agent: I, [2016-01-25T04:09:34.736829 #1992] INFO -- : Wrote data to file '/etc/nailgun_uid'. Data: 6
2016-01-25T13:03:03.530340+00:00 warning: kernel: [ 98.315512] init: mcollective main process (2226) terminated with status 1
2016-01-25T13:03:03.530557+00:00 warning: kernel: [ 98.316898] init: mcollective main process ended, respawning
2016-01-25T13:03:03.530754+00:00 warning: kernel: [ 98.452798] init: mcollective main process (2234) terminated with status 1
2016-01-25T13:03:03.530969+00:00 warning: kernel: [ 98.454178] init: mcollective main process ended, respawning
2016-01-25T13:03:03.531164+00:00 warning: kernel: [ 98.587483] init: mcollective main process (2242) terminated with status 1
2016-01-25T13:03:03.531384+00:00 warning: kernel: [ 98.588889] init: mcollective main process ended, respawning
2016-01-25T13:03:03.531587+00:00 warning: kernel: [ 98.722250] init: mcollective main process (2250) terminated with status 1
2016-01-25T13:03:03.531788+00:00 warning: kernel: [ 98.723662] init: mcollective main process ended, respawning
2016-01-25T13:03:03.532002+00:00 warning: kernel: [ 98.857777] init: mcollective main process (2258) terminated with status 1
2016-01-25T13:03:03.532219+00:00 warning: kernel: [ 98.859190] init: mcollective main process ended, respawning
2016-01-25T13:03:03.532437+00:00 warning: kernel: [ 99.000164] init: mcollective main process (2266) terminated with status 1
2016-01-25T13:03:03.532653+00:00 warning: kernel: [ 99.001548] init: mcollective main process ended, respawning
2016-01-25T13:03:03.532868+00:00 warning: kernel: [ 99.135069] init: mcollective main process (2274) terminated with status 1
2016-01-25T13:03:03.533062+00:00 warning: kernel: [ 99.136461] init: mcollective main process ended, respawning
2016-01-25T13:03:03.533289+00:00 warning: kernel: [ 99.269164] init: mcollective main process (2282) terminated with status 1
2016-01-25T13:03:03.533971+00:00 warning: kernel: [ 99.270542] init: mcollective main process ended, respawning
2016-01-25T13:03:03.533971+00:00 warning: kernel: [ 99.405914] init: mcollective main process (2290) terminated with status 1
2016-01-25T13:03:03.533983+00:00 warning: kernel: [ 99.407302] init: mcollective main process ended, respawning
2016-01-25T13:03:03.534230+00:00 warning: kernel: [ 99.540136] init: mcollective main process (2298) terminated with status 1
2016-01-25T13:03:03.534453+00:00 warning: kernel: [ 99.541617] init: mcollective main process ended, respawning
2016-01-25T13:03:03.534678+00:00 warning: kernel: [ 99.675425] init: mcollective main process (2306) terminated with status 1
2016-01-25T13:03:03.534907+00:00 warning: kernel: [ 99.676895] init: mcollective respawning too fast, stopped

[Michael Polenchuk (mpolenchuk)]

the same https://bugs.launchpad.net/fuel/+bug/1531161

[Dmitry Pyzhov (dpyzhov)]

This bug is caused by /usr/bin/let-rename script hanged for a long time on bootstrap nodes. When it exits mcollective starts to work properly

[Dmitry Pyzhov (dpyzhov)]

Fixed by revert of workaround for bug 1529631

[Dmitry Pyzhov (dpyzhov)]

https://review.openstack.org/#/c/271748/

[Aleksey Zvyagintsev (azvyagintsev)]

Related issue:
https://bugs.launchpad.net/fuel/+bug/1538055

[Igor Marnat (imarnat)]

The bug which we reverted fix for had tag area-linux, thus I corrected area for this bug as well, changed area-mos to area-linux

[Timur Nurlygayanov (tnurlygayanov)]

It looks like the issue was fixed, not reproduced on the latest 20 builds of BVT / deb packages CI jobs.

Now tests failed because of other issues:
https://bugs.launchpad.net/fuel/+bug/1533082
https://bugs.launchpad.net/fuel/+bug/1534594

[Nastya Urlapova (aurlapova)]

@Timur, we don't close issue w/o attached Fuel version.

[Maksym Strukov (unbelll)]

Verified as fixed in 8.0-493

[Artem Panchenko (apanchenko-8)]

verified on 9.0 build #174