verify_networks failed due to nodes are inaccessible via mcollective

Bug #1665670 reported by Ivan Suzdal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Vladimir Sharshov

Bug Description

Detailed bug description:
For some reason verify_network task failed with 'Method verify_networks. Network verification not available because nodes ["1", "2", "3", "4", "5", "6"] not available via mcollective.
Inspect Astute logs for the details'.

astute.log contains:
"""
2017-02-17 11:04:51 INFO [19461] Casting message to Nailgun:
{"method"=>"verify_networks_resp",
 "args"=>
  {"task_uuid"=>"a1371c78-17e8-43f9-952f-2db205d73d30", "status"=>"running"}}
2017-02-17 11:04:51 INFO [19461] Processing RPC call 'verify_networks'
2017-02-17 11:05:01 ERROR [19461] Error running RPC method verify_networks: Network verification not available because nodes ["1", "2", "3", "4", "5", "6"] not available via mcollective, trace:
["/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:218:in `validate_nodes_access'",

 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:in `block in perform_main_job'"]
"""

When I reverted snapshot, I've been able to ping nodes via 'mco ping':
"""
# mco ping
bootstrap time=81.86 ms
master time=84.37 ms
bootstrap time=87.86 ms
bootstrap time=90.96 ms
^C
"""
But it looks like nodes has different names than astute expect.

Reproducibility:
I've seen it at least three times. List of failed jobs:
https://packaging-ci.infra.mirantis.net/job/10.0-pkg-mos-systest-ubuntu-xenial/622
https://packaging-ci.infra.mirantis.net/job/10.0-pkg-mos-systest-ubuntu-xenial/625
https://packaging-ci.infra.mirantis.net/job/10.0-pkg-mos-systest-ubuntu-xenial/626

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: none → 10.1
Revision history for this message
Ivan Suzdal (isuzdal) wrote :

It happened due to desync fuel-agent packages for master node and slaves

Changed in fuel:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.