Deployment is good, but tests failed with 'Could not find any hosts in discovery data provided' message

Bug #1437213 reported by Stanislaw Bogatkin
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Łukasz Oleś
6.0.x
Invalid
Undecided
Unassigned

Bug Description

Failed job: http://jenkins-product.srt.mirantis.net:8080/job/6.1.ubuntu.smoke_neutron/159/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "235"
  build_id: "2015-03-27_03-04-29"
  nailgun_sha: "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf"
  python-fuelclient_sha: "e5e8389d8d481561a4d7107a99daae07c6ec5177"
  astute_sha: "f595715750a2c4820722a96e0236f5c89ca6521c"
  fuellib_sha: "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"
  ostf_sha: "a4cf5f218c6aea98105b10c97a4aed8115c15867"
  fuelmain_sha: "320b5f46fc1b2798f9e86ed7df51d3bda1686c10"

Deployment passed without errors, but after it in astute log we can see next errors:

2015-03-27T04:50:44 info: [673] Processing RPC call 'verify_networks'
2015-03-27T04:50:44 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'systemtype', method 'get_type', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:50:44 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'systemtype', method 'get_type', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:50:44 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'systemtype', method 'get_type', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:51:15 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'net_probe', method 'dhcp_discover', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"Spawning listener for eth1 failed.\nNetwork for iface eth1 is down.Network for iface eth1 is down.Network for iface eth1 is down.", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.4\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}
2015-03-27T04:51:15 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'net_probe', method 'dhcp_discover', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.3\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}
2015-03-27T04:51:15 debug: [673] f6c022a6-c64b-4dda-9062-d565f8db3565: MC agent 'net_probe', method 'dhcp_discover', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"Spawning listener for eth1 failed.\nNetwork for iface eth1 is down.Network for iface eth1 is down.Network for iface eth1 is down.", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.5\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}
2015-03-27T04:51:15 debug: [673] DHCP checker received: #<MCollective::RPC::Result:0x000000027d44c8 @agent="net_probe", @action="dhcp_discover", @results={:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"Spawning listener for eth1 failed.\nNetwork for iface eth1 is down.Network for iface eth1 is down.Network for iface eth1 is down.", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.4\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}>
2015-03-27T04:51:15 debug: [673] DHCP checker received: #<MCollective::RPC::Result:0x000000027a9d18 @agent="net_probe", @action="dhcp_discover", @results={:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.3\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}>
2015-03-27T04:51:15 debug: [673] DHCP checker received: #<MCollective::RPC::Result:0x0000000278afa8 @agent="net_probe", @action="dhcp_discover", @results={:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:err=>"Spawning listener for eth1 failed.\nNetwork for iface eth1 is down.Network for iface eth1 is down.Network for iface eth1 is down.", :out=>"[{\"server_id\": \"10.109.0.2\", \"iface\": \"eth0\", \"yiaddr\": \"10.109.0.5\", \"mac\": \"64:02:4b:22:8c:23\", \"server_ip\": \"10.109.0.2\", \"dport\": 67, \"message\": \"offer\", \"gateway\": \"0.0.0.0\"}]\n", :status=>0}}>
2015-03-27T04:51:15 info: [673] Casting message to Nailgun: {"method"=>"check_dhcp_resp", "args"=>{"task_uuid"=>"f6c022a6-c64b-4dda-9062-d565f8db3565", "status"=>"ready", "progress"=>100, "nodes"=>[{:uid=>"1", :status=>"ready", :data=>[{"server_id"=>"10.109.0.2", "iface"=>"eth0", "yiaddr"=>"10.109.0.4", "mac"=>"64:02:4b:22:8c:23", "server_ip"=>"10.109.0.2", "dport"=>67, "message"=>"offer", "gateway"=>"0.0.0.0"}]}, {:uid=>"2", :status=>"ready", :data=>[{"server_id"=>"10.109.0.2", "iface"=>"eth0", "yiaddr"=>"10.109.0.3", "mac"=>"64:02:4b:22:8c:23", "server_ip"=>"10.109.0.2", "dport"=>67, "message"=>"offer", "gateway"=>"0.0.0.0"}]}, {:uid=>"3", :status=>"ready", :data=>[{"server_id"=>"10.109.0.2", "iface"=>"eth0", "yiaddr"=>"10.109.0.5", "mac"=>"64:02:4b:22:8c:23", "server_ip"=>"10.109.0.2", "dport"=>67, "message"=>"offer", "gateway"=>"0.0.0.0"}]}]}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'systemtype', method 'get_type', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'systemtype', method 'get_type', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'systemtype', method 'get_type', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:node_type=>"target"}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'rpcutil', method 'inventory', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:agents=>["systemtype", "discovery", "net_probe", "fake", "puppetd", "puppetsync", "erase_node", "rpcutil", "execute_shell_command", "uploadfile"], :facts=>{"mcollective"=>"1"}, :classes=>[], :version=>"@DEVELOPMENT_VERSION@", :main_collective=>"mcollective", :collectives=>["mcollective"], :data_plugins=>["agent_data", "fstat_data"]}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'rpcutil', method 'inventory', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:agents=>["systemtype", "discovery", "net_probe", "fake", "puppetd", "puppetsync", "erase_node", "rpcutil", "execute_shell_command", "uploadfile"], :facts=>{"mcollective"=>"1"}, :classes=>[], :version=>"@DEVELOPMENT_VERSION@", :main_collective=>"mcollective", :collectives=>["mcollective"], :data_plugins=>["agent_data", "fstat_data"]}}
2015-03-27T04:51:16 debug: [673] 5f313591-9a16-459f-9ba0-26a7817f61ba: MC agent 'rpcutil', method 'inventory', results: {:sender=>"2", :statuscode=>0, :statusmsg=>"OK", :data=>{:agents=>["systemtype", "discovery", "net_probe", "fake", "puppetd", "puppetsync", "erase_node", "rpcutil", "execute_shell_command", "uploadfile"], :facts=>{"mcollective"=>"1"}, :classes=>[], :version=>"@DEVELOPMENT_VERSION@", :main_collective=>"mcollective", :collectives=>["mcollective"], :data_plugins=>["agent_data", "fstat_data"]}}
2015-03-27T04:51:16 err: [673] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>

2015-03-27T04:51:21 err: [673] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>

2015-03-27T04:51:26 err: [673] No more retries for MCollective client instantiation after exception:
["/usr/lib64/ruby/gems/2.1.0/gems/mcollective-client-2.4.1/lib/mcollective/rpc/client.rb:498:in `discover'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/mclient.rb:152:in `initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/mclient.rb:41:in `initialize'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:34:in `new'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:34:in `get_versions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:51:in `split_on_version'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/network.rb:40:in `check_network'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:105:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:135:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-03-27T04:51:26 err: [673] Error running RPC method verify_networks: #<RuntimeError: Could not find any hosts in discovery data provided>
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/mclient.rb:163:in `rescue in initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/mclient.rb:147:in `initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/mclient.rb:41:in `initialize'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:34:in `new'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:34:in `get_versions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/version.rb:51:in `split_on_version'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/network.rb:40:in `check_network'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:105:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:135:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-03-27T04:51:26 info: [673] Casting message to Nailgun: {"method"=>"verify_networks_resp", "args"=>{"task_uuid"=>"5f313591-9a16-459f-9ba0-26a7817f61ba", "status"=>"error", "error"=>"Method verify_networks. #<RuntimeError: Could not find any hosts in discovery data provided>\n.\nInspect Astute logs for the details"}}

Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dima Shulyak (dshulyak)
Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

Seems the reason of failure is:

"Spawning listener for eth1 failed.\nNetwork for iface eth1 is down."

But message "Could not find any hosts in discovery data provided" is not related to this.

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

eth1 on 10.109.0.4 and 10.109.0.5 were in down state on that time.

Revision history for this message
Dima Shulyak (dshulyak) wrote :

I think It is not a problem, during dhcp verification it is ok for some interfaces to be down.
The problem is that mcollective discovery was performed with empty or malformed uid list.

I will try to do debugging tomorrow, but i cant guarantee that i will be able to work for a long time on this issue..

Changed in fuel:
assignee: Dima Shulyak (dshulyak) → Fuel Python Team (fuel-python)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/168549

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Łukasz Oleś (loles)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/168549
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=d9ac1261db009329cf746c88983305b3ea3d8d7f
Submitter: Jenkins
Branch: master

commit d9ac1261db009329cf746c88983305b3ea3d8d7f
Author: Łukasz Oleś <email address hidden>
Date: Sat Mar 28 05:29:40 2015 +0100

    Get version only when agent version is available

    Change-Id: I8c6391636908a494a6de1ca0480e5d5150801105
    Closes-Bug: #1437213

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Łukasz Oleś (loles) wrote :

It's only 6.1 bug. It doesn't occur in 6.0

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Download full text (4.3 KiB)

The same error on fuel-6.1-379-2015-05-05_12-04-34.iso
---
2015-05-07T09:06:32 err: [688] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>

2015-05-07T09:06:37 err: [688] Retrying RPC client instantiation after exception: #<RuntimeError: Could not find any hosts in discovery data provided>

2015-05-07T09:06:42 err: [688] No more retries for MCollective client instantiation after exception:
["/usr/lib64/ruby/gems/2.1.0/gems/mcollective-client-2.4.1/lib/mcollective/rpc/client.rb:498:in `discover'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:152:in `initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:41:in `initialize'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:377:in `new'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:377:in `unlock_nodes_discovery'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:67:in `rescue in provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:44:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:94:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:50:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:37:in `image_provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-05-07T09:06:43 err: [688] Error running provisioning: #<RuntimeError: Could not find any hosts in discovery data provided>
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:163:in `rescue in initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:147:in `initialize_mclient'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:41:in `initialize'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:377:in `new'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:377:in `unlock_nodes_discovery'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:67:in `rescue in provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/provision.rb:44:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/as...

Read more...

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Alexander, could you please provide cluster configuration, exact scenario and diagnostic snapshot from yours deployment? It is necessary for investigating if the issue reproduced or it is an another issue.

Revision history for this message
Łukasz Oleś (loles) wrote :

It's completely different scenario. This bug was caused because of broken network validation.

Error from comment #8 is caused by image based provisioning and probably was fixed some time ago as IBP is working now

tags: added: on-verification
Revision history for this message
Sergey Novikov (snovikov) wrote :

Verified on fuel-6.1-450-2015-05-22_03-56-41.iso.

Steps to verify *:
    1. Create cluster
    2. Add 1 node with controller role
    3. Add 2 nodes with compute role
    4. Deploy the cluster
    5. Run network verification
    6. Run OSTF

* Group of tests is "deploy_neutron_gre"

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.