Fuel 6.1 - Network Verification Fails with four NICs are Bonded together as Bond0

[Bruce Basil Mathews]

Created lacp-balanced-tcp bond of four interfaces from two different NIC crds:

Network Verification fails. When I first kick of there is this notification

21-07-2015 20:12:59
Iface eth1 on node Untitled (d4:9c) configured to use lacp-balance-tcp mode as part of bond0. Repo availability verification for this node will be skipped.
Iface eth1 on node Untitled (65:18) configured to use lacp-balance-tcp mode as part of bond0. Repo availability verification for this node will be skipped.
Iface eth1 on node Untitled (b5:f4) configured to use lacp-balance-tcp mode as part of bond0. Repo availability verification for this node will be skipped.

Then it fails with this message

Verification failed.
Method verify_networks. 5e577f24-4083-4954-8a13-7b9c3b78418f: MCollective call failed in agent 'net_probe', method 'check_url_retrieval', failed nodes: ID: 11 - Reason: execution expired ID: 12 - Reason: execution expired ID: 14 - Reason: execution expired ID: 13 - Reason: execution expired . Inspect Astute logs for the details

And here are the errors I see in the astute logs:

Date Level Message
2015-07-21 20:23:31 ERR

[921] Error running RPC method verify_networks: 5e577f24-4083-4954-8a13-7b9c3b78418f: MCollective call failed in agent 'net_probe', method 'check_url_retrieval', failed nodes:
ID: 11 - Reason: execution expired
ID: 12 - Reason: execution expired
ID: 14 - Reason: execution expired
ID: 13 - Reason: execution expired
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:112:in `check_results_with_retries'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:60:in `method_missing'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/network.rb:121:in `check_urls_access'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:161:in `check_repositories'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:153:in `check_repositories'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:129:in `block in verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:127:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:127:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-07-21 20:23:31
 ERR

[921] MCollective call failed in agent 'net_probe', method 'check_url_retrieval', failed nodes:
ID: 11 - Reason: execution expired
ID: 12 - Reason: execution expired
ID: 14 - Reason: execution expired
ID: 13 - Reason: execution expired

2015-07-21 20:05:17 ERR

[906] Error running RPC method verify_networks: 0844deb3-826a-4572-bc97-0ebc0e9e56da: MCollective call failed in agent 'net_probe', method 'check_url_retrieval', failed nodes:
ID: 9 - Reason: execution expired
ID: 10 - Reason: execution expired
ID: 7 - Reason: execution expired
ID: 8 - Reason: execution expired
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:112:in `check_results_with_retries'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/mclient.rb:60:in `method_missing'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/network.rb:121:in `check_urls_access'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:161:in `check_repositories'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:153:in `check_repositories'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:129:in `block in verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:127:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:127:in `verify_networks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-07-21 20:05:17 ERR

[906] MCollective call failed in agent 'net_probe', method 'check_url_retrieval', failed nodes:
ID: 9 - Reason: execution expired
ID: 10 - Reason: execution expired
ID: 7 - Reason: execution expired
ID: 8 - Reason: execution expired

2015-07-21 19:09:39 ERR

[929] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-07-21 19:09:39 ERR

[929] No more tasks will be executed on the node 1

2015-07-21 19:09:39 ERR

[929] Task '{"priority"=>100, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}' failed on node 1

2015-07-21 19:09:20 ERR

[929] No more tasks will be executed on the node 3

2015-07-21 19:09:20 ERR

[929] Task '{"priority"=>100, "type"=>"puppet", "uids"=>["3"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}' failed on node 3

2015-07-21 19:09:20 ERR

[929] No more tasks will be executed on the node 2

2015-07-21 19:09:20 ERR

[929] Task ‘{"priority"=>100, "type"=>"puppet", "uids"=>["2"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}' failed on node 2

[Oleksiy Molchanov (omolchanov)]

Moving to incomplete, please attach diagnostic snapshot.

[Denis Kostryukov (dkostryukov)]

Hello All.
You can download the diagnostic snapshot from this link:
https://drive.google.com/a/mirantis.com/file/d/0B1eJLkMydYiecVhvSGxDOWlRMUk/view?usp=sharing

[Alex Schultz (alex-schultz)]

Denis,

Have you confirmed that the 4 port bond works correctly and has internet connectivity by itself? The error points to the repository url checker that is part of network checker which will result in this error if the repositories are unavailable due to no connectivity.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205698
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=d02af0200e8a67a0597d5301f8281a536f60f413
Submitter: Jenkins
Branch: master

commit d02af0200e8a67a0597d5301f8281a536f60f413
Author: Alex Schultz <email address hidden>
Date: Fri Jul 24 15:21:06 2015 -0500

    Add timeout value to 'urlaccesscheck check'

    This change adds a configurable timeout to 'urlaccesscheck check' to
    ensure the script does not hang if there are connectivity issues with
    the URL that is being checked. The default timeout is 60 seconds but
    is configurable via a command line option.

    This change also adds netifaces to the network_checker requirements.txt
    because it was not added as part of
    I910c15c2b39a917eb8428bb69271b5dde364b639 so test and installation may
    fail on a clean environment.

    Change-Id: I363712a6501cc21b89601d5373b5633d5518ca81
    Closes-Bug: 1478138
    Related-Bug: 1471085
    Related-Bug: 1476831
    Closed-Bug: 1478622

[Richard Parker (richard-parker)]

Hello,
Is there a work-around for this?
I am receiving this error also.
Curious as my main interface (eth0) is able to get to the internet.
Thanks,
Rick

[Luz Cazares (luz-cazares)]

I'm getting this error, Is there a workaround? I'm behind a proxy so should I change something in the configuration?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/281789

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tony Breeds (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/281789
Reason: This branch (stable/6.1) is at End Of Life