BVT fail: failed to fetch packages from http://mirror-pkgs.vm.mirantis.net/

[Marek Zawadzki]

console log: http://jenkins-product.srt.mirantis.net:8080/job/7.0.staging.ubuntu.bvt_2/127/console

excerpt from astute log:

stderr: W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty/InRelease

W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty-updates/InRelease

W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty-security/InRelease

W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty/Release.gpg Unable to connect to mirror-pkgs.vm.mirantis.net:http:

W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty-updates/Release.gpg Unable to connect to mirror-pkgs.vm.mirantis.net:http:

W: Failed to fetch http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-06-170127/dists/trusty-security/Release.gpg Unable to connect to mirror-pkgs.vm.mirantis.net:http:

W: Some index files failed to download. They have been ignored, or old ones used instead.

[...]
2015-08-07T18:21:23 err: [663] Task '{"priority"=>100, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}' failed on node 1
2015-08-07T18:21:23 err: [663] No more tasks will be executed on the node 1
2015-08-07T18:21:23 debug: [663] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>100, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}}]}
2015-08-07T18:21:23 debug: [663] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>100, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/fuel_pkgs/fuel_pkgs.pp", "timeout"=>600, "cwd"=>"/"}}}]}
2015-08-07T18:21:23 info: [663] 919c1478-1881-4242-b53a-47b08508eed9: Finished deployment of nodes => roles: {"1"=>"primary-controller"}
2015-08-07T18:21:23 err: [663] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:177:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

[Sergey Kulanov (skulanov)]

Set prio to High, cause it was single case.

Folks, could you please check on your side, what was the issue with networking at that time?

Thanks

[Igor Shishkin (teran)]

@Pawel, any results here?

[Igor Shishkin (teran)]

Moving to 8.0 since there is no real progress started and it doesn't block 7.0

[Pawel Brzozowski (pbrzozowski)]

Investigation of logs gave no results. I could not find any network issues between MSK and CZ. Also host did not mark any overload problems etc. Hopefully this was only one time TCP connection issue.