deployment error on /etc/puppet/modules/osnailyfacter/modular/tools/tool.pp

[Lope Beltran II]

Deployed on Fuel 6.1 - CentOS HA setup with:
 4 controllers
 11 compute
 2 storage
 3 MongoDB (error was encountered here while installing Openstack)

2015-07-15 06:55:58 INFO

[542] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"b946381e-6d42-4e01-87ca-ebdcb2a8291b", "status"=>"error", "error"=>"Method granular_deploy. Deployment failed on nodes 1.\nInspect Astute logs for the details"}}

2015-07-15 06:55:58 ERR

[542] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2015-07-15 06:55:58 INFO

[542] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"b946381e-6d42-4e01-87ca-ebdcb2a8291b", "nodes"=>[{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-mongo", "task"=>{"priority"=>500, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/tools/tools.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}

2015-07-15 06:55:58 INFO

[542] b946381e-6d42-4e01-87ca-ebdcb2a8291b: Finished deployment of nodes => roles: {"1"=>"primary-mongo"}

2015-07-15 06:55:58 ERR

[542] No more tasks will be executed on the node 1

2015-07-15 06:55:58 ERR

[542] Task '{"priority"=>500, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/tools/tools.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 1

2015-07-15 06:55:52 INFO

[542] Retrying to run puppet for following error nodes: 1

2015-07-15 06:55:46 INFO

[542] Retrying to run puppet for following error nodes: 1

2015-07-15 06:55:37 INFO

[542] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"b946381e-6d42-4e01-87ca-ebdcb2a8291b", "nodes"=>[{"uid"=>"1", "progress"=>100, "status"=>"deploying", "role"=>"primary-mongo", "task"=>{"priority"=>500, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/tools/tools.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}

2015-07-15 06:55:35 INFO

[542] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"b946381e-6d42-4e01-87ca-ebdcb2a8291b", "nodes"=>[{"uid"=>"1", "progress"=>48, "status"=>"deploying", "role"=>"primary-mongo", "task"=>{"priority"=>500, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/tools/tools.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}

[Oleksiy Molchanov (omolchanov)]

MOS team, please check, there is an error in Puppet

repodata/primary.xml.gz from mos-security: [Errno 256] No more mirrors to try

[Lope Beltran II (beltranlope)]

Also tried deploying 1 controller 1 compute. Same result.

I think problem persists from this thread https://review.openstack.org/#/c/185915/ even after patch.

[Alexei Sheplyakov (asheplyakov)]

Some bits of repo metadata are missing (in particular, repodata/primary.xml.gz) at http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security => reassigning to OSCI team.

[Alexei Sheplyakov (asheplyakov)]

> I think problem persists from this thread https://review.openstack.org/#/c/185915/ even after patch.

That patch has little to do with the (yum) repo inconsistency (and can't possibly fix it).

[Dmitry Burmistrov (dburmistrov)]

>[node-15.eee.upd.edu.ph] out: http://10.158.55.2:8080/2014.2.2-6.1/centos/security/repodata/primary.xml.gz: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

This bug doesn't relatet to OSCI team

[Lope Beltran II (beltranlope)]

I tried correcting the centos mirrors but same result.

[542] Error running RPC method granular_deploy
[518] Node 20 has failed to deploy. There is no more retries for puppet run.
[518] Node 20(primary-controller) status: error

[Oleksiy Molchanov (omolchanov)]

Can you ping 10.158.55.2 from your node-1?
Can you wget http://10.158.55.2:8080/2014.2.2-6.1/centos/security/repodata/primary.xml.gz from node-1?

[Lope Beltran II (beltranlope)]

Found the problem (like what alexei said).

I mirrored the repo for centos for a local one. But it seems some files mirrored are old files. I checked the online repo, all seems correct, but once you download some of the files, the older version seems to be the one downloaded.

The following files are outdated when mirrored: (but shows the correct date on the website)
/security/repodata/repomd.xml
/updates/repodata/repomd.xml
/updates/repodata/primary.xml.gz

[Oleksiy Molchanov (omolchanov)]

@Lope Beltran II, can we close this issue?

[Lope Beltran II (beltranlope)]

yes. thank you!

[lidiwen (lidiwencs)]

Hi Lope,
I encounter the same problem, would you please tell how to solve this issue?

Do I need change my local mirror to what?

my config on Fuel UI like this: I list the repo URI.
Name URI
mos http://192.168.40.24:8080/2014.2.2-6.1/centos/x86_64
mos-updates http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates
mos-security http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security
Auxiliary http://192.168.40.24:8080/2014.2.2-6.1/centos/auxiliary

@Lope Beltran II (beltranlope)

Thanks for your kindly help.
diwen li

[Lope Beltran II (beltranlope)]

Hi,

Sorry for the late reply on this thread. It got buried in my mail. At that
time I created a local repo for mos-updates and security. It just happens
that there was an inconsistency with the mirror site which was corrected a
few days later. After retrying the deployment, it succeeded.

Regards,
---------------------------------------------------------------------
Lope A. Beltran II
University of the Philippines Diliman
Contact Number : [phone]
---------------------------------------------------------------------

On Wed, Oct 28, 2015 at 11:04 AM, lidiwen <[e-mail]> wrote:

> Hi Lope,
> I encounter the same problem, would you please tell how to solve this
> issue?
>
> Do I need change my local mirror to what?
>
> my config on Fuel UI like this: I list the repo URI.
> Name URI
> mos
> http://192.168.40.24:8080/2014.2.2-6.1/centos/x86_64
> mos-updates
> http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates
> mos-security
> http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security
> Auxiliary
> http://192.168.40.24:8080/2014.2.2-6.1/centos/auxiliary
>
> @Lope Beltran II (beltranlope)
>
> Thanks for your kindly help.
> diwen li
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1474718
>
> Title:
> deployment error on
> /etc/puppet/modules/osnailyfacter/modular/tools/tool.pp
>
> Status in Fuel for OpenStack:
> Invalid
> Status in Fuel for OpenStack 6.1.x series:
> Invalid
> Status in Fuel for OpenStack 7.0.x series:
> Invalid
>
> Bug description:
> Deployed on Fuel 6.1 - CentOS HA setup with:
> 4 controllers
> 11 compute
> 2 storage
> 3 MongoDB (error was encountered here while installing Openstack)
>
> 2015-07-15 06:55:58 INFO
>
> [542] Casting message to Nailgun: {"method"=>"deploy_resp",
> "args"=>{"task_uuid"=>"b946381e-6d42-4e01-87ca-ebdcb2a8291b",
> "status"=>"error", "error"=>"Method granular_deploy. Deployment failed
> on nodes 1.\nInspect Astute logs for the details"}}
>
> 2015-07-15 06:55:58 ERR
>
> [542] Error running RPC method granular_deploy: Deployment failed on
> nodes 1, trace:
>
> ["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:68:in
> `block (3 levels) in deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in
> `each_slice'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:51:in
> `block (2 levels) in deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in
> `each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:49:in
> `block in deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in
> `each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/deployment_engine.rb:45:in
> `deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:179:in
> `deploy_cluster'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:56:in
> `granular_deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/ge...