Failed deploy with vCenter and CephOSD nodes

[okosse]

Step for reproduce:
1. Create ENV with vCenter enabled and CephOSD as backend for glance
2. Add 3 controller nodes
3. Add 2 CephOSD nodes
3. Deploy cluster

Obtained result: Deployment has failed. Method granular_deploy. Deployment failed on nodes 1.
Astute log:
[689] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:64:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:47:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:47:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:45:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:41:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:41:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:131:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

--------iso version--------------
api: '1.0'
astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9
auth_required: true
build_id: 2015-03-19_22-54-44
build_number: '210'
feature_groups:
- mirantis
fuellib_sha: 7764225db5bc653563309912afbb4058283c808b
fuelmain_sha: f3d6353c08d8eb709c7ab100b56dc2bebef4157f
nailgun_sha: 1d2bd383caecc5ec3f86bf93ccca940326f23e97
ostf_sha: b9a090c71682fbea5d9351051827d7d654d07be3
production: docker
python-fuelclient_sha: b223dcaf5fdad2f714cd245958fefe03995d6207
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9
      build_id: 2015-03-19_22-54-44
      build_number: '210'
      feature_groups:
      - mirantis
      fuellib_sha: 7764225db5bc653563309912afbb4058283c808b
      fuelmain_sha: f3d6353c08d8eb709c7ab100b56dc2bebef4157f
      nailgun_sha: 1d2bd383caecc5ec3f86bf93ccca940326f23e97
      ostf_sha: b9a090c71682fbea5d9351051827d7d654d07be3
      production: docker
      python-fuelclient_sha: b223dcaf5fdad2f714cd245958fefe03995d6207
      release: '6.1'

[Ryan Moe (rmoe)]

There was an error with Ceph installation on node-1. Here is the error from the puppet logs:

2015-03-20T10:50:56.104271+00:00 err: (/Stage[main]/Ceph::Params/Package[pushy]/ensure) change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install pushy' returned 1: Error: Nothing to do
2015-03-20T10:50:56.104603+00:00 debug: (/Stage[main]/Ceph::Params/Package[pushy]) Nothing to manage: no ensure and the resource doesn't exist

The pushy package was removed from our repos on 3/19 from what I can tell. This is only an issue on CentOS.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/166337

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/166337
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=59f19988e3a46e0a462483218445cb3f6c819ab9
Submitter: Jenkins
Branch: master

commit 59f19988e3a46e0a462483218445cb3f6c819ab9
Author: Ryan Moe <email address hidden>
Date: Fri Mar 20 10:19:42 2015 -0700

    Don't install pushy when using Ceph

    ceph-deploy no longer depends on pushy and it has been
    removed from our repos.

    Change-Id: I3d3a423c9902c6f5da594c00982355d3995e919a
    Closes-bug: #1434515

[Tatyana Dubyk (tdubyk)]

Verified on centos, vcenter+qemu (6.1.all - 234 iso)
api: '1.0'
astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
auth_required: true
build_id: 2015-03-26_22-54-44
build_number: '234'
feature_groups:
- mirantis
fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
production: docker
python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
      build_id: 2015-03-26_22-54-44
      build_number: '234'
      feature_groups:
      - mirantis
      fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
      fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
      nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
      ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
      production: docker
      python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
      release: '6.1'

[Bogdan Dobrelya (bogdando)]

@Ryan, please update the status of the 6.0.1 backport. This issue has a critical priority and requires updates ASAP

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/175074

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Moe (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/175074

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.0-updates as we don't expect new 6.0 deployments