[Fuel Upgrade] Can't re-deploy environment after upgrade - failed with ceph-deploy --overwrite-conf config pull node-16 returned 1 instead of one of [0]

[Andrey Sledzinskiy]

{
    "build_id": "2014-08-05_00-31-14",
    "mirantis": "yes",
    "build_number": "164",
    "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
    "nailgun_sha": "04ada3cd7ef14f6741a05fd5d6690260f9198095",
    "production": "docker",
    "api": "1.0",
    "fuelmain_sha": "77423fe5a01196fb11776c4db9ee6ae93bfcd4d9",
    "astute_sha": "6db5f5031b74e67b92fcac1f7998eaa296d68025",
    "release": "5.0.1",
    "fuellib_sha": "dec8421a18e3ccfa35978b36b5de3292b53e6b69"
}

Steps:
1. Create next 5.0-26 cluster - Ubuntu, Simple, Neutron Vlan, Ceph for images, Sahara, Ceilometer, 1 controller+mongo, 2 compute, 2 ceph nodes
2. Deploy cluster
3. Upload 5.0.1 upgrade script to master
4. Run upgrade
5. After successful upgrade add 1 compute to existing cluster
6. Run re-deploy

Expected - cluster is re-deployed successfully
Actual - re-deployment failed with next errors in puppet log on node-21:
2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:18:in `evaluate'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `each'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:82:in `perform_changes'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:193:in `sync'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:120:in `sync'

2014-08-06 11:46:49 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'

2014-08-06 11:46:49 ERR

 ceph-deploy --overwrite-conf config pull node-16 returned 1 instead of one of [0]

Logs are attached

The same for 5.1 upgrade script

[Evgeniy L (rustyrobot)]

Set won't fix because stable/5.0 is frozen for non Critical bugs.

[Andrew Woodward (xarses)]

9 times out of 10 this only occurs due to the pulling node being un-able to reach the management interface on the first controller.

This needs a reproducer

[Dmitry Borodaenko (angdraug)]

That's an Incomplete, how can it be release-notes?

[Andrey Sledzinskiy (asledzinskiy)]

I haven't reproduced this issue on 5.1 upgrade script after posting this bug but It's constantly reproduced after upgrade to 5.0.1

[Vladimir Kuklin (vkuklin)]

The bug is that we have different ssh keys generated inside the astute container. We need to mount astute container inside the host node.

[Vladimir Sharshov (vsharshov)]

Solution: share directory '/var/lib/astute' between master node and astute container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114223

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114211
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=af3d1922bfc21345f81be3454115ab6139675c35
Submitter: Jenkins
Branch: stable/5.0

commit af3d1922bfc21345f81be3454115ab6139675c35
Author: Dima Shulyak <email address hidden>
Date: Thu Aug 14 14:45:52 2014 +0300

    Save astute keys and port them back to astute

    Closes-Bug: 1353497
    Change-Id: I5df75fc71fc7ac5d0fb6221caf6ab8097052eb71

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114223
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=7d7e862ccbdbaa36785b95028d37ac91d7aca440
Submitter: Jenkins
Branch: master

commit 7d7e862ccbdbaa36785b95028d37ac91d7aca440
Author: Dima Shulyak <email address hidden>
Date: Thu Aug 14 14:45:52 2014 +0300

    Save astute keys and port them back to astute

    Closes-Bug: 1353497
    Change-Id: I5df75fc71fc7ac5d0fb6221caf6ab8097052eb71

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "2014-08-14_19-52-36",
    "mirantis": "yes",
    "build_number": "170",
    "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
    "nailgun_sha": "af3d1922bfc21345f81be3454115ab6139675c35",
    "production": "docker",
    "api": "1.0",
    "fuelmain_sha": "fd58828f404e4298ed338e8f44c6a326cebd31de",
    "astute_sha": "6db5f5031b74e67b92fcac1f7998eaa296d68025",
    "release": "5.0.1",
    "fuellib_sha": "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"

}

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "2014-08-21_02-01-17",
    "ostf_sha": "c6ecd0137b5d7c1576fa65baef0fc70f9a150daa",
    "build_number": "464",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "25eba6fbb2047f26d9da4d27ffdb742c9c27832a",
    "production": "docker",
    "fuelmain_sha": "25a0c228d998707f90e90877559f17817a749d2f",
    "astute_sha": "efe3cb3668b9079e68fb1534fd4649ac45a344e1",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "52f3ebfa968f0338e0584edf47cff10911109de5"

}