[scale] Unable to update hosts file while deleting node - Host[node-8.test.domain.local] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/hosts/hosts.pp:25 on node node-1.test.domain.local

[Vladimir Khlyunev]

ISO 1561

Steps:
1. Deploy cluster 3 controller+cinder+ceph, 2 computes,
   Neutron VXLAN, cinder for volumes, ceph for images + Rados GW
2. Verify networks
3. Run OSTF
4. Add 1 ceph+cinder+controller
5. Re-deploy cluster
6. Verify networks
7. Run OSTF
8. Add 1 new ceph+cinder+controller and delete one already deployed
    ceph+cinder+controller
9. Re-deploy cluster
10. Verify networks
11. Run OSTF
12. Delete one ceph+cinder+controller
13. Redeploy cluster <-- fail here

Result:
2017-04-03 23:59:02 +0000 Puppet (err): Duplicate declaration: Host[node-8.test.domain.local] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/hosts/hosts.pp:25 on node node-1.test.domain.local

Possible reason - node under deletion is multi-role node, this error is not occurring when we are deleting node with single role

Snapshot https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.ha_scale_group_5/12/artifact/logs/fail_error_add_delete_compute_cinder_ceph-fuel-snapshot-2017-04-03_23-59-09.tar

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/453147

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/453171

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453171
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3f01bb9d23c1a408912736e1137f6c309530a1d6
Submitter: Jenkins
Branch: stable/newton

commit 3f01bb9d23c1a408912736e1137f6c309530a1d6
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 4 15:05:46 2017 +0300

    Use intersection of added vs deleted nodes instead of just deleted ones

    There are cases when nailgun can send us data where there are nodes
    included into list of added nodes and deleted ones. As a result, puppet
    manifests failed with resource ensurance for present and absent hosts.
    To avoid this situation, subtract added nodes from deleted prior to
    actually deleting them from hosts.

    Change-Id: I515f7fae4bbe67123d70572a4f861e879ec580f6
    Closes-Bug: #1679522

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453147
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=21c7651fe55d3737383de673d564b472da75e788
Submitter: Jenkins
Branch: master

commit 21c7651fe55d3737383de673d564b472da75e788
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 4 15:05:46 2017 +0300

    Use intersection of added vs deleted nodes instead of just deleted ones

    There are cases when nailgun can send us data where there are nodes
    included into list of added nodes and deleted ones. As a result, puppet
    manifests failed with resource ensurance for present and absent hosts.
    To avoid this situation, subtract added nodes from deleted prior to
    actually deleting them from hosts.

    Change-Id: I515f7fae4bbe67123d70572a4f861e879ec580f6
    Closes-Bug: #1679522

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/453558

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 10 build #1568
https://custom-ci.infra.mirantis.net/job/10.0.custom.system_test/1665/console

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453558
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6ddeee899a0018f820ffcab0465f3cd337040d9d
Submitter: Jenkins
Branch: stable/mitaka

commit 6ddeee899a0018f820ffcab0465f3cd337040d9d
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 4 15:05:46 2017 +0300

    Use intersection of added vs deleted nodes instead of just deleted ones

    There are cases when nailgun can send us data where there are nodes
    included into list of added nodes and deleted ones. As a result, puppet
    manifests failed with resource ensurance for present and absent hosts.
    To avoid this situation, subtract added nodes from deleted prior to
    actually deleting them from hosts.

    Change-Id: I515f7fae4bbe67123d70572a4f861e879ec580f6
    Closes-Bug: #1679522
    (cherry picked from commit 21c7651fe55d3737383de673d564b472da75e788)

[Ilya Bumarskov (ibumarskov)]

Verified on 9.2 MU 2 (MOS_UBUNTU_ID = 9.0-2017-05-30-154422)