After deletion of 2 controllers from HA-cluster /etc/hosts wasn't cleaned up

[Vladimir Khlyunev]

I got this bug on iso 7.0-301 + MU1, but clear 7.0-301 can be affected too.

Steps to reproduce:
1) Deploy cluster: 3 controllers + 1 compute, Neutron VLAN
2) Remove 2 controllers, re-deploy
3) Check hosts /etc/hosts on all nodes
4) Run OSTF

Result:
/etc/hosts on remaining nodes wasn't changed
also nodes became unavailable by dns name

I will keep my env - feel free to request it

[Matthew Mosesohn (raytrac3r)]

nodes that are being deleted are not included in astute.yaml, so the hosts.pp granular module cannot delete these entries. It can only add entries for those nodes that are present in astute.yaml.

We could try to introduce an array of hosts that are being deleted so we can perform whatever cleanup tasks are necessary. I believe it needs an entire "nodes hash" for each one, just so we can identify its roles, ips, network roles, etc, and ensure all proper migrations can take place.

[Dmitry Klenov (dklenov)]

Matthew, do you see feasible to perform 'set' operation for /etc/hosts file and put only hosts available in astute.yaml there? opposing to apply 'add' / 'delete' operations for new / obsolete hosts correspondingly.

[Andrey Sledzinskiy (asledzinskiy)]

There are also problem after adding new controllers with getting these nodes by dns names:
Steps to reproduce:
1) Deploy cluster: 1 controller + 1 compute, Neutron VLAN
2) Add 2 controllers, re-deploy

Actual result - /etc/hosts was updated on all nodes but nodes aren't available by dns names

[Matthew Mosesohn (raytrac3r)]

Andrey, that's related to a different bug with Fuel Master patching. Your steps aren't going to cause that bug.

[Andrey Sledzinskiy (asledzinskiy)]

Please, disregard my comment, it was env problem

[Michael Polenchuk (mpolenchuk)]

* Nailgun should inject info of removed nodes as well (orchestrator/tasks_serializer.py (class UploadNodesInfo))
* Modify osnailyfacter/modular/hosts/hosts.pp to handle removed nodes

[Stanislaw Bogatkin (sbogatkin)]

Addressed by https://review.openstack.org/#/c/245169/

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 7.0-updates because of Medium importance

[Bogdan Dobrelya (bogdando)]

The fix got stuck, fuel-web folks please merge the dependency

[Dmitry Pyzhov (dpyzhov)]

Moved to 9.0 because of SCF in 8.0

[Vladimir Khlyunev (vkhlyunev)]

Wont fix - medium but after SCF.

[Nastya Urlapova (aurlapova)]

@Dima, it is not Medium issue, because as you can see also from #4 other scenarios are affected. We can put into MU1?

[Dmitry Pyzhov (dpyzhov)]

@Nastya, comment #4 is not applicable to this bug, see comment #6. Moving the bug back to medium. Also we have very small chance to successfully backport fixes related to lifecycle management because this feature is in active development and doesn't present in previous releases.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Polenchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/245169
Reason: outdated

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Polenchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/245135
Reason: outdated

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Dmitry Pyzhov (dpyzhov)]

Leaving as "Won't fix" for 8.0 because fix depends on new task based deployment feature.

[Alexander Kislitsky (akislitsky)]

Fix should be done by introducing yaql based task which stores removed nodes hash. After that /etc/hosts can be fixed on the all nodes on the library side. Reassigning to Matthew.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/309319

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309319
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=eacb493e38a31747dc61a43a081adf00adbeb8eb
Submitter: Jenkins
Branch: master

commit eacb493e38a31747dc61a43a081adf00adbeb8eb
Author: Matthew Mosesohn <email address hidden>
Date: Fri Apr 22 12:43:04 2016 +0300

    Purge deleted /etc/hosts entries for deleted ndoes

    Deleted nodes now will get stored in a file
    /etc/hiera/deleted_nodes.yaml and be inserted into
    the hiera hierarchy.

    host entries for any records present in deleted_nodes.yaml
    will be purged when evaluating hosts task.

    The source YAML for deleted nodes is generated on master
    because nodes that are deployed from scratch or from Error
    state will not be able to identify deleted nodes due to
    certain orchestration limitations.

    Change-Id: I01db215e77a3532a6fa7bf46ab7e20e281e8c165
    Closes-Bug: #1513401

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/315684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/315684
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c923fb21ef584e2e09bd4c514f01872d3874abac
Submitter: Jenkins
Branch: stable/mitaka

commit c923fb21ef584e2e09bd4c514f01872d3874abac
Author: Matthew Mosesohn <email address hidden>
Date: Fri Apr 22 12:43:04 2016 +0300

    Purge deleted /etc/hosts entries for deleted ndoes

    Deleted nodes now will get stored in a file
    /etc/hiera/deleted_nodes.yaml and be inserted into
    the hiera hierarchy.

    host entries for any records present in deleted_nodes.yaml
    will be purged when evaluating hosts task.

    The source YAML for deleted nodes is generated on master
    because nodes that are deployed from scratch or from Error
    state will not be able to identify deleted nodes due to
    certain orchestration limitations.

    Change-Id: I01db215e77a3532a6fa7bf46ab7e20e281e8c165
    Closes-Bug: #1513401
    (cherry picked from commit eacb493e38a31747dc61a43a081adf00adbeb8eb)

[Egor Kotko (ykotko)]

The verification blocked by
https://bugs.launchpad.net/fuel/+bug/1582263

[Oleksiy Molchanov (omolchanov)]

Verified on #465. Passed.