Deleted node appears in nova service-list and host-list

[Kyrylo Romanenko]

Steps:
1. Deploy cluster with multiple controllers.

2 or 3 controllers+ironic
1 compute
1 ironic
etc

Other settings:
Compute QEMU
Network Neutron with VLAN segmentation
Storage Backends Cinder LVM over iSCSI for volumes

2. Run OSTF, verify that test "Check that required services are running" passed OK.
3. Delete first controller that registered as node-1 in nodes list.
4. Redeploy changes.

5. Run OSTF sanity tests.
Expected that "Check that required services are running" will pass.
Actual result: test fails on step:
"2. Check there are no failed services (with XXX state)."

What`s seen in health-check log: http://paste.openstack.org/show/488181/

Nova service-list: http://paste.openstack.org/show/488182/

nova host-list: http://paste.openstack.org/show/488183/

We can see the host "node-1.test.domain.local" in all outputs. So node-1 still appears in nova, while it is already deleted from the cluster and does not appear in "fuel node" table.

It looks that issue does not affects booting of nova instances, but it affect running of OSTF test "Check that required services are running" and possibly can cause other impacts.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Roman Podoliaka (rpodolyaka)]

This is not a bug, but rather an enhancement request: when we delete a node from a cluster in Fuel, we should also update the runtime configuration of OpenStack services, e.g. delete the appropriate services entries in Nova/Cinder/Neutron/etc.

Not sure, what's the best place to track this and who should own this (Library/Nailgun/Puppets?).