Health Check: Check that required services are running test failed

[Vladimir]

Steps to reproduce:

1. Create new environment
2. Choose Neutron, VLAN
3. Add 3 controller
4. Deploy the environment
5. Add 2 compute
6. Deploy the environment
7. Add 1 new compute node and delete one deployed compute
8. Re-deploy
9. Run OSTF tests
10. Delete one compute node
11. Re-deploy cluster
12. Run OSTF

After steps 8 and 11 'Check that required services are running' failed:
Some nova services have not been started..

`nova-manage service list` command on compute node after 8th step shows:
root@node-1:~# nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary Host Zone Status State Updated_At
nova-scheduler node-1.test.domain.local internal enabled :-) 2015-12-11 18:25:12
nova-conductor node-1.test.domain.local internal enabled :-) 2015-12-11 18:26:01
nova-cert node-1.test.domain.local internal enabled :-) 2015-12-11 18:25:09
nova-consoleauth node-1.test.domain.local internal enabled :-) 2015-12-11 18:25:27
nova-cert node-3.test.domain.local internal enabled :-) 2015-12-11 18:25:25
nova-consoleauth node-3.test.domain.local internal enabled :-) 2015-12-11 18:25:26
nova-scheduler node-3.test.domain.local internal enabled :-) 2015-12-11 18:25:27
nova-conductor node-3.test.domain.local internal enabled :-) 2015-12-11 18:25:27
nova-consoleauth node-2.test.domain.local internal enabled :-) 2015-12-11 18:25:37
nova-cert node-2.test.domain.local internal enabled :-) 2015-12-11 18:25:37
nova-conductor node-2.test.domain.local internal enabled :-) 2015-12-11 18:25:38
nova-scheduler node-2.test.domain.local internal enabled :-) 2015-12-11 18:25:38
nova-compute node-4.test.domain.local nova enabled :-) 2015-12-11 18:25:14
nova-compute node-5.test.domain.local nova enabled XXX 2015-12-11 18:00:21
nova-compute node-6.test.domain.local nova enabled :-) 2015-12-11 18:26:01

`nova-manage service list` command on compute node after 11th step shows:
root@node-1:~# nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary Host Zone Status State Updated_At
nova-scheduler node-1.test.domain.local internal enabled :-) 2015-12-11 18:50:12
nova-conductor node-1.test.domain.local internal enabled :-) 2015-12-11 18:51:01
nova-cert node-1.test.domain.local internal enabled :-) 2015-12-11 18:51:09
nova-consoleauth node-1.test.domain.local internal enabled :-) 2015-12-11 18:50:27
nova-cert node-3.test.domain.local internal enabled :-) 2015-12-11 18:50:25
nova-consoleauth node-3.test.domain.local internal enabled :-) 2015-12-11 18:50:26
nova-scheduler node-3.test.domain.local internal enabled :-) 2015-12-11 18:50:27
nova-conductor node-3.test.domain.local internal enabled :-) 2015-12-11 18:50:27
nova-consoleauth node-2.test.domain.local internal enabled :-) 2015-12-11 18:50:37
nova-cert node-2.test.domain.local internal enabled :-) 2015-12-11 18:50:37
nova-conductor node-2.test.domain.local internal enabled :-) 2015-12-11 18:50:38
nova-scheduler node-2.test.domain.local internal enabled :-) 2015-12-11 18:50:38
nova-compute node-4.test.domain.local nova enabled :-) 2015-12-11 18:50:14
nova-compute node-5.test.domain.local nova enabled XXX 2015-12-11 18:00:21
nova-compute node-6.test.domain.local nova enabled XXX 2015-12-11 18:40:01

Snapshot could be found:
https://drive.google.com/open?id=0BzGc8pMVuherXzVvRl9uSDNycWc

[Artem Panchenko (apanchenko-8)]

Hi,

please add info about Fuel version. Also the snapshot you attached can't be downloaded due to permissions issue.
BTW, looks like a duplicate of bug #1516993

[Vladimir (vushakov)]

New snapshot link:
https://drive.google.com/file/d/0BzGc8pMVuherXzVvRl9uSDNycWc/view?usp=sharing

Fuel version:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "263"
  build_id: "263"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "6534117233a5bdc51d7d47361bc7d511e4b11e6f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "26adf12c320936a97a9b0a84169a6e58c530e848"

[Artem Panchenko (apanchenko-8)]

According to the logs you removed 2 compute node before running health checks:

2015-12-11 18:01:15 INFO [64] Total list of nodes to remove: ["node-5"]
2015-12-11 18:40:49 INFO [71] Total list of nodes to remove: ["node-6"]

So failure of test 'Check that required services are running' is expected, see bug #1471172