HAproxy check of keystone/mysql fails after destroy on one of separate services node

[Alexandr Kostrikov]

Test fails on CI [0] with scenario
            1. Revert snapshot separate_all_service
            2. Destroy multirole node with rabbit master
            3. Wait HA is working
            4. Run OSTF
Step to check OSFT fails

Expected that OSTF passed

Acutal:
OSTF fails due to haproxy-check failures
haproxy-status.sh |grep DOWN
keystone-1 node-5 Status: DOWN/L4CON Sessions: 0 Rate: 0
keystone-2 node-5 Status: DOWN/L4CON Sessions: 0 Rate: 0
mysqld node-4 Status: DOWN/L4CON Sessions: 0 Rate: 0

[0] - https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/136/

Status can be lowered as there are similar bugs with medium priority

[Alexandr Kostrikov (akostrikov-mirantis)]

Reproduced at https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/132/testReport/(root)/separate_keystone_service_shutdown/

[Matthew Mosesohn (raytrac3r)]

I'm not seeing mysqld in haproxy-status.sh from primary controller on my env. I'm going to invalidate the mysql part of this bug. I have https://review.openstack.org/#/c/284019/ on review to fix the keystone part.

[Matthew Mosesohn (raytrac3r)]

fixed by https://review.openstack.org/#/c/284019/

[Alexandr Kostrikov (akostrikov-mirantis)]

I have started test to verify it http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom_system_test/2004/console

[Alexandr Kostrikov (akostrikov-mirantis)]

The check is blocked by another bug[0]. Waiting to be able to verify it. If it takes too long, will try it with local environment.
[0] https://bugs.launchpad.net/fuel/+bug/1549276

[Alexandr Kostrikov (akostrikov-mirantis)]

At 8.0 it passes.

[Alexandr Kostrikov (akostrikov-mirantis)]

Checked on swarm. There are different failures on CI.
It is possible that after fixes it will reappear.

[Alexandr Kostrikov (akostrikov-mirantis)]

Checked with https://product-ci.infra.mirantis.net/job/9.0.all/101/