Keystone is unavailable after blocking br-mgmt interface on first controller

[Vasily Gorin]

https://product-ci.infra.mirantis.net/view/7.0_swarm/job/7.0.system_test.ubuntu.ha_neutron_destructive/lastCompletedBuild/testReport/

build 284
Disconnect controllers and check pacemaker status is correct

        Scenario:
              Preconditions:
                 1. Create cluster
                 2. Add 3 nodes with controller roles
                 3. Add 2 nodes with compute roles
                 4. Deploy the cluster

            1. Block traffic on br-mgmt of the first controller
            2. Check pacemaker status
            3. Wait until MySQL Galera is UP on some controller
            4. Run OSTF

Expected:
All OSTF tests passed except one of it.

Actual result:
 OSTF tests failed during connection to keystone.

[Ihor Kalnytskyi (ikalnytskyi)]

All checks on this job is failed.

  1/ ERROR: Verify connection is present and
  2/ FAIL: Disconnect controllers and check pacemaker status is correct
  3/ FAIL: Check 3 in 1 rabbit failover

Obviously, that's about our deployment and not about OSTF. Move it to Fuel Library.

BTW, I found the following info in the logs -

On the controllers node-5.test.domain.local and node-3.test.domain.local, resource master_p_rabbitmq-server has different list of nodes where it is started.. Please refer to OpenStack logs for more details.

[Vladimir Kuklin (vkuklin)]

From what I see in the logs it seems that you are picking incorrect nodes for tests. E.g. for Check 3 in 1 rabbit you are comparing node-5 (which was stopped by corosync) and node-3 which is in a healthy partition. Please recheck that this is not the case for all other tests also.

[Sergii Golovatiuk (sgolovatiuk)]

If you manually do

iptables -I INPUT -i br-mgmt -j DROP && iptables -I OUTPUT -o br-mgmt -j DROP

on primary controller, then try to run OSFT tests from UI. All tests will fail

If you unblock br-mgmt by

iptables -D INPUT -i br-mgmt -j DROP && iptables -D OUTPUT -o br-mgmt -j DROP

wait a minute then run OSTF and it will work normally.