OSTF failed due to Keystone client unavailability after disable of vip__public

[Kyrylo Romanenko]

Steps:
1. Deploy any environment with 3 controllers and nova-network with FlatDHCP, Ubuntu.
2. Checked where public_vip works:
# pcs status
...
 vip__public (ocf::fuel:ns_IPaddr2): Started node-6.domain.tld
...

2. Deleted public_vip:
root@node-6:~# pcs resource disable vip__public

3. Run OSTF tests.
All tests failed with error:
"Keystone client is not available. Please, refer to OpenStack logs to fix this problem"
But we can use keystone: "keystone user-list" on every node.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "521"
  build_id: "2015-06-08_06-13-27"
  nailgun_sha: "4340d55c19029394cd5610b0e0f56d6cb8cb661b"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "7766818f079881e2dbeedb34e1f67e517ed7d479"
  fuel-library_sha: "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d"
  fuel-ostf_sha: "7c938648a246e0311d05e2372ff43ef1eb2e2761"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"

[Tatyanka (tatyana-leontovich)]

How long dare you waiting after fail-over and before ostf run?

[Tatyanka (tatyana-leontovich)]

Just ignore my comment, I miss that you disabled piblic_vip resource

[Ihor Kalnytskyi (ikalnytskyi)]

@Kyrylo,

What do are you expect here?

1/ You have disabled 'vip__public' resource, so it's stopped and won't be started by cluster again (according to pacemaker's documentation). Therefore, you're unable to deal with OpenStack cluster via public network.

2/ OSTF goes to cluster via fueladmin network and using haproxy as proxy. Haproxy forwards OSTF traffic from fueladmin network to public network.

3/ That basically means that OSTF is unable to get access to OpenStack env.

4/ The 'keystone user-list' is working because it uses management network to get access to the OpenStack services.

So as for me, it's invalid. Please correct me if I'm wrong.

[Tatyanka (tatyana-leontovich)]

Agree with Igor with some comments:
Actually ostf try to use proxy and public_vip to get token from keystone, and in this case for sure it will fail according do vip was stopped

[Kyrylo Romanenko (kromanenko)]

I expected to restart 'vip__public' resource on another controller automatically, because we have Failover testcase:
https://mirantis.testrail.com/index.php?/tests/view/951450
"Delete public_vip 2 times for Nova-network"
Scenario:
1. Deploy any environment with 3 controllers and nova-network with FlatDHCP or VLAN Manager
2. Delete public_vip 2 times
3. Verify networks
4. Run OSTF tests

Should i delete public_vip by some other way than through pcs?

[Alexander Gubanov (ogubanov)]

My understanding of phrase "2. Delete public_vip 2 times" means:

1. # find controller with VIP
for i in $(fuel 2>/dev/null node | grep controller | awk '{print $1}'); do ssh 2>/dev/null node-$i "ip netns exec haproxy ip -o -4 a | grep -q hapr-p && hostname" ; done

2. # ssh to node and delete VIP
ssh 2>/dev/null <NODE> "ip netns exec haproxy ip addr del \$(ip netns exec haproxy ip -o -4 a s dev hapr-p | awk '{print \$4}') dev hapr-p"

3. wait few seconds repeat step 1

Please correct me if I'm wrong.