Keystone client is not available. Please, refer to OpenStack logs to fix this problem

[Alexander Kurenyshev]

Steps to reproduce:
1. Create cluster (Centos, Nova Flat)
2. Add 3 nodes with controller roles
3. Add 2 nodes with compute roles
4. Deploy the cluster
5. Validate cluster was set up correctly, there are no dead
services, there are no errors in logs
6. Run verify networks
7. Run OSTF

Expected result:
OSTF passed

Actual result:
OSTF failed with error: Keystone client is not available.

At the logs OSTF Logs:
2015-05-20 05:24:14 DEBUG (v2) Making authentication request to http://10.109.11.2:5000/v2.0/tokens

At the same time at the keystone logs:
2015-05-20 05:24:14.180 30741 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.6/site-packages/keystone/middleware/core.py:270
<...>
2015-05-20 05:24:16.842 30741 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.6/site-packages/keystone/middleware/core.py:270

Finally at the ostf log:
2015-05-20 05:24:16 DEBUG (session) Request returned failure status: 500
2015-05-20 05:24:16 WARNING (config) Can not pass authorization with proxy on 10.109.10.5
2015-05-20 05:24:16 DEBUG (config) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/config.py", line 722, in find_proxy
    insecure=False)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: This region is already configured with backend: <keystone.common.kvs.backends.memcached.MemcachedBackend object at 0x7f4bb6356d90> (Disable debug mode to suppress these details.) (HTTP 500)

Fuel used:
api: '1.0'
astute_sha: 96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548
auth_required: true
build_id: 2015-05-19_10-05-51
build_number: '437'
feature_groups:
- mirantis
fuel-library_sha: 2814c51668f487e97e1449b078bad1942421e6b9
fuel-ostf_sha: 9ce1800749081780b8b2a4a7eab6586583ffaf33
fuelmain_sha: 68796aeaa7b669e68bc0976ffd616709c937187a
nailgun_sha: 593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548
      build_id: 2015-05-19_10-05-51
      build_number: '437'
      feature_groups:
      - mirantis
      fuel-library_sha: 2814c51668f487e97e1449b078bad1942421e6b9
      fuel-ostf_sha: 9ce1800749081780b8b2a4a7eab6586583ffaf33
      fuelmain_sha: 68796aeaa7b669e68bc0976ffd616709c937187a
      nailgun_sha: 593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

[Andrey Maximov (maximov)]

guys, this issue blocks our bvt please fix asap

[Alexander Makarov (amakarov)]

Please provide the env: I cant download 43M logs here

[Alexander Kurenyshev (akurenyshev)]

Dmitry, this are keystone logs you requested

[Alexander Kurenyshev (akurenyshev)]

Alexander, environment is on the CI server srv28-bud.bud.mirantis.net, please use your domain account. Connect to the console you can with `ssh root@10.109.10.2`

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/6841

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184615

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184615
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7d21e927d5e132a9019cbed1199c5d1bd2d1f70d
Submitter: Jenkins
Branch: master

commit 7d21e927d5e132a9019cbed1199c5d1bd2d1f70d
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 20 23:35:18 2015 +0300

    Switch keystone WSGI app to be run as multiprocess

    This commit makes apache spin up a bunch of
    keystone processes in order to avoid race
    condition when keystone cannot safely
    initialize KVS region

    Change-Id: Ia816dad7365c68b2d1144fe315f4bafdf08d16ce
    Partial-bug: #1457037

[Vladimir Kuklin (vkuklin)]

The fix above should fix the issue completely.

[Sergey Novikov (snovikov)]

Verified on fuel-6.1-466-2015-05-25_20-55-26.iso.

Steps to verify:
    1. Create cluster (Centos, Nova Flat)
    2. Add 3 nodes with controller roles
    3. Add 2 nodes with compute roles
    4. Deploy the cluster
    5. Validate cluster was set up correctly, there are no dead
         services, there are no errors in logs
    6. Run verify networks
    7. Run OSTF

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/6841
Reason: Not needed, issue solved in fuel

[Alexander Makarov (amakarov)]

Make application initialization a critical section:
https://review.fuel-infra.org/#/c/10188/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10188
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 98400036628f9ea4557638a0aec57ec9a0da866a
Author: Alexander Makarov <email address hidden>
Date: Thu Aug 6 16:09:59 2015

Make application initialization a critical section

Fix the race condition in multithreaded setup

Change-Id: Ia619cc7a1a4e4ee4d7332c3e3e215cea00ca3328
Closes-Bug: 1457037

[Alexander Makarov (amakarov)]

Patch https://review.fuel-infra.org/#/c/10188/

Tried out #140 ISO (MOS 7.0)
Set up env at 172.16.57.178
1 controller/1 compute
Configured Apache to run keystone in 1 admin and 1 main process 10 threads each.
OSTF: 16 tests failed without the patch, 14 tests failed with the patch.

Leontiy, can you please check this patch as appropriate? In the description it is said env had 3 controller and 2 compute nodes.

[Maksym Strukov (unbelll)]

Can't reproduce on 7.0-155. Neutron+Ubuntu, 3 conrtollers, 2 computes

{"build_id": "2015-08-10_09-22-35", "build_number": "155", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-10_09-22-35", "build_number": "155", "api": "1.0", "fuel-library_sha": "1851b4dff75170dbd63f6e15cde734e348e86d27", "nailgun_sha": "fba3442befbef2eeb2f870b19d1833a229d242b3", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "bdca75d0256338519c7eddd8a840ee6ecba7f992"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1851b4dff75170dbd63f6e15cde734e348e86d27", "nailgun_sha": "fba3442befbef2eeb2f870b19d1833a229d242b3", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "bdca75d0256338519c7eddd8a840ee6ecba7f992"}

[Tatyanka (tatyana-leontovich)]

Reopen issues according it reproduced on CI, but decrease priority

Authorization Failed: An unexpected error prevented the server from fulfilling your request: This region is already configured with backend: <keystone.common.kvs.backends.memcached.MemcachedBackend object at 0x7f11c40e8a10> (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-d16ea895-80c9-4871-8357-b13764cbecc0)

Failed test
https://product-ci.infra.mirantis.net/view/7.0_swarm/job/7.0.system_test.ubuntu.ha_neutron_destructive_2/24/testReport/

Scenario:
""Check l3-agent rescheduling after reset non-primary controller

        Scenario:
            1. Revert snapshot with neutron cluster
            2. Create an instance with a key pair
            3. Manually reschedule router from primary controller
               to another one
            4. Reset controller with l3-agent
            5. Check l3-agent was rescheduled
            6. Check network connectivity from instance via
               dhcp namespace
            7. Run OSTF

Actual result:
OSTF tests are failed with
Authorization Failed: An unexpected error prevented the server from fulfilling your request: This region is already configured with backend: <keystone.common.kvs.backends.memcached.MemcachedBackend object at 0x7f11c40e8a10> (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-d16ea895-80c9-4871-8357-b13764cbecc0)

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "265"
  build_id: "265"
  nailgun_sha: "0dfcf73deb8ae99654f3da2ea95b7b68b9ee7273"
  python-fuelclient_sha: "9643fa07f1290071511066804f962f62fe27b512"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "e63709d16bd4c1949bef820ac336c9393c040d25"
  fuel-library_sha: "4fdf3d6b070204366593012428395d173698678a"
  fuel-ostf_sha: "582a81ccaa1e439a3aec4b8b8f6994735de840f4"
  fuelmain_sha: "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/11100

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11100
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 8f5b482b14255ee623559f8e2ddbc8f3fabb82ac
Author: Boris Bobrov <email address hidden>
Date: Thu Sep 3 13:05:28 2015

Move region configuration to a critical section

Change-Id: I65f85aedd5b087499b889540417b9502e050ce7c
Closes-bug: 1457037

[Tatyanka (tatyana-leontovich)]

Verified 270 ISO , work fine, thank you folks !!

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/13348

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/13350

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/13350
Reason: not needed in 8.0

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/13348
Reason: not needed in 8.0