OSTF Heat failed - Authentication failed.

[Vladimir Khlyunev]

[root@nailgun ~]# fuel --fuel-version
astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
auth_required: true
build_id: 2015-05-23_04-08-32
build_number: '455'
feature_groups:
- mirantis
fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
      build_id: 2015-05-23_04-08-32
      build_number: '455'
      feature_groups:
      - mirantis
      fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
      fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
      fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
      nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

Traceback:
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/heatmanager.py", line 125, in _wait_for_stack_status
    interval):
  File "/usr/lib/python2.6/site-packages/fuel_health/test.py", line 63, in call_until_true
    elif func():
  File "/usr/lib/python2.6/site-packages/fuel_health/heatmanager.py", line 112, in check_status
    stack = self.heat_client.stacks.get(stack_id)
  File "/usr/lib/python2.6/site-packages/heatclient/v1/stacks.py", line 146, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/usr/lib/python2.6/site-packages/heatclient/common/http.py", line 254, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/heatclient/common/http.py", line 207, in _http_request
    % resp.content)
HTTPUnauthorized: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1

Steps to reproduce:
1. Deploy cluster:
-CentOS HA
-Neutron VLAN
-Ceph
-Sahara
2. Add 3 controller+ceph
3. Add 2 compute+ceph
4. Deploy cluster
5. Run OSTF tests for "Platform services functional tests."

Expected Result: all passed
Actual result:
"Typical stack actions: create, delete, show details, etc.", "Advanced stack actions: suspend, resume and check.", "Check stack rollback" and "Update stack actions: inplace, replace and update whole template." are failed.

[Kairat Kushaev (kkushaev)]

Guys, could you please attach a fuel snapshot to the bug (if possible).

[Kairat Kushaev (kkushaev)]

I analyzed the logs in depth.
It turns out that heat cannot authorize from time to time (!) using token provided by keystone clients.
For example, the observation of ostf-test log showed the following:
1. Heat successfully requests stack-creation.
2. With the same token it cannot authorize when executing stack-show.
3. With the same token it successfully creates a stack again
4. With the same token it cannot authorize when executing stack-show.
5. With the same token it successfully creates and executes stack-show.
6. With the same token it cannot delete a stack(unauthorized).
I wonder why it is so. Keystone logs shows nothing because debug mode is turned off which make debugging quite complex.
So I asked QA to redeploy the Fuel instance with debug=True.
After that I will try to reproduce an issue again.

[Kairat Kushaev (kkushaev)]

 I cannot reproduce a problem (possibly due to issues with RabbitMQ). As soon as issues will be fixed I will try again.

[Kairat Kushaev (kkushaev)]

Here is the log that demonstrates the behaviour above:
http://paste.openstack.org/show/241795/

[Boris Bobrov (bbobrov)]

I am not sure where the problem is. I see a trace in keystone logs, but it could happen only if galera/mysql stopped working. I am investigating the issue on a separate env now.

[Boris Bobrov (bbobrov)]

http://paste.openstack.org/show/242203/ -- the trace

[Vasily Gorin (vgorin)]

I thinkI have the same problem, but other steps to reproduce.

1. Create any environment with 3 cluster and 1 compute and Ubuntu.
2. Disable primary controller and wait for FUEL know that this node has gone away.
3. Run OSTF tests.

Actual result:
All Platform services functional tests were failed.

( All tests passed after enabling controller)
Log:
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 176, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 655, in _create_server
    self.status_timeout(client.servers, server.id, 'ACTIVE')
  File "/usr/lib/python2.6/site-packages/fuel_health/test.py", line 129, in status_timeout
    conf.compute.build_interval):
  File "/usr/lib/python2.6/site-packages/fuel_health/test.py", line 63, in call_until_true
    elif func():
  File "/usr/lib/python2.6/site-packages/fuel_health/test.py", line 116, in check_status
    thing = things.get(thing_id)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 563, in get
    return self._get("/servers/%s" % base.getid(server), "server")
  File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 93, in _get
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 489, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 467, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 440, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 433, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance could not be found (HTTP 404) (Request-ID: req-ef575f7a-15a7-4bed-917f-b5e963e70b8a)

[Vasily Gorin (vgorin)]

Sorry, I think I rushed. It is other problem.

[Andrey Sledzinskiy (asledzinskiy)]

Reproduced again on CentOS, HA, Neutron Vlan, Ceph for images+Rados for objects, 3 controller+ceph, 2 compute+cinder, 1 compute

[Boris Bobrov (bbobrov)]

Andrey, why there is no 10.109.10.2/var/log/docker-logs/remote/node-3.test.domain.local/heat-api.log? I see from the logs that HAProxy sent ALL requests that resulted in 401 to node-3.

[Boris Bobrov (bbobrov)]

It seems to be a duplicate of https://bugs.launchpad.net/mos/+bug/1454794 btw

[Vladimir Khlyunev (vkhlyunev)]

https://bugs.launchpad.net/mos/+bug/1454794 looks very similar, I will mark this bug as a duplicate after investigation.

[Boris Bobrov (bbobrov)]

Could not reproduce. Andrey Sledzinskiy could not reproduce it either.

[Sergii Golovatiuk (sgolovatiuk)]

Could be this patch related?

https://review.openstack.org/#/c/186478/

[Evgeny Sikachev (esikachev)]

I testing 500 iso with 6.1. Could not reproduce