Mistral fails to maintain a keystone session while deploying an overcloud

[Emilien Macchi]

Trying to deploy a containerized overcloud from a containerized undercloud in OVB environment, the overcloud gets deployed but Mistral Executor fails when Zaqar is trying to post the message on the queue:

https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/var/log/containers/mistral/executor.log.txt.gz#_2018-04-04_01_56_12_631

https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/var/log/containers/zaqar/zaqar.log.txt.gz#_2018-04-04_01_56_10_669

https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/var/log/containers/keystone/keystone.log.txt.gz#_2018-04-04_01_56_10_548

Maybe irrelevant but this messages popup in our logs:
Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet?

Therefore, the overcloud fails to finish the deployment.
I've been working on aligning the mistral/zaqar configurations:
https://trello.com/c/GMYssQ9b/44-align-openstack-configs-with-tht-and-instack-undercloud
But it didn't help; so now wondering about key rotations etc.
Note that we haven't hit this bug in multinode jobs; maybe because job is faster than OVB? Do we have some sort of expiration?

[Juan Antonio Osorio Robles (juan-osorio-robles)]

I don't think it's a token provider configuration issue, since we only have 2 keys by default. And the deployment seems to start and go forward for quite a while: https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

from there I can see it goes up to step 5, and in the end it fails with this exception: No JSON object could be decoded

which I guess comes from mistral client.

At some point in the zaqar logs I can see that it fails with authorization failed:
https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/var/log/containers/zaqar/zaqar.log.txt.gz#_2018-04-04_01_56_10_669

which gets reflected in the mistral executor logs here https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/var/log/containers/mistral/executor.log.txt.gz#_2018-04-04_01_56_12_627

which is what Emilien reported.

I think that the issue is that mistral (server) is not refreshing the token that zaqar is using. The token works for a while and expires after an hour (which is what we configure). The theory has backup data because of the log timings:

The deploy starts at 0:55
https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-04-04_00_55_54

And we see the error at 1:55
https://logs.rdoproject.org/56/542556/100/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z64d11a27268e46db803351bb52f7cc25/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-04-04_01_55_56

So ultimately it seems to me that it's an issue on how mistral creates the client (in a way that doesn't refresh the keystone tokens). This should have been handled already though, and it does seem to me that mistral is using sessions correctly (as far as I can tell). Are we using an old mistral container?

This would have usually gotten handled by the session object from keystoneauth1, which I thought was being used in zaqar

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Going through the zaqarclient codebase, it seems to me that they don't use keystone sessions to do http requests, instead, they use the session to get the token explicitly and further in their "transport drivers" they build another object to do the requests. in the case of the http driver, they create another requests session and put the headers explicitly there. This is where they do authentication:

https://github.com/openstack/python-zaqarclient/blob/master/zaqarclient/auth/keystone.py#L175

And they seem to use a deprecated method (get_token), however, that code does seem to trigger reauthentication (deep into the rabbithole). So... I'm not sure what the issue is. But I think it's either in zaqarclient or mistral.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/558922

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558922
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=02cacfd53a9cf168d7e1967faef3211333800eb6
Submitter: Zuul
Branch: master

commit 02cacfd53a9cf168d7e1967faef3211333800eb6
Author: Emilien Macchi <email address hidden>
Date: Wed Apr 4 13:42:41 2018 -0700

    undercloud: increase token expiration time

    We did it in the past (3 years ago!) in instack-undercloud:
    https://github.com/openstack/instack-undercloud/commit/43e792c6844d4a7081b718d7f89b0c40f5cfb708
    in the context of: https://bugzilla.redhat.com/show_bug.cgi?id=1235908

    This time, we have the same problem when the undercloud is
    containeirized.
    This patch is actually setting parity with keystone config from
    instack-undercloud, but also raising an actual issue that will be
    addressed this cycle.

    In the meantime, let's increase the token expiration so we can move
    forward with testing the containerized undercloud.

    Change-Id: Iceaaf53fae44b5bcda9f6517f163939ba6be3d49
    Related-Bug: #1761050

[Dougal Matthews (d0ugal)]

therve told me about some testing related to this bug, it seems you can replicate the same issue with the heat actions. So this is likely a bug specific to Mistral.

[Dougal Matthews (d0ugal)]

We have a related Mistral bug: https://bugs.launchpad.net/mistral/+bug/1595084

[Dougal Matthews (d0ugal)]

Added the security-hardening tag as we now require a 4 hour Keystone token that has security disadvantages.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572448

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brad P. Crochet (<email address hidden>) on branch: master
Review: https://review.openstack.org/572448
Reason: Will see if Change-Id: Ia2a19a3fcd8808475a16d4d439e085e62a00dfdc works better

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dougal Matthews (<email address hidden>) on branch: master
Review: https://review.openstack.org/462056
Reason: Abandoning in favor of...
https://review.openstack.org/#/c/572448/
https://review.openstack.org/#/c/585904/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585904
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=0f17aecbe9a45744341fa4ca1a498de4f57ed531
Submitter: Zuul
Branch: master

commit 0f17aecbe9a45744341fa4ca1a498de4f57ed531
Author: Brad P. Crochet <email address hidden>
Date: Wed Jul 25 18:40:43 2018 -0400

    Use keystone group for loading auth params

    This really needs to come from mistral-extra.

    Change-Id: I8c1ff35df46347c2f247f74720942f9884908449
    Partial-Bug: #1595084
    Partial-Bug: #1761050

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brad P. Crochet (<email address hidden>) on branch: master
Review: https://review.opendev.org/572448

[kobig (kobi.ginon)]

hi @d0ugal , @alex-schultz
i wonder why this change is abandoned ? (unless i misunderstood)
i m having thee same error's and the same scenario with containerized undercloud
with Rocky version
The scenario mainly reproduces on large deployment (more then 30 Blades)
can you suggest a solution ?

[Renat Akhmerov (rakhmerov)]

Hi, we're looking for volunteers to take this bug. Ideally, from the TripleO team. Anybody is interested? The Mistral core team won't likely be able to take it in Ussuri 2.

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo