Service tokens seems to be incorrectly implemented in kolla-ansible

[Franciszek Przewoźny]

Hi all,

It seems, that configuration done by kolla-ansible regarding service tokens seems to be not compliant with the documentation. If I'm right, this might mean that OSSA-2023-003 isn't fully fixed there.

https://docs.openstack.org/cinder/2023.2/configuration/block-storage/service-token.html#configuration

1. Send service token
  Section [service_user] is not defined at all for Cinder: https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/cinder/templates/cinder.conf.j2, Nova got it defined: https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/nova/templates/nova.conf.j2#L211

2. Service role
  From documentation "A service role is nothing more than a Keystone role that allows a deployment to identify a service without the need to make them admins", and later: "The default service role is service, but we can use a different name or even have multiple service roles", where in kolla-ansible, Nova service jinja does not have parameter to be set for service_token_roles, and it uses default "service", Cinder on the other hand have HARDCODED "admin" value (https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/cinder/templates/cinder.conf.j2#L119), and that's where I'm a bit lost. I thought that service users "nova" and "cinder" should have only f.e. "service" role in "service" project, not some multiple roles including "admin" one.

3. Policies not prepared for "service" role only
  Configuration of other role "service" for services (tested with Neutron and Placement), cannot be done as there are policies restricting some API calls, f.e.:

  Neutron:
  ERROR neutron.notifiers.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403)

  Policy restricting that call is:
  "os_compute_api:os-server-external-events:create": "rule:context_is_admin"

  And rule:context_is_admin definition is:
  "context_is_admin": "role:admin"

  Which totally blocks any effort put into getting rid of "admin" service role for other services.

My environment isn't kolla-ansible, but I'm using kolla's jinja templates as a configuration reference sometimes. I'm raising this ticket, as it's a bit suspicious for me that something else is configured in kolla, than is described in manual.

Best regards,
Franciszek

[sean mooney (sean-k-mooney)]

[service_user] for services that work like not is used for making request to other services
in the context of the cve cinder does not actually need that section.

note ironic does not use a [service_user] section it uses the details form the relevant service section i.e. cinder instead and assumes that it has the admin and service roles.

so cinder does not need the [service_user] section for the cve mitigation in its config.
it may use it for the other uses of service_user which is to test that when calling other services with a user token that may expire it was valid when cinder received it and should be honoured as if it was still valid by the receiving service.

you are correct that policies in services are not prepared for service role only in general which is why we recommend using both the service and admin role for the user configured in [service_user] in the interm period.

the i have not looked at the details of all of kolla config but i proposed the original patch for the service_user support for nova in kolla
https://github.com/openstack/kolla-ansible/commit/ddadaa282e72cc437470859766ac963ac757a26a

since my original version the patch was updated to use the admin role when verifying the service token in cinder

https://github.com/openstack/kolla-ansible/commit/ddadaa282e72cc437470859766ac963ac757a26a#diff-61b022281f112ed49c0169e67dd999658a41397df88acb4c224904d1c11b03cdR110

that is not correct and that could have security implications.
the service role is intentionally referenced in the upstream docs regarding this because its a role a normal admin.human should not have.

it was intentionally used so that admin could not delete cinder volume atachemetn with there normal admin accounts.

by enforceing the token has the admin role instead that protection has been removed but the cve is not reintudced for end users as normal users are still blocked.

kolla should be adding the service role to the service accounts (nova neutron ectra) and they should not be setting service_token_roles = admin in the cinder config.

[Sven Kieske (s-kieske)]

Thanks for the pointers Sean, I added this topic to our Whiteboard, to discuss it next week.

Maybe I also can take a look at the code a little earlier and propose a new patch:

https://etherpad.opendev.org/p/KollaWhiteBoard#L60

[Sven Kieske (s-kieske)]

I guess this comment chain is the culprit:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/comments/bb24d1f0_252d8b74

to quote it:

myself:

> question, mainly for mnasiadka:

> should we add the cinder patch to all other services in this patch also?

> That would mean validating service tokens for every service.

> We could maybe also argue to split this work up in a second/more changesets.

> I have a slight preference to split this up to not let this security bugfix wait for much longer.

> What do you think?

mnasiadika:

> I think we should split the rest, it doesn't seem to be required now.

then in patchset 4:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/3..4/ansible/roles/cinder/templates/cinder.conf.j2

we have:

```
# TODO(mnasiadka): Work out if that's the desired approach or should we add service role to nova user
service_token_roles = admin

then in patchset 6 the comment got removed:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/5..6/ansible/roles/cinder/templates/cinder.conf.j2

I didn't comment on that because I thought mnasiadka did indeed had checked that this was necessary.
```

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/908145

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909002

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Bartosz Bezak <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/815577
Reason: Kolla community decided not to implement service role for all service users in one change. Rather then doing it selectively. As not all openstack projects implemented service role support yet.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909002
Committed: https://opendev.org/openstack/kolla-ansible/commit/01c359665b53ef51837bd2bde649f969e0223d7f
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 01c359665b53ef51837bd2bde649f969e0223d7f
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909084

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909085

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909084
Committed: https://opendev.org/openstack/kolla-ansible/commit/ec9a30ca9c60e329bc69a95ceef3e3496b17f61e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit ec9a30ca9c60e329bc69a95ceef3e3496b17f61e
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f
    (cherry picked from commit 01c359665b53ef51837bd2bde649f969e0223d7f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909085
Committed: https://opendev.org/openstack/kolla-ansible/commit/2d8a5c588aacb29b4d0c39a45acbe2c604637ca1
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 2d8a5c588aacb29b4d0c39a45acbe2c604637ca1
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f
    (cherry picked from commit 01c359665b53ef51837bd2bde649f969e0223d7f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/908405
Committed: https://opendev.org/openstack/kolla-ansible/commit/1ef765f69044abae1b838b261481f420a4007121
Submitter: "Zuul (22348)"
Branch: master

commit 1ef765f69044abae1b838b261481f420a4007121
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909542

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909543

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909544

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909545

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909544
Committed: https://opendev.org/openstack/kolla-ansible/commit/47d97bda7e23a94ab8706acfd0887567d91af237
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 47d97bda7e23a94ab8706acfd0887567d91af237
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909542
Committed: https://opendev.org/openstack/kolla-ansible/commit/847bcd4364a7765295f73ebc46b81d84f8fdfd6a
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 847bcd4364a7765295f73ebc46b81d84f8fdfd6a
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909543
Committed: https://opendev.org/openstack/kolla-ansible/commit/ffcf271f55135e34cf1e7892f48a8bfc617334bf
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ffcf271f55135e34cf1e7892f48a8bfc617334bf
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909545
Committed: https://opendev.org/openstack/kolla-ansible/commit/ee1861fe0080c61092b1fdc3e15c36bf77f9429d
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit ee1861fe0080c61092b1fdc3e15c36bf77f9429d
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 16.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.6.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912066

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sven Kieske <email address hidden>" on branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912066
Reason: superseded by https://review.opendev.org/c/openstack/kolla-ansible/+/912452