senlin

`trust` middleware unable to fetch trusts on `bobcat` release

Bug #2048452 reported by Pham Le Gia Dai on 2024-01-07

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	senlin	Fix Released	Undecided	Unassigned

Bug Description

Here is the logs on senlin side:
2024-01-07 13:51:16.833 4179100 INFO senlin.api [-] Starting Senlin API on 0.0.0.0:8778
2024-01-07 13:51:16.835 4179100 INFO senlin.api.common.wsgi [-] Starting 1 workers
2024-01-07 13:51:16.842 4179100 INFO senlin.api.common.wsgi [-] Started child 4179110
2024-01-07 13:51:17.564 4179110 INFO eventlet.wsgi.server [-] (4179110) wsgi starting up on http://0.0.0.0:8778
2024-01-07 13:51:17.844 4179110 DEBUG eventlet.wsgi.server [-] (4179110) accepted ('127.0.0.1', 58194) server /home/daiplg/senlin/.tox/venv/lib/python3.8/site-packages/eventlet/wsgi.py:1004
2024-01-07 13:51:17.850 4179110 DEBUG senlin.api.middleware.version_negotiation [-] Processing request: GET /v1/clusters Accept: */* process_request /home/daiplg/senlin/senlin/api/middleware/version_negotiation.py:48
2024-01-07 13:51:17.852 4179110 DEBUG senlin.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /home/daiplg/senlin/senlin/api/middleware/version_negotiation.py:66
2024-01-07 13:51:31.619 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering container.dockerinc.docker-1.0 -> <class 'senlin.profiles.container.docker.DockerProfile'>
2024-01-07 13:51:31.620 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering os.heat.stack-1.0 -> <class 'senlin.profiles.os.heat.stack.StackProfile'>
2024-01-07 13:51:31.621 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering os.nova.server-1.0 -> <class 'senlin.profiles.os.nova.server.ServerProfile'>
2024-01-07 13:51:31.629 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.affinity-1.0 -> <class 'senlin.policies.affinity_policy.AffinityPolicy'>
2024-01-07 13:51:31.629 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.batch-1.0 -> <class 'senlin.policies.batch_policy.BatchPolicy'>
2024-01-07 13:51:31.630 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.deletion-1.0 -> <class 'senlin.policies.deletion_policy.DeletionPolicy'>
2024-01-07 13:51:31.630 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.deletion-1.1 -> <class 'senlin.policies.deletion_policy.DeletionPolicy'>
2024-01-07 13:51:31.630 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.health-1.0 -> <class 'senlin.policies.health_policy.HealthPolicy'>
2024-01-07 13:51:31.631 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.health-1.1 -> <class 'senlin.policies.health_policy.HealthPolicy'>
2024-01-07 13:51:31.631 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.health-1.2 -> <class 'senlin.policies.health_policy.HealthPolicy'>
2024-01-07 13:51:31.631 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.loadbalance-1.0 -> <class 'senlin.policies.lb_policy.LoadBalancingPolicy'>
2024-01-07 13:51:31.632 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.loadbalance-1.1 -> <class 'senlin.policies.lb_policy.LoadBalancingPolicy'>
2024-01-07 13:51:31.632 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.loadbalance-1.2 -> <class 'senlin.policies.lb_policy.LoadBalancingPolicy'>
2024-01-07 13:51:31.632 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.loadbalance-1.3 -> <class 'senlin.policies.lb_policy.LoadBalancingPolicy'>
2024-01-07 13:51:31.633 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.region_placement-1.0 -> <class 'senlin.policies.region_placement.RegionPlacementPolicy'>
2024-01-07 13:51:31.633 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.scaling-1.0 -> <class 'senlin.policies.scaling_policy.ScalingPolicy'>
2024-01-07 13:51:31.633 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering senlin.policy.zone_placement-1.0 -> <class 'senlin.policies.zone_placement.ZonePlacementPolicy'>
2024-01-07 13:51:31.647 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering openstack -> <module 'senlin.drivers.os' from '/home/daiplg/senlin/senlin/drivers/os/__init__.py'>
2024-01-07 13:51:31.648 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering openstack_test -> <module 'senlin.tests.drivers.os_test' from '/home/daiplg/senlin/senlin/tests/drivers/os_test/__init__.py'>
2024-01-07 13:51:31.648 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering heat -> <class 'senlin.engine.notifications.heat_endpoint.HeatNotificationEndpoint'>
2024-01-07 13:51:31.649 4179110 INFO senlin.engine.registry [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Registering nova -> <class 'senlin.engine.notifications.nova_endpoint.NovaNotificationEndpoint'>
2024-01-07 13:51:32.206 4179110 ERROR senlin.drivers.sdk [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] Internal Server Error (HTTP 500): keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
2024-01-07 13:51:32.208 4179110 DEBUG senlin.api.common.serializers [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] JSON response : {"code": 500, "error": {"code": 500, "message": "Internal Server Error (HTTP 500)", "type": "InternalError"}, "explanation": "The server has either erred or is incapable of performing the requested operation.", "title": "Internal Server Error"} to_json /home/daiplg/senlin/senlin/api/common/serializers.py:86
2024-01-07 13:51:32.210 4179110 INFO eventlet.wsgi.server [None req-441ef3c3-3900-453c-9b43-d03c6db93137 38f29bf102c187d055d965841e242721fbd82b8ea6a29b0b581b305a6ec3262f 49815584e6c84aa39a86ee321586e934 - - - -] 127.0.0.1 - - [07/Jan/2024 13:51:32] "GET /v1/clusters HTTP/1.1" 500 438 14.363658

This is the logs on the keystone side: https://launchpadlibrarian.net/583330085/keystone-log.txt

The error seems to be happening here: https://opendev.org/openstack/senlin/src/branch/master/senlin/api/middleware/trust.py#L54

And this also related to this bug: https://bugs.launchpad.net/keystone/+bug/1959674

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-08: Fix proposed to senlin (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/senlin/+/904947

Changed in senlin:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-18: Fix merged to senlin (master)

Reviewed: https://review.opendev.org/c/openstack/senlin/+/904947
Committed: https://opendev.org/openstack/senlin/commit/2d6f537b3843a1ac884ef711fd764d5e3586d054
Submitter: "Zuul (22348)"
Branch: master

commit 2d6f537b3843a1ac884ef711fd764d5e3586d054
Author: Pham Le Gia Dai <email address hidden>
Date: Mon Jan 8 16:32:49 2024 +0700

Fix: `TrustMiddleware` unable to fetch trusts/credentials from identity service

    When using token to init a new openstacksdk connection, the SDK try to fetch
    another token from it, causing keystone raise exception and return 500 status
    response. Refer: https://bugs.launchpad.net/keystone/+bug/1959674
    Switch from `token` to `admin_token` to make the SDK session use the provided
    token directly instead of fetching a new one.

    Closes-bug: #2048452
    Depends-On: https://review.opendev.org/c/openstack/senlin/+/905555
    Change-Id: I8f9b2db3d4851cf54c2113b2fb0ae97ae38ac286

Changed in senlin:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.