[aodh.notifier] Not setting user_domain_id raises keystone error: The resource could not be found.

Bug #1831181 reported by Jorge Niedbalski on 2019-05-31
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
New
Undecided
Jorge Niedbalski
OpenStack AODH Charm
Undecided
Jorge Niedbalski
Ubuntu Cloud Archive
Status tracked in Stein
Mitaka
Medium
Unassigned
Ocata
Medium
Unassigned
Queens
Medium
Unassigned
Rocky
Medium
Unassigned
Stein
Medium
Unassigned
Train
Medium
Unassigned
charm-interface-keystone
Undecided
Jorge Niedbalski
aodh (Ubuntu)
Status tracked in Eoan
Bionic
Medium
Unassigned
Disco
Medium
Unassigned
Eoan
Medium
Unassigned

Bug Description

[Environment]

Xenial/Queens

[Description]

If no user_domain_id is provided then, the following exception will be raised by the
aodh-notifier component:

$ openstack alarm-history show 1424b6fb-1d9d-4d69-b00a-ad455aad0ecb

==> /var/log/aodh/aodh-notifier.log <==
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self._plugin.get_auth_ref(session, **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier authenticated=False, log=False)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self.request(url, 'POST', **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier raise exceptions.from_response(resp, method, url)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier NotFound: (http://10.5.0.36:35357/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-50d54903-d2af-4fd1-be68-a92498d91b85)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier

Domain configuration for the aodh service is as follows:

root@juju-95497c-controller-1:/home/ubuntu# grep domain /etc/aodh/aodh.conf
gnocchi_external_domain_name = service_domain
project_domain_name = service_domain
user_domain_name = service_domain

[Test Case]

1) Configure aodh service with the following configuration

root@juju-95497c-controller-1:/home/ubuntu# grep domain /etc/aodh/aodh.conf
gnocchi_external_domain_name = service_domain
project_domain_name = service_domain
user_domain_name = service_domain

2) Check the alarm history

ubuntu@niedbalski-bastion:~/cases/229352$ openstack alarm-history show 1424b6fb-1d9d-4d69-b00a-ad455aad0ecb

==> /var/log/aodh/aodh-notifier.log <==
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self._plugin.get_auth_ref(session, **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier authenticated=False, log=False)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self.request(url, 'POST', **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier raise exceptions.from_response(resp, method, url)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier NotFound: (http://10.5.0.36:35357/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-50d54903-d2af-4fd1-be68-a92498d91b85)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier

*** With the patch applied the latest traceback doesn't gets generated.

[Regression Potential]

 * None has been identified

### Explanation

The following line https://github.com/openstack/aodh/blob/master/aodh/keystone_client.py#L51
enforces the existence of the user_domain_id for getting the authenticated keystone session.

[Proposed Solution]

* Use user_domain_name as a fallback if no user_domain_id is provided.
* The charm templates should provide only user_domain_id if provided orelse
fallback to user_domain_name.

Changed in aodh:
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in charm-aodh:
assignee: nobody → Jorge Niedbalski (niedbalski)
status: New → In Progress
Changed in aodh (Ubuntu):
status: New → In Progress
Changed in charm-interface-keystone:
status: New → In Progress
assignee: nobody → Jorge Niedbalski (niedbalski)
Jorge Niedbalski (niedbalski) wrote :
Jorge Niedbalski (niedbalski) wrote :

Required charm-keystone-layer for review https://review.opendev.org/662582

Changed in aodh (Ubuntu Disco):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu Cosmic):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu Bionic):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu Xenial):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu Eoan):
assignee: Jorge Niedbalski (niedbalski) → nobody
status: In Progress → New

Reviewed: https://review.opendev.org/662582
Committed: https://git.openstack.org/cgit/openstack/charm-interface-keystone/commit/?id=5bc07627efedd6b3afa4a860a6c9f01d58251849
Submitter: Zuul
Branch: master

commit 5bc07627efedd6b3afa4a860a6c9f01d58251849
Author: Jorge Niedbalski <email address hidden>
Date: Fri May 31 17:30:26 2019 -0400

    Add service_domain_id to the auto_accessors list.

    service_domain_id might be required by dependant charms
    such as aodh and should be exposed as an accessor attribute.

    Partial-Bug: #1831181

    Change-Id: Idae7aec048b9203d2b1a75da8d68cf97271f1b53
    Signed-off-by: Jorge Niedbalski <email address hidden>

Changed in charm-interface-keystone:
status: In Progress → Fix Released
Changed in aodh (Ubuntu Eoan):
importance: Undecided → High
status: New → Triaged
importance: High → Medium
Changed in aodh (Ubuntu Disco):
importance: Undecided → Medium
status: New → Triaged
Changed in aodh (Ubuntu Cosmic):
importance: Undecided → Medium
status: New → Triaged
Changed in aodh (Ubuntu Bionic):
importance: Undecided → Medium
status: New → Triaged
Changed in aodh (Ubuntu Xenial):
importance: Undecided → Medium
status: New → Triaged
Corey Bryant (corey.bryant) wrote :

This bug should be "Fix Released" for upstream aodh master, stable/stein, stable/rocky, and stable/queens according to the patches that have been merged. See comment #5 at https://bugs.launchpad.net/cloud-archive/mitaka/+bug/1831181/comments/5.

Reviewed: https://review.opendev.org/663133
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=58b8feeece1c05b1fc71097cac932b92ec3bf00a
Submitter: Zuul
Branch: master

commit 58b8feeece1c05b1fc71097cac932b92ec3bf00a
Author: Jorge Niedbalski <email address hidden>
Date: Tue Jun 4 15:58:15 2019 -0400

    Add user_domain_id to service_credentials.

    Keystone authentication might use user_domain_id
    as a fallback if no user_domain_name is provided.

    Change-Id: I96008794a2f8cce9ee291b2e1d967deb281d988e
    Closes-Bug: #1831181
    Signed-off-by: Jorge Niedbalski <email address hidden>

Changed in charm-aodh:
status: In Progress → Fix Committed
Changed in aodh (Ubuntu Xenial):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Bionic):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Cosmic):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Disco):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Eoan):
status: Triaged → Fix Committed
Changed in aodh (Ubuntu Disco):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in aodh (Ubuntu Cosmic):
assignee: nobody → Jorge Niedbalski (niedbalski)
no longer affects: aodh (Ubuntu Xenial)
Jorge Niedbalski (niedbalski) wrote :
Changed in aodh (Ubuntu Bionic):
assignee: nobody → Jorge Niedbalski (niedbalski)
Jorge Niedbalski (niedbalski) wrote :
Jorge Niedbalski (niedbalski) wrote :

The attachment "Disco SRU debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
tags: added: sts-sru-needed
Corey Bryant (corey.bryant) wrote :

Cosmic is EOL so let's just fix this in Rocky.

Changed in aodh (Ubuntu Cosmic):
status: Triaged → Won't Fix
Łukasz Zemczak (sil2100) wrote :

What is the test case for validation here? I know one could potentially figure something out from the description of the problem, but we'd prefer a straightforward test-case provided as part of the SRU information. Regression potential could be good as well, although seeing the change I guess there's not much that can regress because of that.

Changed in aodh (Ubuntu Bionic):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Disco):
assignee: Jorge Niedbalski (niedbalski) → nobody
Changed in aodh (Ubuntu Cosmic):
assignee: Jorge Niedbalski (niedbalski) → nobody
no longer affects: aodh (Ubuntu Cosmic)
Jorge Niedbalski (niedbalski) wrote :

@sil2100

updated SRU template as requested.

description: updated

Hello Jorge, or anyone else affected,

Accepted aodh into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/aodh/8.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in aodh (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-disco
Jorge Niedbalski (niedbalski) wrote :

With the patch applied, the problem isn't longer reproduced: https://pastebin.canonical.com/p/sPT82rwBCh/ , marking verification as completed.

tags: added: verification-done verification-done-disco
removed: verification-needed verification-needed-disco
Jorge Niedbalski (niedbalski) wrote :

Please @SRU, check the bionic debdiff that's also attached into this case.

Thank you.

Dan Streetman (ddstreet) on 2019-08-06
tags: added: sts-sponsor
Dan Streetman (ddstreet) on 2019-08-06
tags: removed: sts-sponsor
James Page (james-page) on 2019-08-07
Changed in charm-aodh:
milestone: none → 19.07
James Page (james-page) wrote :

Hello Jorge, or anyone else affected,

Accepted aodh into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed

======
Totals
======
Ran: 92 tests in 1103.8351 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 544.0273 sec.

tags: added: verification-stein-done
removed: verification-stein-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package aodh - 8.0.0-0ubuntu1.1

---------------
aodh (8.0.0-0ubuntu1.1) disco; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/stein branch.

  [ Sahid Orentino Ferdjaoui ]
  * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
    client might require the user domain name. (LP: #1831181).

 -- Corey Bryant <email address hidden> Tue, 07 May 2019 13:45:39 -0400

Changed in aodh (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Jorge Niedbalski (niedbalski) wrote :

@Corey, @Sahid.

Any chance to check the bionic debdiff attached? plus R/Q backports in UCA?

Thank you

David Ames (thedac) on 2019-08-12
Changed in charm-aodh:
status: Fix Committed → Fix Released
Corey Bryant (corey.bryant) wrote :

Thanks Jorge. I've uploaded new versions of aodh with your patches to the bionic unapproved queue [1] and rocky-staging.

[1] https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=aodh

Hello Jorge, or anyone else affected,

Accepted aodh into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/aodh/6.0.1-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in aodh (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
Corey Bryant (corey.bryant) wrote :

Hello Jorge, or anyone else affected,

Accepted aodh into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-rocky-needed

Change abandoned by Alex Kavanagh (tinwood) (<email address hidden>) on branch: stable/19.04
Review: https://review.opendev.org/664660
Reason: dropping as in 19.07 and backport is not necessary.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers