[aodh.notifier] Not setting user_domain_id raises keystone error: The resource could not be found.

[Jorge Niedbalski]

[Environment]

Xenial/Queens

[Description]

If no user_domain_id is provided then, the following exception will be raised by the
aodh-notifier component:

$ openstack alarm-history show 1424b6fb-1d9d-4d69-b00a-ad455aad0ecb

==> /var/log/aodh/aodh-notifier.log <==
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self._plugin.get_auth_ref(session, **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier authenticated=False, log=False)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self.request(url, 'POST', **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier raise exceptions.from_response(resp, method, url)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier NotFound: (http://10.5.0.36:35357/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-50d54903-d2af-4fd1-be68-a92498d91b85)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier

Domain configuration for the aodh service is as follows:

root@juju-95497c-controller-1:/home/ubuntu# grep domain /etc/aodh/aodh.conf
gnocchi_external_domain_name = service_domain
project_domain_name = service_domain
user_domain_name = service_domain

[Test Case]

1) Configure aodh service with the following configuration

root@juju-95497c-controller-1:/home/ubuntu# grep domain /etc/aodh/aodh.conf
gnocchi_external_domain_name = service_domain
project_domain_name = service_domain
user_domain_name = service_domain

2) Check the alarm history

ubuntu@niedbalski-bastion:~/cases/229352$ openstack alarm-history show 1424b6fb-1d9d-4d69-b00a-ad455aad0ecb

==> /var/log/aodh/aodh-notifier.log <==
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self._plugin.get_auth_ref(session, **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier authenticated=False, log=False)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier return self.request(url, 'POST', **kwargs)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier raise exceptions.from_response(resp, method, url)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier NotFound: (http://10.5.0.36:35357/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-50d54903-d2af-4fd1-be68-a92498d91b85)
2019-05-31 02:07:26.663 15274 ERROR aodh.notifier

*** With the patch applied the latest traceback doesn't gets generated.

[Regression Potential]

 * None has been identified

### Explanation

The following line https://github.com/openstack/aodh/blob/master/aodh/keystone_client.py#L51
enforces the existence of the user_domain_id for getting the authenticated keystone session.

[Proposed Solution]

* Use user_domain_name as a fallback if no user_domain_id is provided.
* The charm templates should provide only user_domain_id if provided orelse
fallback to user_domain_name.

[Jorge Niedbalski (niedbalski)]

Aodh fix for review https://review.opendev.org/662559

[Jorge Niedbalski (niedbalski)]

Required charm-keystone-layer for review https://review.opendev.org/662582

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662582
Committed: https://git.openstack.org/cgit/openstack/charm-interface-keystone/commit/?id=5bc07627efedd6b3afa4a860a6c9f01d58251849
Submitter: Zuul
Branch: master

commit 5bc07627efedd6b3afa4a860a6c9f01d58251849
Author: Jorge Niedbalski <email address hidden>
Date: Fri May 31 17:30:26 2019 -0400

    Add service_domain_id to the auto_accessors list.

    service_domain_id might be required by dependant charms
    such as aodh and should be exposed as an accessor attribute.

    Partial-Bug: #1831181

    Change-Id: Idae7aec048b9203d2b1a75da8d68cf97271f1b53
    Signed-off-by: Jorge Niedbalski <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/663133

[Jorge Niedbalski (niedbalski)]

Stein: https://review.opendev.org/662765
Queens: https://review.opendev.org/662766
Rocky: https://review.opendev.org/662765

Master: https://review.opendev.org/662559

[Corey Bryant (corey.bryant)]

This bug should be "Fix Released" for upstream aodh master, stable/stein, stable/rocky, and stable/queens according to the patches that have been merged. See comment #5 at https://bugs.launchpad.net/cloud-archive/mitaka/+bug/1831181/comments/5.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663133
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=58b8feeece1c05b1fc71097cac932b92ec3bf00a
Submitter: Zuul
Branch: master

commit 58b8feeece1c05b1fc71097cac932b92ec3bf00a
Author: Jorge Niedbalski <email address hidden>
Date: Tue Jun 4 15:58:15 2019 -0400

    Add user_domain_id to service_credentials.

    Keystone authentication might use user_domain_id
    as a fallback if no user_domain_name is provided.

    Change-Id: I96008794a2f8cce9ee291b2e1d967deb281d988e
    Closes-Bug: #1831181
    Signed-off-by: Jorge Niedbalski <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/19.04
Review: https://review.opendev.org/664660

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Disco SRU debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Cosmic is EOL so let's just fix this in Rocky.

[Łukasz Zemczak (sil2100)]

What is the test case for validation here? I know one could potentially figure something out from the description of the problem, but we'd prefer a straightforward test-case provided as part of the SRU information. Regression potential could be good as well, although seeing the change I guess there's not much that can regress because of that.

[Jorge Niedbalski (niedbalski)]

@sil2100

updated SRU template as requested.

[Brian Murray (brian-murray)]

Hello Jorge, or anyone else affected,

Accepted aodh into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/aodh/8.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jorge Niedbalski (niedbalski)]

With the patch applied, the problem isn't longer reproduced: https://pastebin.canonical.com/p/sPT82rwBCh/ , marking verification as completed.

[Jorge Niedbalski (niedbalski)]

Please @SRU, check the bionic debdiff that's also attached into this case.

Thank you.

[James Page (james-page)]

Hello Jorge, or anyone else affected,

Accepted aodh into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Sahid Orentino (sahid-ferdjaoui)]

======
Totals
======
Ran: 92 tests in 1103.8351 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 544.0273 sec.

[Launchpad Janitor (janitor)]

This bug was fixed in the package aodh - 8.0.0-0ubuntu1.1

---------------
aodh (8.0.0-0ubuntu1.1) disco; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/stein branch.

  [ Sahid Orentino Ferdjaoui ]
  * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
    client might require the user domain name. (LP: #1831181).

 -- Corey Bryant <email address hidden> Tue, 07 May 2019 13:45:39 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Jorge Niedbalski (niedbalski)]

@Corey, @Sahid.

Any chance to check the bionic debdiff attached? plus R/Q backports in UCA?

Thank you

[Corey Bryant (corey.bryant)]

Thanks Jorge. I've uploaded new versions of aodh with your patches to the bionic unapproved queue [1] and rocky-staging.

[1] https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=aodh

[Brian Murray (brian-murray)]

Hello Jorge, or anyone else affected,

Accepted aodh into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/aodh/6.0.1-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted aodh into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Kavanagh (tinwood) (<email address hidden>) on branch: stable/19.04
Review: https://review.opendev.org/664660
Reason: dropping as in 19.07 and backport is not necessary.

[Corey Bryant (corey.bryant)]

@Sahid/@Jorge,

If you have cycles can you help test this according to the steps in the [Test Case] section?

Thanks,
Corey

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted aodh into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jorge Niedbalski (niedbalski)]

Deployed on q/r/b the problem is no longer reproduced https://pastebin.canonical.com/p/hPV8SVv2Th/, marking the verification as completed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package aodh - 6.0.1-0ubuntu1.2

---------------
aodh (6.0.1-0ubuntu1.2) bionic; urgency=medium

  * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
    client might require the user domain name. (LP: #1831181).

 -- Jorge Niedbalski <email address hidden> Wed, 26 Jun 2019 19:09:47 -0400

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package aodh - 8.0.0-0ubuntu1.1~cloud0
---------------

 aodh (8.0.0-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 aodh (8.0.0-0ubuntu1.1) disco; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/stein branch.
 .
   [ Sahid Orentino Ferdjaoui ]
   * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
     client might require the user domain name. (LP: #1831181).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package aodh - 7.0.0-0ubuntu4~cloud1
---------------

 aodh (7.0.0-0ubuntu4~cloud1) bionic-rocky; urgency=medium
 .
   * d/gbp.conf: Create stable/rocky branch.
   * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
     client might require the user domain name. (LP: #1831181).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package aodh - 6.0.1-0ubuntu1.2~cloud0
---------------

 aodh (6.0.1-0ubuntu1.2~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 aodh (6.0.1-0ubuntu1.2) bionic; urgency=medium
 .
   * d/p/0001-Add-user_domain_name-to-keystone-client.patch: Keystone
     client might require the user domain name. (LP: #1831181).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/aodh 9.0.0.0rc1 release candidate.