heat ERROR: Authentication failed: Authentication required

[Eric K]

With standard devstack instance, all the datasources work fine, but after a while the Heat datasource begins to show persistent error: ERROR: Authentication failed: Authentication required.

Left alone for enough time, the error appears to be reproduced every time.

Could be a heat issue because it doesn't happen with any of the other standard openstack services.

Here's the full log:
https://drive.google.com/file/d/0B7XlvjMOO4wZTERqMEdESmNPNVk/view?usp=sharing

The first error appears at 2016-03-18 16:55:17.516 and appears to continue steadily after that on (almost?) every poll.

[Eric K (ekcs)]

Note: I did a quick search for heat authentication issues, but none of the reported cases I found fits this profile: starts out working and then stops at some point.

[Eric K (ekcs)]

Additional info: once the error occurs, if I restart congress (leaving everything else untouched), the error goes away, but seems to eventually creeps up again.

[Eric K (ekcs)]

Confirmed that the issue doesn't seem to affect heat CLI.
While Congress is encountering authentication errors accessing heat, I could use the heat CLI to access heat information without issue. (e.g., $ heat config-list).

Also, here are the corresponding logs between Congress showing error authenticating and h-api showing failed authentication token.

Congress:
2016-03-18 16:55:17.516 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2016-03-18 16:55:17.516 TRACE congress.datasources.datasource_driver HTTPUnauthorized: ERROR: Authentication failed: Authentication required
2016-03-18 16:55:27.546 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2016-03-18 16:55:27.546 TRACE congress.datasources.datasource_driver HTTPUnauthorized: ERROR: Authentication failed: Authentication required
2016-03-18 16:55:37.582 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2016-03-18 16:55:37.582 TRACE congress.datasources.datasource_driver HTTPUnauthorized: ERROR: Authentication failed: Authentication required

h-api:
2016-03-18 16:55:07.478 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:07.478 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: a8a3a526ca5f49a4b2572cc49a0f6c2f", "code": 404, "title": "Not Found"}}
2016-03-18 16:55:07.479 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:17.514 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:17.514 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: a8a3a526ca5f49a4b2572cc49a0f6c2f", "code": 404, "title": "Not Found"}}
2016-03-18 16:55:17.515 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:27.543 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:27.543 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: a8a3a526ca5f49a4b2572cc49a0f6c2f", "code": 404, "title": "Not Found"}}
2016-03-18 16:55:27.544 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:37.577 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-18 16:55:37.578 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Co...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/298394

[Eric K (ekcs)]

Cause identified: keystone token not refreshed after expiry. Some clients handle refresh (e.g., neutron, nova), but some don't (e.g., glance, ironic, murano, heat). For the clients that do not handle refresh, the datasource driver needs to handle the refresh. The workaround has been implemented in other drivers, but not in heat.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/299950

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tim Hinrichs (<email address hidden>) on branch: master
Review: https://review.openstack.org/298394
Reason: Subsumed by another fix using Keystone sessions

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299121
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=4d489150710f7a2ee4cc3f6f838540594c7b58dd
Submitter: Jenkins
Branch: master

commit 4d489150710f7a2ee4cc3f6f838540594c7b58dd
Author: Anusha Ramineni <email address hidden>
Date: Wed Mar 30 10:09:13 2016 +0530

    Use keystone sessions to authenticate

    Tried out keystone session based authentication to solve our current
    issue with token expiry. Just an alternate solution to

    https://review.openstack.org/#/c/298960/1
    https://review.openstack.org/#/c/298394/

    Partial-Bug:#1563677
    Partial-Bug:#1564115
    Partial-Bug:#1563495
    Closes-Bug:#1559362
    Change-Id: I8a8a4fe5547b4aaa8a4735efd79857750e555578

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299950
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=9248b6fa81b3d0413b084314ae208e59fcb9957d
Submitter: Jenkins
Branch: stable/mitaka

commit 9248b6fa81b3d0413b084314ae208e59fcb9957d
Author: Anusha Ramineni <email address hidden>
Date: Wed Mar 30 10:09:13 2016 +0530

    Use keystone sessions to authenticate

    Tried out keystone session based authentication to solve our current
    issue with token expiry. Just an alternate solution to

    https://review.openstack.org/#/c/298960/1
    https://review.openstack.org/#/c/298394/

    Partial-Bug:#1563677
    Partial-Bug:#1564115
    Partial-Bug:#1563495
    Closes-Bug:#1559362
    Change-Id: I8a8a4fe5547b4aaa8a4735efd79857750e555578

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/congress 3.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302558

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302558
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=6b1ce68b1a9f8cb0a6839497cf15064f36037179
Submitter: Jenkins
Branch: master

commit 33bbaccf8e10b77b1374b6eed1eb3b4b90c09f00
Author: Eric K <email address hidden>
Date: Wed Mar 30 16:58:06 2016 -0700

    Use keystone session in murano_driver

    Partial-Bug: #1564115
    Partial-Bug: #1563495
    Partial-Bug: #1563677

    Change-Id: If6223188f182063319e299a9226afd8a137051b9

commit 9248b6fa81b3d0413b084314ae208e59fcb9957d
Author: Anusha Ramineni <email address hidden>
Date: Wed Mar 30 10:09:13 2016 +0530

    Use keystone sessions to authenticate

    Tried out keystone session based authentication to solve our current
    issue with token expiry. Just an alternate solution to

    https://review.openstack.org/#/c/298960/1
    https://review.openstack.org/#/c/298394/

    Partial-Bug:#1563677
    Partial-Bug:#1564115
    Partial-Bug:#1563495
    Closes-Bug:#1559362
    Change-Id: I8a8a4fe5547b4aaa8a4735efd79857750e555578

commit d554770b6d5fb916ec9c9bcc39c833a0773a1825
Author: Eric K <email address hidden>
Date: Wed Mar 23 15:34:16 2016 -0700

    Update standalone setup instructions

    1. Update "Standalone. Get Congress running all by itself."
       to reflect that the instructions are for installing alongside an
       existing OpenStack deployment. We may consider creating true
       standalone-install instructions in the future.
    2. Update deprecated openstack command
       $ openstack service create congress --type "policy"
    3. Update | awk "/ congress / { print \$2 }" to grab service id
    4. Start Congress before configuring datasource drivers to avoid
       error:
    Unable to establish connection to http://x.x.x.x:1789/v1/data-sources
    5. Added bare congress.conf

    Change-Id: I69f100f778f93f48a1b393a315ce8a0808ef771f

commit e20bcc4bc178c4dd68e349ab38d4b94f2ad22ddb
Author: Thomas Goirand <email address hidden>
Date: Fri Mar 25 15:57:17 2016 +0100

    Also package congress_tempest_tests

    The folder congress_tempest_tests has to be packaged in order for the
    Debian packaging CI to work. Otherwise, we get:
    ImportError: No module named congress_tempest_tests.plugin
    when running templest.

    Please allow this distribution specific patch to reach upstream.

    Change-Id: If14b5afec201068af7cc6fc903cdf6fa72887c2a

commit cf8c8b5953ad3ca817e08f472ae544cf0114ad3c
Author: Eric K <email address hidden>
Date: Mon Mar 21 21:40:23 2016 -0700

    Fix for broken antlr3 in stand-alone install

    Restored and updated the permanent symlinks and updated setup.cfg
    to facilitate proper standalone installation of antlr3 w congress.

    Change-Id: I7b603bcf3785374d5e83d0e5a98669dbd93d8cd9

commit 8ed5b8442aba782bc5b3f180702922f328eb61af
Author: Anusha Ramineni <email address hidden>
Date: Wed Mar 23 15:37:03 2016 +0530

    Build docs using tox

    This commit helps to build the docs using tox, tox -edocs
    It would be useful to check for any errors if any docs are
    changed.

    C...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/305613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/305613
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=2453c6f43e6054d61fcfa193f97fcbb12034ef1f
Submitter: Jenkins
Branch: stable/liberty

commit 2453c6f43e6054d61fcfa193f97fcbb12034ef1f
Author: Anusha Ramineni <email address hidden>
Date: Wed Mar 30 10:09:13 2016 +0530

    Use keystone sessions to authenticate

    Tried out keystone session based authentication to solve our current
    issue with token expiry. Just an alternate solution to

    https://review.openstack.org/#/c/298960/1
    https://review.openstack.org/#/c/298394/

    Partial-Bug:#1563677
    Partial-Bug:#1564115
    Partial-Bug:#1563495
    Closes-Bug:#1559362
    Change-Id: I8a8a4fe5547b4aaa8a4735efd79857750e555578

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/congress 4.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/congress 3.0.0 release.