authentication failed when getting trusts

[Yanyan Hu]

Unauthorized error happened when senlin trust middleware tries to invoke keystone_v3 driver to get trusts. The exception dump is as followed.

How to reproduce:
1. clean senlin DB(or just clean credential table);
2. restart senlin-engine and senlin-api
3. perform any senlin cmd like 'senlin cluster-list'

JSON response : {"code": 500, "error": {"code": 500, "message": "Invalid username or password (Disable debug mode to suppress these details.) (HTTP 401) (Request-ID: req-339a564a-48c7-42d8-8c4d-f7ca785f9288)", "traceback": "Traceback (most recent call last):\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/api/middleware/fault.py\", line 143, in process_request\n return req.get_response(self.application)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/oslo_middleware/base.py\", line 114, in __call__\n response = req.get_response(self.application)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/common/wsgi.py\", line 501, in __call__\n response = request.get_response(self.application)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/common/wsgi.py\", line 501, in __call__\n response = request.get_response(self.application)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py\", line 434, in __call__\n response = req.get_response(self._app)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/common/wsgi.py\", line 501, in __call__\n response = request.get_response(self.application)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1317, in send\n application, catch_exc_info=False)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/request.py\", line 1281, in call_application\n app_iter = application(self.environ, start_response)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/home/stack/huyanyan/python/virtualenv/SENLINENV/lib/python2.7/site-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/common/wsgi.py\", line 498, in __call__\n response = self.process_request(request)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/api/middleware/trust.py\", line 80, in process_request\n trust_id = self._get_trust(req.context)\n File \"/home/stack/huyanyan/openstack_repo/openstack/senlin/senlin/api/middleware/trust.py\", line 60, in _get_trust\n raise ex\nInternalError: Invalid username or password (Disable debug mode to suppress these details.) (HTTP 401) (Request-ID: req-339a564a-48c7-42d8-8c4d-f7ca785f9288)\n", "type": "InternalError"}, "explanation": "The server has either erred or is incapable of performing the requested operation.", "title": "Internal Server Error"}

Error log in keystone is like this:
2015-11-27 03:31:41.648480 21825 WARNING keystone.common.wsgi [req-f6fd23b2-0236-4107-a519-c2df4874b014 - - - - -] Authorization failed. Invalid username or password (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 127.0.0.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/250657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250657
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=11ccdcc81df33308d52d1f660d91ba5e3176e3b1
Submitter: Jenkins
Branch: master

commit 11ccdcc81df33308d52d1f660d91ba5e3176e3b1
Author: yanyanhu <email address hidden>
Date: Fri Nov 27 02:09:36 2015 -0500

    Set default auth_plugin to token when creating SDK connection

    This patch revises default auth_plugin type to token when creating
    SDK connection. This is because Senlin service uses token and trust
    to talk with other backend openstack services. For authenticate
    method in sdk driver, auth_plugin type will be decided based on
    credential given. If token is not provided, auth_plugin type will be
    set to password.

    Change-Id: Ief88395cf7f663a501e762028566d24ffde8dfcd
    Closes-Bug: #1520440

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/senlin 1.0.0.0b2 development milestone.