"Authorization Failure" on Swift when running Pig job (vanilla Hadoop 2.3.0, keystone v3)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Sahara |
Fix Released
|
Critical
|
Dmitry Mescheryakov |
Bug Description
All the steps described below are performed with username admin, password empty, tenant/project demo (variable exported)
The issue has been reproduced with a fresh devstack installation from milestone-proposed.
Configure a pool with Vanilla Hadoop 2.3.0 (even if it should not depend on the Hadoop version). One master and one worker is enough.
Take the pig job from sahara-extra repository. Upload input, example.pig, udf.jar to mytest swift container.
$ swift upload mytest input
(etc)
Define a data source myinput for "input" file and a myoutput for the output file, use admin/empty as credentials.
Define two job binaries for the two executable, and then a job-template using them.
Submit the defined job-template using myinput and myoutput as input/output data.
The job will stay in the "Pending" state.
The sahara logs contains a stacktrace:
2014-04-09 11:22:06.391 DEBUG sahara.
2014-04-09 11:22:07.701 DEBUG keystoneclient.
2014-04-09 11:22:07.703 INFO urllib3.
2014-04-09 11:22:07.707 DEBUG urllib3.
2014-04-09 11:22:07.708 DEBUG keystoneclient.
RESP BODY: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
from (pid=22143) _send_request /opt/stack/
2014-04-09 11:22:07.708 DEBUG keystoneclient.
2014-04-09 11:22:07.709 ERROR swiftclient [-] Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)
2014-04-09 11:22:07.709 TRACE swiftclient Traceback (most recent call last):
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/
2014-04-09 11:22:07.709 TRACE swiftclient self.url, self.token = self.get_auth()
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/
2014-04-09 11:22:07.709 TRACE swiftclient insecure=
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/
2014-04-09 11:22:07.709 TRACE swiftclient insecure=insecure)
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/
2014-04-09 11:22:07.709 TRACE swiftclient raise ClientException
2014-04-09 11:22:07.709 TRACE swiftclient ClientException: Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)
2014-04-09 11:22:07.709 TRACE swiftclient
2014-04-09 11:22:07.913 ERROR sahara.context [-] Thread 'Starting Job Execution 5ba25bf0-
2014-04-09 11:22:07.913 TRACE sahara.context Traceback (most recent call last):
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/
2014-04-09 11:22:07.913 TRACE sahara.context func(*args, **kwargs)
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/
2014-04-09 11:22:07.913 TRACE sahara.context upload_
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/
2014-04-09 11:22:07.913 TRACE sahara.context raw_data = dispatch.
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/
2014-04-09 11:22:07.913 TRACE sahara.context res = i_swift.
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/
2014-04-09 11:22:07.913 TRACE sahara.context raise ex.SwiftClientE
2014-04-09 11:22:07.913 TRACE sahara.context SwiftClientExce
I have reports that it works when v2 Keystone API is used.
Changed in sahara: | |
milestone: | none → juno-1 |
Changed in sahara: | |
milestone: | juno-1 → icehouse-rc2 |
Changed in sahara: | |
assignee: | nobody → Dmitry Mescheryakov (dmitrymex) |
importance: | Undecided → Critical |
status: | New → In Progress |
Changed in sahara: | |
milestone: | icehouse-rc2 → 2014.1 |
I also experienced this error with a different system configuration. I am testing RDO/Icehouse with a custom Sahara/Horizon install. I get the same exception trace when trying to start a job against a cluster using the hdp2 plugin. When I changed use_identity_ api_v3= false in sahara.conf, it started working.
According to the keystone v3 api docs, the endpoint "/v3/tokens" does not exist. It has been replaced with "/v3/auth/tokens" which appears to have a different format for the data it accepts on a POST. I was able to confirm the 404 error on the "/v3/tokens" endpoint using httpie. I was unable to get the proper format for "/v3/auth/tokens".
My feeling is that Sahara is creating the url for "/v3/tokens" but I could not find the code that produces this request. There were some hints that the url was being created by appending "/tokens" to a version number for the auth.