All the steps described below are performed with username admin, password empty, tenant/project demo (variable exported)
The issue has been reproduced with a fresh devstack installation from milestone-proposed.
Configure a pool with Vanilla Hadoop 2.3.0 (even if it should not depend on the Hadoop version). One master and one worker is enough.
Take the pig job from sahara-extra repository. Upload input, example.pig, udf.jar to mytest swift container.
$ swift upload mytest input
(etc)
Define a data source myinput for "input" file and a myoutput for the output file, use admin/empty as credentials.
Define two job binaries for the two executable, and then a job-template using them.
Submit the defined job-template using myinput and myoutput as input/output data.
The job will stay in the "Pending" state.
The sahara logs contains a stacktrace:
2014-04-09 11:22:06.391 DEBUG sahara.utils.ssh_remote [-] [cluster-instance-1397032308-master-001] _execute_command took 3.4 seconds to complete from (pid=22143) _log_command /opt/stack/sahara/sahara/utils/ssh_remote.py:407
2014-04-09 11:22:07.701 DEBUG keystoneclient.session [-] REQ: curl -i -X POST http://10.34.1.187:5000/v3/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "service", "passwordCredentials": {"username": "admin", "password": "empty"}}}' from (pid=22143) request /opt/stack/python-keystoneclient/keystoneclient/session.py:245
2014-04-09 11:22:07.703 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.34.1.187
2014-04-09 11:22:07.707 DEBUG urllib3.connectionpool [-] "POST /v3/tokens HTTP/1.1" 404 93 from (pid=22143) _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:330
2014-04-09 11:22:07.708 DEBUG keystoneclient.session [-] RESP: [404] CaseInsensitiveDict({'date': 'Wed, 09 Apr 2014 09:22:07 GMT', 'vary': 'X-Auth-Token', 'content-length': '93', 'content-type': 'application/json'})
RESP BODY: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
from (pid=22143) _send_request /opt/stack/python-keystoneclient/keystoneclient/session.py:285
2014-04-09 11:22:07.708 DEBUG keystoneclient.session [-] Request returned failure status: 404 from (pid=22143) request /opt/stack/python-keystoneclient/keystoneclient/session.py:261
2014-04-09 11:22:07.709 ERROR swiftclient [-] Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)
2014-04-09 11:22:07.709 TRACE swiftclient Traceback (most recent call last):
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1192, in _retry
2014-04-09 11:22:07.709 TRACE swiftclient self.url, self.token = self.get_auth()
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1166, in get_auth
2014-04-09 11:22:07.709 TRACE swiftclient insecure=self.insecure)
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/python-swiftclient/swiftclient/client.py", line 364, in get_auth
2014-04-09 11:22:07.709 TRACE swiftclient insecure=insecure)
2014-04-09 11:22:07.709 TRACE swiftclient File "/opt/stack/python-swiftclient/swiftclient/client.py", line 301, in get_keystoneclient_2_0
2014-04-09 11:22:07.709 TRACE swiftclient raise ClientException('Authorization Failure. %s' % err)
2014-04-09 11:22:07.709 TRACE swiftclient ClientException: Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)
2014-04-09 11:22:07.709 TRACE swiftclient
2014-04-09 11:22:07.913 ERROR sahara.context [-] Thread 'Starting Job Execution 5ba25bf0-cbd8-48fb-bcf6-f16820a892f5' fails with exception: 'Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)'
2014-04-09 11:22:07.913 TRACE sahara.context Traceback (most recent call last):
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/sahara/sahara/context.py", line 124, in _wrapper
2014-04-09 11:22:07.913 TRACE sahara.context func(*args, **kwargs)
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/sahara/sahara/service/edp/job_manager.py", line 144, in run_job
2014-04-09 11:22:07.913 TRACE sahara.context upload_job_files(oozie_server, wf_dir, job, hdfs_user)
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/sahara/sahara/service/edp/job_manager.py", line 186, in upload_job_files
2014-04-09 11:22:07.913 TRACE sahara.context raw_data = dispatch.get_raw_binary(main)
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/sahara/sahara/service/edp/binary_retrievers/dispatch.py", line 30, in get_raw_binary
2014-04-09 11:22:07.913 TRACE sahara.context res = i_swift.get_raw_data(context.ctx(), job_binary)
2014-04-09 11:22:07.913 TRACE sahara.context File "/opt/stack/sahara/sahara/service/edp/binary_retrievers/internal_swift.py", line 88, in get_raw_data
2014-04-09 11:22:07.913 TRACE sahara.context raise ex.SwiftClientException(e.message)
2014-04-09 11:22:07.913 TRACE sahara.context SwiftClientException: Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)
I have reports that it works when v2 Keystone API is used.
I also experienced this error with a different system configuration. I am testing RDO/Icehouse with a custom Sahara/Horizon install. I get the same exception trace when trying to start a job against a cluster using the hdp2 plugin. When I changed use_identity_
According to the keystone v3 api docs, the endpoint "/v3/tokens" does not exist. It has been replaced with "/v3/auth/tokens" which appears to have a different format for the data it accepts on a POST. I was able to confirm the 404 error on the "/v3/tokens" endpoint using httpie. I was unable to get the proper format for "/v3/auth/tokens".
My feeling is that Sahara is creating the url for "/v3/tokens" but I could not find the code that produces this request. There were some hints that the url was being created by appending "/tokens" to a version number for the auth.