Sahara

[EDP] Job with transient cluster is not running

Bug #1244286 reported by Vadim Rovachev on 2013-10-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Sahara	Invalid	Undecided	Unassigned

Bug Description

Preconditional steps:
By Example:
Savanna launched in 'admin' tenant
Created data sources and job binary in 'qa' tenant

Step 1.
Create nodegroup templates and cluster template in tenant 'qa'
Create job in tenant 'qa'

Step 2.
Launch job with transient cluster

Expected result:
Job SUCCEEDED

Actual result:
Job in state Pending

Savanna log:
---------------------------------------------------------------------
2013-10-24 14:04:44.031 21600 DEBUG keystoneclient.httpclient [-] REQ BODY: {"auth": {"tenantName": "admin", "passwordCredentials": {"username": "qa-user", "password": "swordfish
"}}}
request /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/keystoneclient/httpclient.py:105
RESP: [401] CaseInsensitiveDict({'date': 'Thu, 24 Oct 2013 14:04:43 GMT', 'transfer-encoding': 'chunked', 'content-type': 'application/json', 'vary': 'X-Auth-Token'})
RESP BODY: {"error": {"message": "Invalid project", "code": 401, "title": "Not Authorized"}}
2013-10-24 14:04:44.071 21600 DEBUG keystoneclient.httpclient [-] RESP: [401] CaseInsensitiveDict({'date': 'Thu, 24 Oct 2013 14:04:43 GMT', 'transfer-encoding': 'chunked', 'content-type': 'application/json', 'vary': 'X-Auth-Token'})
RESP BODY: {"error": {"message": "Invalid project", "code": 401, "title": "Not Authorized"}}
request /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/keystoneclient/httpclient.py:119
Request returned failure status: 401
2013-10-24 14:04:44.072 21600 DEBUG keystoneclient.httpclient [-] Request returned failure status: 401 request /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/keystoneclient/httpclient.py:123
2013-10-24 14:04:44.073 21600 DEBUG keystoneclient.v2_0.client [-] Authorization Failed. get_raw_token_from_identity_service /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:164
2013-10-24 14:04:44.073 21600 ERROR swiftclient [-] Unauthorised. Check username, password and tenant name/id
2013-10-24 14:04:44.073 21600 TRACE swiftclient Traceback (most recent call last):
2013-10-24 14:04:44.073 21600 TRACE swiftclient File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1103, in _retry
2013-10-24 14:04:44.073 21600 TRACE swiftclient self.url, self.token = self.get_auth()
2013-10-24 14:04:44.073 21600 TRACE swiftclient File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1079, in get_auth
2013-10-24 14:04:44.073 21600 TRACE swiftclient insecure=self.insecure)
2013-10-24 14:04:44.073 21600 TRACE swiftclient File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 302, in get_auth
2013-10-24 14:04:44.073 21600 TRACE swiftclient insecure=insecure)
2013-10-24 14:04:44.073 21600 TRACE swiftclient File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 237, in get_keystoneclient_2_0
2013-10-24 14:04:44.073 21600 TRACE swiftclient raise ClientException('Unauthorised. Check username, password'
2013-10-24 14:04:44.073 21600 TRACE swiftclient ClientException: Unauthorised. Check username, password and tenant name/id
2013-10-24 14:04:44.073 21600 TRACE swiftclient
2013-10-24 14:04:44.161 21600 ERROR savanna.context [-] Thread 'cluster-creating-ab31ed39-a3d5-4c57-9ea5-eafa04fe2295' fails with exception: 'Unauthorised. Check username, password and tenant name/id'
2013-10-24 14:04:44.161 21600 TRACE savanna.context Traceback (most recent call last):
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/context.py", line 120, in _wrapper
2013-10-24 14:04:44.161 21600 TRACE savanna.context func(*args, **kwargs)
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 185, in _provision_cluster
2013-10-24 14:04:44.161 21600 TRACE savanna.context jm.run_job(je)
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 113, in run_job
2013-10-24 14:04:44.161 21600 TRACE savanna.context upload_job_files(u.get_jobtracker(cluster), wf_dir, job, hdfs_user)
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 161, in upload_job_files
2013-10-24 14:04:44.161 21600 TRACE savanna.context raw_data = dispatch.get_raw_binary(lib)
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/binary_retrievers/dispatch.py", line 28, in get_raw_binary
2013-10-24 14:04:44.161 21600 TRACE savanna.context res = i_swift.get_raw_data(context.ctx(), job_binary)
2013-10-24 14:04:44.161 21600 TRACE savanna.context File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/binary_retrievers/internal_swift.py", line 85, in get_raw_data
2013-10-24 14:04:44.161 21600 TRACE savanna.context raise ex.SwiftClientException(e.message)
2013-10-24 14:04:44.161 21600 TRACE savanna.context SwiftClientException: Unauthorised. Check username, password and tenant name/id
2013-10-24 14:04:44.161 21600 TRACE savanna.context
2013-10-24 14:04:58.808 21600 DEBUG savanna.openstack.common.periodic_task [-] Running periodic task SavannaPeriodicTasks.update_job_statuses run_periodic_tasks /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/periodic_task.py:177
2013-10-24 14:04:58.808 21600 DEBUG savanna.service.periodic [-] Updating job statuses update_job_statuses /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/periodic.py:56
2013-10-24 14:05:00.029 21600 ERROR savanna.service.edp.job_manager [-] Error during update job execution c0ad424b-0b2a-477c-8a73-0a970b987fd0: __init__() takes exactly 1 argument (2 given)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager Traceback (most recent call last):
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 74, in update_job_statuses
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager get_job_status(je.id)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 60, in get_job_status
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager job_info = client.get_job_status(job_execution.oozie_job_id)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/oozie.py", line 46, in get_job_status
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager _check_status_code(resp, 200)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/oozie.py", line 72, in _check_status_code
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager raise OozieException(message)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager TypeError: __init__() takes exactly 1 argument (2 given)
2013-10-24 14:05:00.029 21600 TRACE savanna.service.edp.job_manager
2013-10-24 14:05:00.030 21600 DEBUG savanna.openstack.common.loopingcall [-] Dynamic looping call sleeping for 0.86 seconds _inner /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/loopingcall.py:132
2013-10-24 14:05:00.890 21600 DEBUG savanna.openstack.common.periodic_task [-] Running periodic task SavannaPeriodicTasks.terminate_unneeded_clusters run_periodic_tasks /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/periodic_task.py:177
2013-10-24 14:05:00.890 21600 DEBUG savanna.service.periodic [-] Terminating unneeded clusters terminate_unneeded_clusters /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/periodic.py:64
2013-10-24 14:05:00.890 21600 DEBUG savanna.service.periodic [-] Terminating unneeded clusters terminate_unneeded_clusters /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/periodic.py:67
2013-10-24 14:05:00.923 21600 DEBUG savanna.openstack.common.loopingcall [-] Dynamic looping call sleeping for 42.92 seconds _inner /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/loopingcall.py:132
6f1-3b58-4ee8-b384-9c251bf610c32013-10-24 14:05:43.881 21600 DEBUG savanna.openstack.common.periodic_task [-] Running periodic task SavannaPeriodicTasks.update_job_statuses run_periodic_tasks /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/periodic_task.py:177
2013-10-24 14:05:43.882 21600 DEBUG savanna.service.periodic [-] Updating job statuses update_job_statuses /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/periodic.py:56
2013-10-24 14:05:43.938 21600 ERROR savanna.service.edp.job_manager [-] Error during update job execution c0ad424b-0b2a-477c-8a73-0a970b987fd0: __init__() takes exactly 1 argument (2 given)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager Traceback (most recent call last):
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 74, in update_job_statuses
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager get_job_status(je.id)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/job_manager.py", line 60, in get_job_status
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager job_info = client.get_job_status(job_execution.oozie_job_id)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/oozie.py", line 46, in get_job_status
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager _check_status_code(resp, 200)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager File "/home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/service/edp/oozie.py", line 72, in _check_status_code
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager raise OozieException(message)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager TypeError: __init__() takes exactly 1 argument (2 given)
2013-10-24 14:05:43.938 21600 TRACE savanna.service.edp.job_manager
2013-10-24 14:05:43.939 21600 DEBUG savanna.openstack.common.loopingcall [-] Dynamic looping call sleeping for 45.00 seconds _inner /home/ubuntu/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/loopingcall.py:132
---------------------------------------------------------------------
Reason:

in file <savanna_path>/savanna/service/edp/binary_retrievers/internal_swift.py

method

def _get_conn(user, password):
    return swiftclient.Connection(su.retrieve_auth_url(append_tokens=False),
                                  user,
                                  password,
                                  tenant_name=CONF.os_admin_tenant_name,
                                  auth_version="2.0")

User/Password is parameters of method but tenant is transferred from a savanna configuration

Sergey Lukjanov (slukjanov) on 2013-11-19

summary:

- Job with transient cluster is not running
+ [EDP] Job with transient cluster is not running

Revision history for this message

Sergey Lukjanov (slukjanov) wrote on 2013-11-19:

Should be verified, it's very important.

Changed in savanna:
milestone:	none → icehouse-1
importance:	Undecided → High

Sergey Lukjanov (slukjanov) on 2013-11-19

Changed in savanna:
importance:	High → Undecided

Sergey Lukjanov (slukjanov) on 2013-11-26

Changed in savanna:
assignee:	nobody → Alexander Ignatov (aignatov)

Revision history for this message

Alexander Ignatov (aignatov) wrote on 2013-12-03:

Tried to do the same steps described in bug description. Not able to reproduce. Job after creating transient cluster has started successfully and got the SUCCEEDED result. After it cluster has been deleted successfully.

Changed in savanna:
status:	New → Invalid

Revision history for this message

Alexander Ignatov (aignatov) wrote on 2013-12-03:

Moved bug to invalid state. If it'll be reproduced again please reopen it.

Sergey Lukjanov (slukjanov) on 2013-12-03

Changed in savanna:
assignee:	Alexander Ignatov (aignatov) → nobody

Thierry Carrez (ttx) on 2013-12-03

Changed in savanna:
milestone:	icehouse-1 → none

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.