Here is a failure case, fyi. It seems that if you submit a job right after the cluster becomes "Active", it can fail. Apparently Oozie is not quite ready. Edge case, but part of the general problem:
2014-02-25 11:11:17.339 10952 ERROR savanna.context [-] Thread 'Starting Job Execution 428a39c1-817b-4c93-be1b-f1cc3124e378' fails with exception: 'HTTPConnectionPool(host='10.0.3.2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib.BadStatusLine'>: '')'
2014-02-25 11:11:17.339 10952 TRACE savanna.context Traceback (most recent call last):
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/context.py", line 124, in _wrapper
2014-02-25 11:11:17.339 10952 TRACE savanna.context func(*args, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/service/edp/job_manager.py", line 167, in run_job
2014-02-25 11:11:17.339 10952 TRACE savanna.context job_execution)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/service/edp/oozie.py", line 37, in add_job
2014-02-25 11:11:17.339 10952 TRACE savanna.context "Content-Type": "application/xml;charset=UTF-8"
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 425, in post
2014-02-25 11:11:17.339 10952 TRACE savanna.context return self.request('POST', url, data=data, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
2014-02-25 11:11:17.339 10952 TRACE savanna.context resp = self.send(prep, **send_kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
2014-02-25 11:11:17.339 10952 TRACE savanna.context r = adapter.send(request, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
2014-02-25 11:11:17.339 10952 TRACE savanna.context raise ConnectionError(e)
2014-02-25 11:11:17.339 10952 TRACE savanna.context ConnectionError: HTTPConnectionPool(host='10.0.3.2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib.BadStatusLine'>: '')
Here is a failure case, fyi. It seems that if you submit a job right after the cluster becomes "Active", it can fail. Apparently Oozie is not quite ready. Edge case, but part of the general problem:
2014-02-25 11:11:17.339 10952 ERROR savanna.context [-] Thread 'Starting Job Execution 428a39c1- 817b-4c93- be1b-f1cc3124e3 78' fails with exception: 'HTTPConnection Pool(host= '10.0.3. 2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib. BadStatusLine' >: '')' tmckay/ src/savanna2/ savanna/ context. py", line 124, in _wrapper tmckay/ src/savanna2/ savanna/ service/ edp/job_ manager. py", line 167, in run_job tmckay/ src/savanna2/ savanna/ service/ edp/oozie. py", line 37, in add_job xml;charset= UTF-8" tmckay/ src/savanna2/ .tox/venv/ lib/python2. 7/site- packages/ requests/ sessions. py", line 425, in post 'POST', url, data=data, **kwargs) tmckay/ src/savanna2/ .tox/venv/ lib/python2. 7/site- packages/ requests/ sessions. py", line 383, in request tmckay/ src/savanna2/ .tox/venv/ lib/python2. 7/site- packages/ requests/ sessions. py", line 486, in send send(request, **kwargs) tmckay/ src/savanna2/ .tox/venv/ lib/python2. 7/site- packages/ requests/ adapters. py", line 378, in send ool(host= '10.0.3. 2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib. BadStatusLine' >: '')
2014-02-25 11:11:17.339 10952 TRACE savanna.context Traceback (most recent call last):
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context func(*args, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context job_execution)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context "Content-Type": "application/
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context return self.request(
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context resp = self.send(prep, **send_kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context r = adapter.
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/
2014-02-25 11:11:17.339 10952 TRACE savanna.context raise ConnectionError(e)
2014-02-25 11:11:17.339 10952 TRACE savanna.context ConnectionError: HTTPConnectionP