Comment 10 for bug 1265068

Revision history for this message
Trevor McKay (tmckay) wrote :

Here is a failure case, fyi. It seems that if you submit a job right after the cluster becomes "Active", it can fail. Apparently Oozie is not quite ready. Edge case, but part of the general problem:

2014-02-25 11:11:17.339 10952 ERROR savanna.context [-] Thread 'Starting Job Execution 428a39c1-817b-4c93-be1b-f1cc3124e378' fails with exception: 'HTTPConnectionPool(host='10.0.3.2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib.BadStatusLine'>: '')'
2014-02-25 11:11:17.339 10952 TRACE savanna.context Traceback (most recent call last):
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/context.py", line 124, in _wrapper
2014-02-25 11:11:17.339 10952 TRACE savanna.context func(*args, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/service/edp/job_manager.py", line 167, in run_job
2014-02-25 11:11:17.339 10952 TRACE savanna.context job_execution)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/savanna/service/edp/oozie.py", line 37, in add_job
2014-02-25 11:11:17.339 10952 TRACE savanna.context "Content-Type": "application/xml;charset=UTF-8"
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 425, in post
2014-02-25 11:11:17.339 10952 TRACE savanna.context return self.request('POST', url, data=data, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
2014-02-25 11:11:17.339 10952 TRACE savanna.context resp = self.send(prep, **send_kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
2014-02-25 11:11:17.339 10952 TRACE savanna.context r = adapter.send(request, **kwargs)
2014-02-25 11:11:17.339 10952 TRACE savanna.context File "/home/tmckay/src/savanna2/.tox/venv/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
2014-02-25 11:11:17.339 10952 TRACE savanna.context raise ConnectionError(e)
2014-02-25 11:11:17.339 10952 TRACE savanna.context ConnectionError: HTTPConnectionPool(host='10.0.3.2', port=11000): Max retries exceeded with url: /oozie//v1/jobs (Caused by <class 'httplib.BadStatusLine'>: '')