Comment 0 for bug 1419890

Capomastro simply cannot run builds on Jenkins anymore after a spec upgrade. Apparently this is because unit jenkins/0/'s /var/lib/jenkins is inside a persistent volume, and this is kept after migrations or upgrades of the spec, which somehow leads to some inconsistency in Capomastro's database?

I can reproduce it on Staging after a normal deployment + a few dependencies builds + complete redeployment of the spec (using the same Jenkins storage volume). IS has the exact same scenario and possibly would have caught this before, but I decided to file a bug so we can investigate it a bit more when Capomastro goes live.

To be honest the log from this failure does not look related at all, but this is what I get from Celery when this happens:

[2015-02-04 01:29:00,049: INFO/MainProcess] Received task: jenkins.tasks.build_job[b4c3a6d2-a978-4f21-ba41-18ba567c710c]
[2015-02-04 01:29:00,068: INFO/Worker-1] Starting new HTTP connection (1): 10.35.94.203
[2015-02-04 01:29:00,881: ERROR/MainProcess] Task jenkins.tasks.build_job[b4c3a6d2-a978-4f21-ba41-18ba567c710c] raised unexpected: ContentDecodingError(DecodeError('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',)),)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 218, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 398, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/jenkins/tasks.py", line 19, in build_job
    client = job.server.get_client()
  File "/usr/lib/python2.7/dist-packages/jenkins/models.py", line 27, in get_client
    self.url, username=self.username, password=self.password)
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/jenkins.py", line 43, in __init__
    JenkinsBase.__init__(self, baseurl)
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 33, in __init__
    self.poll()
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 55, in poll
    self._data = self._poll()
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 59, in _poll
    return self.get_data(url)
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 63, in get_data
    response = requester.get_url(url, params)
  File "/usr/lib/python2.7/dist-packages/jenkinsapi/utils/requester.py", line 89, in get_url
    return requests.get(self._update_url_scheme(url), **requestKwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 394, in send
    r.content
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 679, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 621, in generate
    raise ContentDecodingError(e)
ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))