[staging] Fetching of diagnostic snapshot failed: HTTP Error 504: Gateway Time-out

[Ilya Shakhat]

Staging job failed with the following traceback:
Fetching of diagnostic snapshot failed: Traceback (most recent call last):
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 90, in wrapper
    "fail", name)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 343, in create_diagnostic_snapshot
    task = env.fuel_web.task_wait(env.fuel_web.client.generate_logs(), 60 * 10)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 130, in wrapped
    response = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/models/nailgun_client.py", line 329, in generate_logs
    return self.client.put("/api/logs/package")
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 83, in put
    return self._open(req)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 92, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/7.0.staging.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 109, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 504: Gateway Time-out

http://jenkins-product.srt.mirantis.net:8080/job/7.0.staging.ubuntu.bvt_2/162/console

[Matthew Mosesohn (raytrac3r)]

This issue can be attributed to overloading on srv17-msk. Later in the logs, all provisioning timed out as well. There's nothing we can do if API calls are taking >60s to acknowledge, which is in the nginx logs. This is not typical of any actual deployment and instead more closely related to our build environment. The only thing that can be fixed is ensuring enough CPU and memory resources are available before starting a BVT run.