10.0-community.main.ubuntu.bvt_2 #988 failed

[Ivan Suzdal]

Job URL: https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/988
Snapshot: https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/988/artifact/logs/fail_error_ceph_rados_gw-fuel-snapshot-2016-12-22_07-52-50.tar

-----
Traceback:
...
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request
    raise exceptions.from_response(resp, method, url)
GatewayTimeout: Gateway Timeout (HTTP 504
...

nginx/access_services.log from the master node:
10.109.10.1 - - [22/Dec/2016:07:51:18 +0000] "GET /api/tasks/5 HTTP/1.1" 504 176 "-" "run_system_test.py keystoneauth1/2.11.1 python-requests/2.11.1 CPython/2.7.6"

Atop log from the master node:
http://paste.openstack.org/show/593123/
-----

Looks like keystone request failed due to poor I/O performance on the master node.

[Oleksiy Molchanov (omolchanov)]

Yes, master is high loaded, it seems to be a root cause.

[Ivan Kolodyazhny (e0ne)]

The same error occurs with https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1191/