[devops] Fuel master node overload causes system tests failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
High
|
Fuel DevOps |
Bug Description
Fuel version info (7.0 build #139): http://
System tests failed on CI because they got 504 error from Nginx during cluster deployment:
http://
2015/08/06 02:59:53 [error] 642#0: *575 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.109.0.1, server: localhost, request: "GET /api/tasks/2 HTTP/1.1", upstream: "uwsgi:
...
10.109.0.1 - - [06/Aug/
According to monitoring logs the VM for Fuel master node was overloaded.
ATOP logs from CI node:
http://
ATOP logs from Fuel master node VM:
http://
Looks like there was an issue with disk system performance (IOwait values are high) on CI node 'mc2n7-srt'.
Changed in fuel: | |
status: | New → Confirmed |
tags: | added: area-devops |
@Artem, what are your expectations?
This load is generated by tests so probably proper solution would be researching and improving tests in that direction(like retries).