In environment with big amount of nodes, fuel CLI fails hitting timeout

Bug #1795656 reported by Alexander Rubtsov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Oleksiy Molchanov

Bug Description

--- Environment ---
* MOS: 9.2
* 185 slave nodes
* Fuel node is a VM with:
- 20 vCPUs
- 192 Gb RAM

--- Description ---
On executing fuel commands, we are facing "504 gateway timeout" error
[root@fuel ~]# fuel deployment --env 1 --download --dir /var/lib/ericsson/pre_deploy;date;
504 Server Error: Gateway Time-out for url: http://192.168.0.11:8000/api/v1/clusters/1/orchestrator/deployment/?split=1 (<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
)

Out of the box the default value of uwsgi_read_timeout (60 seconds) is used.
Increasing the timeout to 120 seconds has helped in this particular environment.

Please check if the default value in MOS needs to be tuned to suit production environment (or there is some other way to fix it)

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla1 for 9.0-updates

Changed in fuel:
importance: Undecided → High
assignee: nobody → MOS Maintenance (mos-maintenance)
milestone: none → 9.x-updates
tags: added: customer-found sla1
Changed in fuel:
milestone: 9.x-updates → 9.2-mu-9
status: New → Confirmed
Changed in fuel:
assignee: MOS Maintenance (mos-maintenance) → Oleksiy Molchanov (omolchanov)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Oleksiy Molchanov <email address hidden>
Review: https://review.fuel-infra.org/39372

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/fuel-library (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/39372
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 4272320998f54637dc512e36ed349c76467c64c1
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Oct 4 12:50:29 2018

Increase uwsgi_read_timeout for fuel-api to 120

Closes-Bug: 1795656
Change-Id: Id511610fe4f336333b4750719c9c9852f4bf59ef

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Mikhail Samoylov (msamoylov) wrote :

[root@nailgun ~]# grep -R 'uwsgi_read_timeout' /etc/nginx/
/etc/nginx/conf.d/services.conf: uwsgi_read_timeout 120;
[root@nailgun ~]# fuel --fuel-version
api: '1'
auth_required: true
feature_groups: []
openstack_version: mitaka-9.0
release: '9.2'

[root@nailgun ~]#

Revision history for this message
Mikhail Samoylov (msamoylov) wrote :

Verified

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.