Fuel for OpenStack

504 gateway timeout occasionally blocks Web UI

Bug #1391935 reported by Matthew Mosesohn
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 6.0
5.1.x
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 5.1.1
6.0.x
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 6.0

Bug Description

User reported case.

[root@fuel ~]# fuel --debug node
GET http://192.168.0.100:8000/api/v1/nodes/
HTTP Error 504: Gateway Time-out (<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.0.15</center>
</body>
</html>

What happens is in nailgun logs there are many instances of:
Tue Nov 4 16:34:16 2014 - *** uWSGI listen queue of socket ":8001" (fd: 3) full !!! (100/100) ***

Raising uWSGI listeners and # of listen sockets should reduce this problem.

Matthew Mosesohn (raytrac3r)
Changed in fuel:
importance: Undecided → High
OpenStack Infra (hudson-openstack)
Changed in fuel:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/133526
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e2e1166a55f2dea80ed29dc37ebcaed93f8122c3
Submitter: Jenkins
Branch: master

commit e2e1166a55f2dea80ed29dc37ebcaed93f8122c3
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 10 19:56:10 2014 +0400

    Raise number of nailgun uWSGI workers and sockets

    Fixes issue where nailgun stops responding to requests
    because all workers are occupied and the queue of
    available uWSGI sockets fills up by increasing the
    number of workers (double) and raising the connection
    limit from 100 to 4096.

    Closes-Bug: #1391935

    Change-Id: Id6bb0fc259ee78fa5dc33ce072ba426e595278f7

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/134219

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/134219
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=88a8b82db851434bca098918eda644074b24943f
Submitter: Jenkins
Branch: stable/5.1

commit 88a8b82db851434bca098918eda644074b24943f
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 10 19:56:10 2014 +0400

    Raise number of nailgun uWSGI workers and sockets

    Fixes issue where nailgun stops responding to requests
    because all workers are occupied and the queue of
    available uWSGI sockets fills up by increasing the
    number of workers (double) and raising the connection
    limit from 100 to 4096.

    Closes-Bug: #1391935

    Change-Id: Id6bb0fc259ee78fa5dc33ce072ba426e595278f7
    (cherry picked from commit e2e1166a55f2dea80ed29dc37ebcaed93f8122c3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/134481

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/134481
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=39dd5e2bd811bbcc472d48890f9e3c5ae499cc64
Submitter: Jenkins
Branch: master

commit 39dd5e2bd811bbcc472d48890f9e3c5ae499cc64
Author: Matthew Mosesohn <email address hidden>
Date: Fri Nov 14 12:37:48 2014 +0400

    Raise number of nailgun uWSGI workers and sockets

    Fixes issue where nailgun stops responding to requests
    because all workers are occupied and the queue of
    available uWSGI sockets fills up by increasing the
    number of workers (double) and raising the connection
    limit from 100 to 4096.

    This requires a kernel setting of net.core.somaxconn, which
    is now being applied on Fuel Master host because it cannot
    be done on an unprivileged Docker container.

    Change-Id: I14724d5c6953570d53b6990ef0d6dccf07ddbaa4
    Closes-Bug: #1391935

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/134511

Mike Scherbakov (mihgen)
Changed in fuel:
status: Fix Committed → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/5.1)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/134511
Reason: Abandoned due to reverted in master

Matthew Mosesohn (raytrac3r)
Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Nikolay Markov (nmarkov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/135018

Changed in fuel:
assignee: Nikolay Markov (nmarkov) → Matthew Mosesohn (raytrac3r)
status: New → In Progress
Fabrizio Soppelsa (fsoppelsa)
Changed in fuel:
milestone: 6.0 → 5.1.1
tags: added: customer-found
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/135018
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7cec9cc552386ff57366b73097eb7cebc58ad3f0
Submitter: Jenkins
Branch: master

commit 7cec9cc552386ff57366b73097eb7cebc58ad3f0
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 17 20:21:55 2014 +0400

    Raise number of nailgun uWSGI workers and sockets

    Fixes issue where nailgun stops responding to requests
    because all workers are occupied and the queue of
    available uWSGI sockets fills up by increasing the
    number of workers (double) and raising the connection
    limit from 100 to 4096.

    This requires a kernel setting of net.core.somaxconn, which
    is now being applied on Fuel Master host because it cannot
    be done on an unprivileged Docker container.

    Change-Id: Ib25d45c75c34a71b24fe620bf1077f71c2b8af6d
    Closes-Bug: #1391935

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/135636

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/135636
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=85e71c24eed03f69538ca311487deee9f8f189e5
Submitter: Jenkins
Branch: stable/5.1

commit 85e71c24eed03f69538ca311487deee9f8f189e5
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 17 20:21:55 2014 +0400

    Raise number of nailgun uWSGI workers and sockets

    Fixes issue where nailgun stops responding to requests
    because all workers are occupied and the queue of
    available uWSGI sockets fills up by increasing the
    number of workers (double) and raising the connection
    limit from 100 to 4096.

    This requires a kernel setting of net.core.somaxconn, which
    is now being applied on Fuel Master host because it cannot
    be done on an unprivileged Docker container.

    Change-Id: Ib25d45c75c34a71b24fe620bf1077f71c2b8af6d
    Closes-Bug: #1391935
    (cherry picked from commit 7cec9cc552386ff57366b73097eb7cebc58ad3f0)

Revision history for this message
Leontii Istomin (listomin) wrote :

I have faced with this issue with 6.1-233.

api: '1.0'
astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
auth_required: true
build_id: 2015-03-26_21-32-43
build_number: '233'
feature_groups:
- mirantis
fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
production: docker
python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
release: '6.1'

Trird to deploy the following confguration:
Baremetal, Ubuntu, HA, Neutron-gre,Ceph-all, Debug, nova-quotas, 6.1-233
Controllers:3 Computes:197

After I faced with this issue https://bugs.launchpad.net/fuel/+bug/1438265, I deleted one node, added another one and click to "deploy changes" button. After that UI was unreachable, Fuel CLI respond with 504.
From nailgun/app.log:
uWSGI listen queue of socket ":8001" (fd: 3) full !!! (4097/4096)

docker-logs folder is attached

tags: added: scale
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.