Fuel for OpenStack

Fuel did not respond some time after start deployment on big cluster

  1. Series newton
  2. Bug #1657721
Bug #1657721 reported by Sergey Galkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Fuel Sustaining
Fuel for OpenStack 11.0
Nominated for Ocata by Alexander Kislitsky
Mitaka
Won't Fix
High
Alexander Kislitsky
Fuel for OpenStack 9.x-updates
Newton
Confirmed
High
Fuel Sustaining
Fuel for OpenStack 10.1

Bug Description

1. Install 9.0
2. Upgrade to 9.2 from http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2017-01-13-184421/x86_64
3. Create, configure and provision cluster with 374 nodes.
4. On the cluster page select 'Deployment' only and press 'Deploy on 374 nodes'

After this Fuel hangs on the long time

On example
[root@fuel ~]# fuel2 node list

504 Server Error: Gateway Time-out for url: http://10.21.0.2:8000/api/v1/version (<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
)

Fuel logs available on
http://mos-scale-share.mirantis.com/fuel-9.2-2017-01-19-logs.tar.gz

Revision history for this message
Sergey Galkin (sgalkin) wrote :

After 10-20 minutes Fuel start working again

Oleksiy Molchanov (omolchanov)
Changed in fuel:
milestone: none → 9.3
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
importance: Undecided → High
Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

For encrease speed of nodes API response we need handle list of required fields of node data and use this list in the Fuel CLI.

Changed in fuel:
status: New → Triaged
Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

Nailgun is working on data and graph serialization. It takes dozens minutes for hundreds of nodes. As fix we shouldn't acquire all processors for serialization.

Changed in fuel:
milestone: 9.3 → 11.0
assignee: Fuel Sustaining (fuel-sustaining-team) → Alexander Kislitsky (akislitsky)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/425630

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/425630
Reason: Should be done in different way.

Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

For 10 we should move serizalization to the nodes.
For 9.x we have workaround for scale - set LCM_SERIALIZERS_CONCURRENCY_FACTOR = CPU_NUMBER - 1 to the Nailgun settings. It saves one CPU for handling API requests. For 9.x set status to Won't fix.

Changed in fuel:
assignee: Alexander Kislitsky (akislitsky) → Fuel Sustaining (fuel-sustaining-team)
status: In Progress → Confirmed
Oleksiy Molchanov (omolchanov)
tags: added: area-python
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.