Swift processes constantly overload controllers after reboot

Bug #1589999 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
High
MOS Swift
Mitaka
Won't Fix
High
MOS Swift

Bug Description

Fuel version info (9.0 mos #427): http://paste.openstack.org/show/508639/

After all controllers graceful reboot (simultaneous), could becomes broken for 1 hour because different services can't start and work properly due to heavy overload (CPU). After 1 hour my environment was stabilized and OpenStack basic function started to work (OSTF passed), but all controllers are still overloaded by Swift python processes:

http://paste.openstack.org/show/508640/

node specs: http://paste.openstack.org/show/508641/

Also Swift processes are being constantly re-spawned by something (PIDs are changed):

http://paste.openstack.org/show/508548/

Here is a part of swift logs:

http://paste.openstack.org/show/508547/

Steps to reproduce:

1. Create cluster with 3 controllers and 2 computes, defaults for storage
2. Reboot all cluster nodes
3. Run OSTF

Expected result: tests are passed

Actual result: test fails for ~ 1 hour due to controller nodes overload

Diagnostic snapshot: https://drive.google.com/file/d/0BzaZINLQ8-xkcjV4MDF2TktSSVk/view?usp=sharing

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → MOS Swift (mos-swift)
Dina Belova (dbelova)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Anastasia Kuznetsova (akuznetsova) wrote :

As for me, it is to risky to try to fix it quickly before HCF, because:
- tested case is not usual (rebooting of all nodes), it means, that issue is reproduced only in specific scenario, not after every deployment;
- finally, env goes to "working" status (> "After 1 hour my environment was stabilized and OpenStack basic function started to work (OSTF passed)")

Revision history for this message
Dina Belova (dbelova) wrote :

Agree with Nastya, moving the bug to 9.0-updates for the 9.0 series.

tags: added: area-swift move-to-mu
removed: swift
tags: added: swarm-fail
Changed in fuel:
status: Confirmed → Won't Fix
tags: added: move-to-9.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.