Large amount of tftp processes on the Fuel master

Bug #1625716 reported by Andrew Kalach
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
In Progress
Medium
Alexey Shtokolov
Mitaka
Confirmed
Medium
Fuel Sustaining

Bug Description

During Rally test suite run on big scale.
On Fuel master node where discovered large number of tftp processes:

  [root@fuel ~]# ps auxwwwwf | grep in.tftp | wc -l
  1721
  [root@fuel ~]# w
  16:47:00 up 4 days, 22:58, 8 users, load average: 119.04, 139.14, 92.04

which causes loading system and slowing down of "Nova_create_and_delete_secgroups" test.

It was found that two nodes which weren't included in the cluster and weren't discovered by Fuel, but resided in the lab environment constantly generating error messages: "Could not find kernel images" trying to boot.

After that nodes were powered off the tftp processes on the Fuel master were released.

Conclusion: expected behavior - cluster shouldn't be affected by some nodes which aren't included into the cluster. Undiscovered nodes shouldn't create pefromance impact on the Fuel master node.

Tags: area-library
Andrew Kalach (akndex)
description: updated
description: updated
description: updated
Changed in mos:
milestone: none → 9.2
Revision history for this message
Michael Semenov (msemenov) wrote :

Very hard to reproduce such status of nodes, but it's definitely needs attention that not properly deployed 2 nodes can slow down 200-nodes cluster. It seems that on 500, 1000, 2000 nodes it can be reproduced much easier, and potential consequences might be much worse. So, let's try to fix it in 9.2.

no longer affects: mos
tags: added: area-library
no longer affects: fuel/newton
Changed in fuel:
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/384361

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Richard Berwald (rberwald)
status: Confirmed → In Progress
Changed in fuel:
assignee: Richard Berwald (rberwald) → Alexey Shtokolov (ashtokolov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/384361
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.