swift-object-reconstructor memory leak
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Medium
|
Unassigned |
Bug Description
version: 2.5.0-0ubuntu1~
We have a cluster of two nodes with 120 disks each, two storage policies: EC and replica-2
Since we started to put data in a container with an EC policy we noticed the memory used by
swift-object-
out of memory (the reconstructor process was using 85% of 128 GB of RAM).
On the irc channel that appeared to be common knoledge but I couldn't find any bug related to
that.
The work around of restarting the process every N hours has the strong downside of making
the reconstruction start from the beginning each time and so not really a solution with nodes with
lots of storage/devices as the process will never complete.
Changed in swift: | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in swift: | |
importance: | High → Medium |
I confirm having the same behavior here, with a EC ring of 48 disk, over 4 nodes. 12 disks per node, currently filled at 75%.
At the end of the reconstruction process (which takes about 15 days) the reconstructor process is consuming more than 80GB on the 96GB RAM available on each node.
I need to kill the reconstructor once it has reach the end of the last device to free the memory.
I can't kill it earlier as it always start from the first device and perform one device after the other.