Memory leak during file list update

[Ben]

Version: r1728

When I refresh my file list, memory usage steadily increases while shared directories are scanned for new or modified files, and does not decrease again after the refresh is complete. If I initiate another refresh, memory usage again increases by the same amount. Since I share a very large number of files and directories (226709 files), memory usage can surpass 1GB in less than a day.

[Ben (mimenarrator)]

This bug also exists in version 0.75.

[Ben (mimenarrator)]

Here's a strange bit of information that might help find the cause of this bug: the first refresh after I rebuilt my hash database did not cause memory usage to increase, but the bug returned on subsequent refreshes.

[kwolfe (mr-kwolfe)]

I've also noticed this problem since DC++ 0.75 when sharing many files and directories. After its been open for a few days I need to close and reopen DC++ because it is consuming most of the memory on my computer.

[Ben (mimenarrator)]

It looks to me like this leak was probably introduced in revision 1638, when some sharemanager memory was automanaged.

[Ben (mimenarrator)]

I'd just like to add that I've now verified that this bug continues to exist in r1765. It can be pretty bad for users with large file list so I would think a fix would be a priority.

[Steven Sheehy (steven-sheehy)]

I can confirm the memory leak and it is indeed introduced in revision #1638. The memory leak is due to never releasing ShareManager::Directory objects. A simple printf("Deleting Directory: %s\n", name.c_str()); in the Directory destructor proves this.

[Big Muscle (bigmuscle)]

Each directory object contains list of file and subdirectories. Each file and (sub)directory has a reference to its parent object. And it seems that these lists are never cleared, therefore reference count to parent objects is never decreased and such object will never be destroyed.

[Big Muscle (bigmuscle)]

So it seems that using common pointer to parent object (instead of intrusive pointer) solves the problem.

ShareManager::Directory and ShareManager::Directory::File >>> GETSET(Directory*, parent, Parent);
their constructors: parent(aParent.get())

[Steven Sheehy (steven-sheehy)]

That is the same conclusion I came to in my testing. Committed a fix to trunk.

[Razzloss (razzloss)]

What about Linuxdcpp fix? You intend to wait for next core release and upgrade to that? Or are just using windows users as guinea pigs?

--RZ

[Ben (mimenarrator)]

To get r1815 to compile on my machine I must edit \boost\boostnoncopyable.hpp to make const noncopyable& operator=( const noncopyable& ); and noncopyable( const noncopyable& ); protected instead of private, though I imagine there is a more elegant solution.

[Steven Sheehy (steven-sheehy)]

Razzloss: Yeah, I was thinking to just wait until next core release since we're not planning a release anytime soon. But if you want to check in the patch, go right ahead.

[Razzloss (razzloss)]

Done. Apparently there's quite a few people using bzr-trunk as their primary client, so less annoying (and fixed) bugs in it means more useful bug reports for us (hopefully), when people are able to actually test it without restarting every couple of hours...

--RZ

[dairinin (nowhere-hakkenden)]

Running for two days. Rehashing every 30 minutes. Added some files to share, removed - no problems with memory:

$ cat /proc/`pidof linuxdcpp`/status
Name: linuxdcpp
State: S (sleeping)
Tgid: 24231
Pid: 24231
PPid: 24228
TracerPid: 0
Uid: 102 102 102 102
Gid: 100 100 100 100
FDSize: 256
Groups: 100
VmPeak: 459116 kB
VmSize: 388008 kB
VmLck: 0 kB
VmHWM: 100816 kB
VmRSS: 85288 kB
VmData: 268716 kB
VmStk: 84 kB
VmExe: 2104 kB
VmLib: 16692 kB
VmPTE: 460 kB
Threads: 8
SigQ: 0/36864
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001006
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
voluntary_ctxt_switches: 542185
nonvoluntary_ctxt_switches: 102076

[Steven Sheehy (steven-sheehy)]

Reverted fix due to bug #480065.

[Jacek Sieka (arnetheduck)]

tentative fix in bzr - items weren't being reparented correctly...