excessive memory usage on file list download

[beefstick]

Description of the issue

I have just downloaded a number of file lists ranging from several kB to several MB and this is what happens

Mem: 2053260k total, 1891236k used, 162024k free, 2496k buffers
Swap: 2095100k total, 255924k used, 1839176k free, 384880k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27935 me 21 1 2250m 1.1g 8540 S 2 56.1 1:20.87 linuxdcpp

The filelists are all parsed correctly as they should but the memory usage increases to <insane> and stays that way forever.
Since filelists are never re-used by the program in any way they should be unloaded from memory as well or at least placed into cached so that it can be overwritten.
An other option is to create the possibility for downloading partial file lists to limit the use of RAM.

Steps to reproduce

Download several filelists of up to 60 MB or even more

Expected result

Increase of RAM usage to accommodate for parsing filelist, take out info needed, dump filelist to reduce RAM usage.

Actual result

Increase or RAM usage to accommodate for parsing filelist, take out info needed, leave unneeded info in RAM

Version

Compiled from latest BZR

OS

Opensuse 11.3 to 12.1

No backtrace since no crash

[Steven Sheehy (steven-sheehy)]

There are four scenarios where the filelist is loaded as far as I can tell.

1) You manually load a file list from disk by opening it in the file menu and it opens up the tab to browse the share
2) You download someone's list and it opens up the tab to browse the share when the download is complete.
3) You download a directory and the list is downloaded by default
4) You choose the match queue option in search, transfers, hub, or favorite users window.

In option 1 & 2, the list is loaded in memory in order to support the viewing of that list. Until you close the tab associated with that list the memory will obviously not be released.

Option 3 doesn't look to be storing the loaded list, it simply uses it to find additional sources then discards it.

Option 4 might have some issue. It caches a single list in a static TTH map for some reason. The next time you match a file list this cache should be cleared and the memory released.

Can you try this:

- Confirm in which scenarios above you are seeing the issue.
- If it's option 4, see if matching the queue of a person with a large list increases the memory to the levels you mentioned for a sustained period of time.
- Then see if matching the queue of a person with a small list causes the memory to reduce to normal levels.

[beefstick (splitwitch)]

Okay, I did a test for all 4 options. I hope I will give enough info, if you need more I am happy to supply but I am not available for the coming 1.5 weeks.

A quick recap of what is in the log below: I see the issue in all 4 scenarios
Matching a queue with a smaller file list does not decrease memory usage.

Here is my log:
***********
Fresh reboot of the system
linuxDC++ Idle 1 adcs hub connected, no queue, no uploads, autosearch for alternates=off, max file list size set to 300MB otherwise lists of more than 20MB will not open at all.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3198 me 21 1 906m 164m 15m S 1 8.2 0:25.83 linuxdcpp

1) You manually load a file list from disk by opening it in the file menu and it opens up the tab to browse the share

Filelist = 64.4MB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3198 me 21 1 1324m 502m 15m S 1 25.1 1:25.04 linuxdcpp

Close file list + wait 5 minutes to make sure that memory can be given back

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3198 me 21 1 1324m 502m 15m S 2 25.1 1:31.14 linuxdcpp

Nothing given back

2) You download someone's list and it opens up the tab to browse the share when the download is complete.

Started from scratch with freshly started linuxdcpp (%mem=8.2) downloaded the same file list (64.4MB)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4777 me 21 1 1370m 535m 15m S 1 26.7 1:40.66 linuxdcpp

Note that because of the download the memory increased 1.6% more than is the case for 1), I have no idea why but it might be causing something.
Close file list and wait 5 minutes

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4777 me 21 1 1370m 536m 15m S 1 26.7 1:50.26 linuxdcpp

Nothing given back

3) You download a directory and the list is downloaded by default

Started from scratch with freshly started linuxdcpp (%mem=8.2) downloaded file list 9.0MB

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5376 me 21 1 1185m 222m 15m S 1 11.1 0:38.33 linuxdcpp

Memory increase even though file list is not opened by the program to show it on screen

4) You choose the match queue option in search, transfers, hub, or favorite users window.

Match queue on 1 user (filelist 3.91MB) after 3)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5376 me 21 1 1185m 249m 15m S 1 12.4 0:48.92 linuxdcpp

Memory increase even though file list is not opened
Match again (without waiting) on different user (filelist 3.16MB)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5376 me 21 1 1185m 255m 16m S 1 12.8 1:10.23 linuxdcpp

Again memory increase while file list is smaller
Match again on different user (filelist 11.98MB)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5376 me 21 1 1249m 289m 16m S 1 14.4 1:23.87 linuxdcpp

Memory increase on big file list, as expected.
Waiting for 10 minutes
Match again on different user (filelist 2.98MB, smallest I could find)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5376 me 21 1 1250m 289m 16m S ...

[beefstick (splitwitch)]

I observed some additional, reproducible behavior when matching a queue on several 10's of persons. I see a rapid memory increase, but sometimes some memory is released. Though never so much to get back to the idle state of a fresh started program. In the end I expect the memory is released because the program can not be partially in swap space because I only see this behavior if the usage of swap increases a lot.

[Night (night.)]

this is probobly due to using FastAlloc<> in directorylisting. the memory remains cached even after closing the filelist.
its not very noticable with small filelists but can easily be seen with opening big ones.

[beefstick (splitwitch)]

Changed
class File : public FastAlloc<File> {
To
class File /*: public FastAlloc<File>*/ {

Same for class Directory

Did a quick test, might be solving something, will do more tests soon and report.

[Launchpad Janitor (janitor)]

[Expired for LinuxDC++ because there has been no activity for 60 days.]

[Steven Sheehy (steven-sheehy)]

beefstick, what was the result of your testing?

[beefstick (splitwitch)]

Sorry forgot about this, work work work...

With change in source
idle
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11787 me 21 1 1028m 146m 14m S 7 7.4 0:22.22 linuxdcpp

first file list 38.45MB
11787 me 21 1 1508m 387m 14m S 1 19.5 1:00.28 linuxdcpp

Closed no change

second file list 608.85kB
11787 me 21 1 1501m 387m 14m S 1 19.5 1:09.60 linuxdcpp

closed no change

match queue on file, memory increase
12152 me 21 1 1688m 434m 15m S 5 21.7 1:15.13 linuxdcpp

without change in source
idle
12061 me 21 1 1034m 148m 14m S 1 7.4 0:19.12 linuxdcpp

first file list
12061 me 21 1 1362m 374m 14m S 1 18.7 0:57.09 linuxdcpp

Closed no change

second file list
12061 me 21 1 1434m 374m 14m S 2 18.7 1:00.74 linuxdcpp

closed no change

match queue on file, memory increase
12061 me 21 1 1642m 402m 14m S 0 20.0 1:18.76 linuxdcpp

As you can see the change in the source might even be a bad idea. When downloading a file list there is a bigger increase compared to the unchanged source. However, a match on queue shows a simmilar increase for both configurations.

As for now I have no further ideas what might be causing this behavior. At the moment I manage by restarting my client after all file list transfers are finished.