Overloads RAM and the swap

[Byron Chamann]

Binary package hint: thunderbird

When I run Thunderbird it begins to overload the RAM, swap and the processor. This happened after upgrading 08/01/2011.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: thunderbird 3.1.7+build3+nobinonly-0ubuntu0.10.04.1
ProcVersionSignature: Ubuntu 2.6.32-27.49-generic 2.6.32.26+drm33.12
Uname: Linux 2.6.32-27-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Mon Jan 10 21:41:02 2011
InstallationMedia: Ubuntu 10.04.1 LTS "Lucid Lynx" - Release i386 (20100816.1)
ProcEnviron:
 LANG=es_GT.utf8
 SHELL=/bin/bash
SourcePackage: thunderbird

[Braden (braden)]

Upon initial startup, Thunderbird typically consumes 250-380MB of memory. Over the next several minutes, this steadily increases to >900MB with no particular action taken. Clicking on different mail folders at this point will push things up further, frequently to around 1.5GB.

"Global Search and Indexer" is not enabled.

[Vseerror (vseerror)]

Braden, this isn't a great bug report. there is not enough information here to indicate a confirmed bug. You should indicate/explain
- that you started Thunderbird in safe mode
- what does "no particular action taken" mean? Does it mean you didn't touch the UI?
- what account types you have defined and are being checked for new mail/articles/feeds
- what is "several minutes" 3, 10, 30?

[Vseerror (vseerror)]

additionally, isn't this a continuation of bug 526568?

[Ludovic-mozilla (ludovic-mozilla)]

Can you copy about:memory ?

[Vseerror (vseerror)]

let's continue in bug 526568, unless you have strong reason to believe bug 526568 was fixed, and that 671613 didn't exist in 3.1 or is a new issue.

*** This bug has been marked as a duplicate of bug 526568 ***

[Braden (braden)]

It's difficult for me to say with certainty whether bug 526568 was fixed. Memory usage improved since that bug was initially filed; but a footprint of 500-600 MB continued to be typical. However, what I'm observing now is definitely a departure from the 3.1 behavior.

(In reply to comment #1)
> Braden, this isn't a great bug report. there is not enough information here
> to indicate a confirmed bug. You should indicate/explain
> - that you started Thunderbird in safe mode

I haven't tried that yet. I will.

> - what does "no particular action taken" mean? Does it mean you didn't
> touch the UI?

Yes.

> - what account types you have defined and are being checked for new
> mail/articles/feeds

I have two IMAP accounts configured.

> - what is "several minutes" 3, 10, 30?

5-7, I think. I can try to watch it a bit closer.

I'll provide the additional information requested (safe mode, about:memory) shortly.

[Vseerror (vseerror)]

unconfirmed please until safe mode is tested, and we have the cause and details needed for a developer to fix whatever is ailing

[Vseerror (vseerror)]

(In reply to Braden from comment #5)
> I'll provide the additional information requested (safe mode, about:memory)
> shortly.

If you do it with thunderbird 6 or 7 started in safe mode that would be even better.

[Magore (magore)]

I was also seeing >2G memory usage (Under 10.04LTS 32bit Ubuntu with Thunderbird 3.1.11 using IMAP email)
One change I made that had a really big impact on this memory usage was updating all of the max_cached_connections values from 5 to 1 in the config editor
I now only see about 850Meg ram usage

PS I have >200k messages and >300 folders and >300 filters

[Vseerror (vseerror)]

Mike,

>2G is pretty nasty!
can you reproduce high memory with max_cached_connections=5 *and* starting in safe mode? see http://support.mozillamessaging.com/en-US/kb/safe-mode

how many imap accounts?
and, did you by chance enable the hidden preference mail.check_all_imap_folders_for_new ?

[Mozilla-ex (mozilla-ex)]

MSGDB:5 logging would be helpful, along with the size of your folders. Each imap connection will hold flag state in memory, so if you have 10K messages in a folder, and have a lot of tags, that could easily be a couple 100K of memory. But GB is hard to get to...

[Vseerror (vseerror)]

don't have the info needed to debug
=> incomplete

[Ato-henrik-n1b (ato-henrik-n1b)]

Created attachment 581244
Graph of memory consumption over ~3 hours, safe mode, Tb 8.0

I'm enclosing a graph showing the memory consumption over 3 hours.

Thunderbird was started in safe mode when gathering data for the graph.

1 imap account
10 pop accounts
mail.check_all_imap_folders_for_new=False
mail.server.server11.max_cached_connections=5

I'll provide whichever other details desired to be able to look at this.

[Mozilla-ex (mozilla-ex)]

A MSGDB:5 log using the following instructions would be helpful - https://wiki.mozilla.org/MailNews:Logging

You should also check how big your various .msf files are in your profile.

[Ato-henrik-n1b (ato-henrik-n1b)]

Created attachment 581945
MSGDB:5 log corresponding to about 2h 20minutes

(In reply to David :Bienvenu from comment #13)
> A MSGDB:5 log using the following instructions would be helpful -
> https://wiki.mozilla.org/MailNews:Logging

Upploading the MSGDB:5 log, covering about 2h 20m. At the end of this time,
RSS memory was at ~800 MBytes.

> You should also check how big your various .msf files are in your profile.

That's interesting. I didn't realize this would matter, as long as I don't
visit the folders in question in the GUI ...

Anyway, here you go:

15:20 ~/Library/Thunderbird/Profiles/default.6b0
henrik@brunello $ find . -name "*.msf" -print0 | xargs -0 ls -ltdT | grep Dec
-rw-r--r--@ 1 henrik henrik 65781 15 Dec 15:18:24 2011 ./ImapMail/mail.autonomica.se/Junk-2.msf
-rw-r--r--@ 1 henrik henrik 204972 15 Dec 15:13:24 2011 ./Mail/localhost/Junk.msf
-rw-r--r--@ 1 henrik henrik 197037 15 Dec 15:09:13 2011 ./Mail/localhost/Inbox.msf
-rw-r--r--@ 1 henrik henrik 1385434 15 Dec 15:09:13 2011 ./Mail/localhost/robots.sbd/misc.msf
-rw-r--r--@ 1 henrik henrik 3768506 15 Dec 15:09:13 2011 ./Mail/localhost/robots.sbd/root.msf
-rw-r--r--@ 1 henrik henrik 3349 15 Dec 15:08:38 2011 ./Mail/smart mailboxes/Inbox.msf
-rw-r--r--@ 1 henrik henrik 2254 15 Dec 15:08:38 2011 ./Mail/smart mailboxes/Junk.msf
-rw-r--r--@ 1 henrik henrik 8472385 15 Dec 14:58:40 2011 ./Mail/localhost/Sent.msf
-rw-r--r--@ 1 henrik henrik 4818167 15 Dec 14:58:40 2011 ./Mail/localhost/list.sbd/macosx.msf
-rw-r--r--@ 1 henrik henrik 78212397 15 Dec 14:58:40 2011 ./<email address hidden>
-rw-r--r--@ 1 henrik henrik 2738 15 Dec 14:58:40 2011 ./Mail/smart mailboxes/Sent.msf
-rw-r--r--@ 1 henrik henrik 119043 15 Dec 14:54:08 2011 ./Mail/localhost/Drafts.msf
-rw-r--r--@ 1 henrik henrik 2007 15 Dec 14:53:51 2011 ./Mail/smart mailboxes/Drafts.msf
-rw-r--r--@ 1 henrik henrik 3681160 15 Dec 14:53:36 2011 ./Mail/localhost/Archives.sbd/2011.msf
-rw-r--r--@ 1 henrik henrik 17311185 15 Dec 14:53:36 2011 ./Mail/localhost/Trash.msf
-rw-r--r--@ 1 henrik henrik 2604 15 Dec 14:53:36 2011 ./Mail/smart mailboxes/Trash.msf
-rw-r--r--@ 1 henrik henrik 3909422 15 Dec 14:31:45 2011 ./<email address hidden>
-rw-r--r--@ 1 henrik henrik 23174618 15 Dec 14:29:12 2011 ./<email address hidden>
-rw-r--r--@ 1 henrik henrik 19328610 15 Dec 14:28:08 2011 ./Mail/localhost/ietf.sbd/i-d-announce.msf
-rw-r--r--@ 1 henrik henrik 9849 15 Dec 14:27:50 2011 ./ImapMail/mail.autonomica.se/INBOX.msf
-rw-r--r--@ 1 henrik henrik 63861071 15 Dec 14:27:36 2011 ./ImapMail/mail.autonomica.se/Robots.sbd/RT.msf
-rw-r--r--@ 1 henrik henrik 38692314 15 Dec 13:28:27 2011 ./Mail/localhost/list.sbd/python-dev.msf
-rw-r--r--@ 1 henrik henrik 8622400 15 Dec 12:58:18 2011 ./Mail/localhost/list.sbd/python-ideas.msf
-rw-r--r--@ 1 henrik henrik 1208661 15 Dec 12:26:56 2011 ./ImapMail/mail.autonomica.se/Archives.sbd/2011.msf
-rw-r--r--@ 1 henrik henrik 61366296 15 Dec 12:03:16 2011 ./<email address hidden>
-rw-r--r-- 1 henrik henrik 2612 15 Dec 11:48:39 2011 ./Mail/localho...

[Vseerror (vseerror)]

henrik, you did this/see this with thunderbird started in safe mode??
 https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
If you do not see the problem in safe mode, what addons are you using?

800mb isn't surprising with several indexes in the 10MB-70MB range, and msdblog peaking with ~37 dbs open

BTW, this bug is closed, so if we determine there is a legit problem here, the issue should be pursued in a new bug

[Ato-henrik-n1b (ato-henrik-n1b)]

(In reply to Wayne Mery (:wsmwk) from comment #15)
> henrik, you did this/see this with thunderbird started in safe mode??

Yes, this is in safe mode.

> https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
> If you do not see the problem in safe mode, what addons are you using?
>
> 800mb isn't surprising with several indexes in the 10MB-70MB range, and
> msdblog peaking with ~37 dbs open

Hm. Faint comfort. And it doesn't stop there, it keeps increasing till
I restart it again; I've seen it at least up to 1.2 GBytes before restarting,
and yes, it's in safe mode; I've been keeping it there for the last few
days in order to have consistent data to report.

Anything I can tweak in the setting to make this footprint smaller would
be appreciated; it's just causing too much swapping to happen (i.e., too
little memory left for other applications) when it takes this much memory.

> BTW, this bug is closed, so if we determine there is a legit problem here,
> the issue should be pursued in a new bug

Understood.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marius B. Kotsbak (mariusko)]

Translated bug report from Spanish to English. I see this in Precise too (at least the RAM usage).

[Marius B. Kotsbak (mariusko)]

This one might also be related: https://bugzilla.mozilla.org/show_bug.cgi?id=759092

[Vseerror (vseerror)]

Braden, did you stop seeing this problem?

bug 723248 limits open dbs to 30, which should help. Check esp bug 723248 comment 29.

fix is in version 15, available as beta from http://www.mozilla.org/en-US/thunderbird/channel/

[Magore (magore)]

(In reply to Wayne Mery (:wsmwk) from comment #9)
> Mike,
>
> >2G is pretty nasty!
> can you reproduce high memory with max_cached_connections=5 *and* starting
> in safe mode? see http://support.mozillamessaging.com/en-US/kb/safe-mode
>
> how many imap accounts?
> and, did you by chance enable the hidden preference
> mail.check_all_imap_folders_for_new ?

Hi,
   I am now running thunderbird 14.0 and no longer have the problem - even with max-connections=5
I am using IMAP with one account with 514 folders and 614 filters
~/.thunderbird/mail/ImapMail/mail.connect.uwaterloo.ca$ find . -type f | grep "\.msf$" | wc
    514 921 26878
~/.thunderbird/mail/ImapMail/mail.connect.uwaterloo.ca$ grep "^name=" msgFilterRules.dat | wc
    650 1142 12952

Here is my current memory usage - note that it is great!
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15057 magore 20 0 1181m 159m 43m S 0 1.0 4:35.15 thunderbird

[Braden (braden)]

(In reply to Wayne Mery (:wsmwk) from comment #17)
> Braden, did you stop seeing this problem?

Yes, I believe I did. For some releases now, I've not observed the sort of dramatic memory growth that prompted this bug report. It's still not unusual for Thunderbird to consume 400-700 MB of memory; but that's after running for a while. And I can see if that improves in Thunderbird 15.

[Ato-henrik-n1b (ato-henrik-n1b)]

In 15.0b3 the memory consumption behaviour I see has changed dramatically
from that of 8.0. Attaching a new graph of RSS vs. time

[Ato-henrik-n1b (ato-henrik-n1b)]

Created attachment 653305
Graph of memory consumption over ~4 hours, safe mode, Tb 15.0b3

[Vseerror (vseerror)]

Braden, thanks for the update.
WFM based on comment 19. (cause/solution unknown)

Henrik's issue no doubt solved by bienvenu's patch in bug 723248 delivered in beta version 15. TB15 goes public in about a week.

[Marius B. Kotsbak (mariusko)]

This might have been fixed as bug https://bugzilla.mozilla.org/show_bug.cgi?id=723248

[Paul White (paulw2u)]

Upstream reports reolved:

#671613 closed "RESOLVED WORKSFORME" on 2012-08-20,
#759092 closed "RESOLVED WORKSFORME" on 2012-11-14
#723248 closed "RESOLVED FIXED" on 2012-04-26, target Thunderbird 15.0

No comments here for over 6 years so closing as being fixed