[MASTER] Committing to urlclassifier3.sqlite causes excessive CPU usage and disk I/O

[Parag Warudkar]

Note: the follow up bug for this is bug #229745:
 #229745 - after fix for #215728 - Committing to urlclassifier3.sqlite still causes excessive CPU usage and disk I/O (the 2nd)

===

Browsing with Firefox 3.0b5 on Hardy (All updates applied) causes Xorg to use 50-60% CPU all the time.

This is not limited to a particular site or page - it happens any time when Firefox is rendering pages.

The high CPU usage makes browsing very jerky - switching tabs takes more time than it should, pages appear frozen for a brief moment.

===
From https://bugzilla.mozilla.org/show_bug.cgi?id=430530#c4:

After some testing, it looks like this gets pretty bad on linux.

Once we hit sqlite's cache size, sqlite starts hitting the disk pretty hard.
While the database is small this isn't a big deal (we don't hit the cache
limit), and once we're up-to-date it isn't as bad (we aren't adding as much
data).

But during the middle/end of the push to get the complete database, things take
way too long, and we're thrashing the disk the whole time.

We can reduce the amount of IO by increasing the cache size, and that'll help
some (it'll help a whole lot if we increase the cache size by enough to hold
the db). We might also need to look at throttling the update process a bit, so
that we don't slam the system all at once...

===

To reproduce:

1. enable browser.safebrowsing.malware.enabled + browser.safebrowsing.enabled
2. observe IO on urlclassifier3.sqlite while browsing the web

[Miguel Martinez (el-quark)]

This is also happening here. Firefox 3 on uptodate Hardy is also causing a lot of I/O, which is probably the eventual cause of the high CPU usage. Furthermore, there is also more people affected, as evidenced in the following forum thread:

http://ubuntuforums.org/showthread.php?t=759673

[liamdawe (liamdawe)]

I am also having the exact same problem doing "top" in terminal shows firefox using massive amounts of cpu like a person pointed out in the thread linked above!

[Andrea Garbarini (garba)]

same problem here, this is making firefox unusable for me, i hope the devs understand how serious this issue is

[unggnu (unggnu)]

I can confirm this with current Hardy. The cpu usage seems to mainly come from the high disc activity.

[Miguel Martinez (el-quark)]

Some say that disabling the xulrunner language packs does help. I can't confirm it right now, though. I will try it and see what happens.

[Miguel Martinez (el-quark)]

I have disabled the language addons (first only xulrunner and then both xulrunner and firefox) and the hard disk trashing still occurs. Please note that I have been running the firefox update that came this morning (April 22). However, epiphany-browser (using gecko 1.9) works just fine.

[Alexander Sack (asac)]

please submit info about this firefox package version you are exactly using. Further, please provide an example url that consumes that much CPU.

Also submit a complete list of extensions you are using. try to run firefox -safe-mode to see if disabling extensions helps.

[Miguel Martinez (el-quark)]

I'm using firefox version 3.0~b5+nobinonly-0ubuntu3 and xulrunner-1.9 version 1.9~b5+nobinonly-0ubuntu3. Extensions used are:
-Adblock Filterset.G updater 0.3.1.3
-Adblock Plus 0.7.5.4
-Flashblock 1.5.5
-Ubuntu Firefox Modifications 0.5

I need to wait some time for the HD access to occurr, and it has happened in http://www.phoronix.com/scan.php?page=home and at http://planet-f1.com/ I've just had it happen in google, with no other tabs opened and loaded directly after my home page. I was browsing with epiphany while this happened, so that may invalidate the results. Some users also report issues in ubuntuforums.com although I have been using epiphany for two days and can't comment.

I will try to use firefox in safe mode and report results as soon as possible.

[Miguel Martinez (el-quark)]

OK, I just ran firefox in safe mode and just left it at my home page http://www.phys.ufl.edu/fermisurface/ After a while, it started using CPU and using the hard disk, although memory usage according to htop didn't skyrocket. Completely closing firefox takes a while once the "thrashing" has started: more than it takes to close the window as if it were doing some I/O operations after that. The system goes back to normal as soon as it is closed.

It would be great if some other could confrim or deny what I'm seeing here.

[bendis (bendis)]

Hi,

I have exactly the same symptoms. It does not matter what page is loaded in firefox or whether the safe-mode is on or not, the cpu and disk trashing starts in a minute or two after firefox is launched. Firefox becomes unresponsive and closing it takes about ten seconds. Once closed, cpu and disk calm down.

This problem started to occur a few days ago...

My versions are:
firefox 3.0~b5+nobinonly-0ubuntu3
xulrunner-1.9 1.9~b5+nobinonly-0ubuntu3

[bendis (bendis)]

Well, I just found out what is firefox doing during this high cpu and disk activity - it does something nasty with the urlclassifier3.sqlite file (about 20MB in size) in may home firefox profile directory. The urlclassifier3.sqlite is being touched and the urlclassifier3.sqlite-journal file constantly grows.

I just googled around and found this: http://www.mozillazine.org/talkback.html?article=22714 (look for "Firefox 3 beta 1 memory hog on Windows").

Regards,
Bendis

[Nick B. (futurepilot)]

Yes, it does not matter what site you visit. The high CPU and disk trashing will happen almost immediately after launching Firefox 95% of the time and then periodically afterwards, even if Firefox is idle.

[Gumper (rlutzinger)]

Same problems as everyone else. Lots of HD activity and FF becomes unresponsive. This just started happening about 2 days ago for me.

[50sQuiff (aflitcroft)]

I have absolutely the same symptoms being described here too. This has started happening after a very recent update.

[unggnu (unggnu)]

Have you tried a new profile? Close Firefox, move .mozilla to a backup location and then restart the browser. Maybe temporary but it fixes the problem for me so bendis could be right.

[unggnu (unggnu)]

Btw. some information about urlclassifier3.sqlite http://forums.mozillazine.org/viewtopic.php?t=640383. In the standard configuration checking of "suspected forgery" is disabled in Firefox in Ubuntu.

[sebbouckaert (sebastiaan-bouckaert)]

Same happens here with Firefox 3.0 b5+nobinonly. And this started after updating Hardy on april 22th (CET). I also happen to use Nvidia-new driver and Compiz.

[bendis (bendis)]

to unggnu: I just deleted the urlclassifier file - Firefox created a new one (9KB in size) and everything seems fine now...

Bendis

[Bryce Harrington (bryce)]

From the comments (esp. #19) it seems this is not an xorg issue, so closing that component.

[Christian Wolf (christianwolf)]

If Firefox 3.0 b6 does not make it into the final release, I suggest to roll back to FF 3.0b4 and to skip b5.

I also had to stop using FF b5 on my Hardy systems since it is barely usable after a few minutes of browsing.

Together with the random scrollkeeper attacks on my HD, CPU and memory which started to show up again since a few days, this is a rather unpleasant desktop experience at the last day before release.

I would call this a typical example of Murphy's law..... ;-)

[unggnu (unggnu)]

@bendis
If you don't disable the forgery and attack site options the db will be updated gradual again.

I guess the beta5 of Firefox will stay for three years (LTS desktop support time) or are there plans to upgrade it with newer Hardy releases?

[getaceres (getaceres)]

The same happens to me. Since the last update (2 or 3 days ago) I get an unusable firefox that eats CPU constantly.

[Chris Coulson (chrisccoulson)]

Changing this from Incomplete to Confirmed, as I can only see one request for information here (Alexander), and it was answered by Miguel.

This is also happening on my system (same version of firefox and xulrunner as everyone else)

[Dcamp (dcamp)]

There's a thread at http://ubuntuforums.org/showthread.php?t=759673 complaining that some users are seeing painful amounts of disk IO during safebrowsing updates.

I'm not sure how widespread this is, but regardless I think we can help this a lot with some index and cache size tweaking. I'm working on getting some numbers and a patch together now.

[Reed Loden (reed)]

Heh, I _just_ mailed you about this, and then I look at my bugmail, and I see this bug was just filed. Hah.

[Reed Loden (reed)]

See also http://ubuntuforums.org/showthread.php?t=763773 and http://ubuntuforums.org/showthread.php?t=763742.

[Alexander Sack (asac)]

If you see this, please stop any firefox running, backup your urlclassifier3.sqlite from your .mozilla/firefox/... profile. then remove that file from your profile and restart firefox. Does this help?

[Alexander Sack (asac)]

btw, please keep the backup in case we need more info.

[Beltzner (beltzner)]

Dave, can you renom when you've got those numbers together to help us understand the impact/size of the problem?

[Alexander Sack (asac)]

no replies to my last question. setting to incomplete again.

[Chris Coulson (chrisccoulson)]

Alexander - I got rid of the file last night, and I didn't notice any more lock-ups. The file was re-created but only 9kB. I will need to use it a bit more tonight after work before I can conclude whether it has fixed the issue.

[Nick B. (futurepilot)]

I removed the urlclassifier3.sqlite file, but it did not help any.

[Chris Coulson (chrisccoulson)]

My urlclassifier3.sqlite file has grown back to 17.4MB this evening and the periodic high CPU load and disk I/O seems to have returned

[Dcamp (dcamp)]

After some testing, it looks like this gets pretty bad on linux.

Once we hit sqlite's cache size, sqlite starts hitting the disk pretty hard. While the database is small this isn't a big deal (we don't hit the cache limit), and once we're up-to-date it isn't as bad (we aren't adding as much data).

But during the middle/end of the push to get the complete database, things take way too long, and we're thrashing the disk the whole time.

We can reduce the amount of IO by increasing the cache size, and that'll help some (it'll help a whole lot if we increase the cache size by enough to hold the db). We might also need to look at throttling the update process a bit, so that we don't slam the system all at once...

I've sent mail to Dr. Hipp looking for suggestions, and any other suggestions would be appreciated.. :/

[Parag Warudkar (parag-warudkar)]

Alexander - your last question was "If you see this, please stop any firefox running, backup your
urlclassifier3.sqlite from your .mozilla/firefox/... profile. then
remove that file from your profile and restart firefox. Does this help?"

And it was answered 3 times - Chris Coulson (2 times - helped initially but file grew and had the same problem) and Nick B (1 time - did not help any.)

Setting it to Confirmed again.

[unggnu (unggnu)]

The file keeps growing to previous size if the phishing and attack site filter isn't disabled.

If the file can be compressed well maybe it is a good idea to upload it here so that the devs can retrace it. With 7z my 22 MB file is reduced to 5.7 MB. I don't know if it is too much for Launchpad but since I don't experience the problem atm I couldn't upload it. Theoretically only both options have to be enabled and Firefox restarted several times (Firefox needs some time to download the part) to gain the full size of the file again since it is downloaded in chunks.

[unggnu (unggnu)]

It seems to happen again with a fresh profile and both options enabled so I upload the file. If it is too big or not needed anymore just delete it. For testing copy both files to Firefox profile and wait some time ~1 minute I guess. It even happens with a blank page.

[f3a97 (f3a97)]

Hi,
      same problem here too.

I've previously opened a new bug, where I attached some iostat and Systemtap measuerements of FF3 (iotime and disktop).

Please see them here:

https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/221842

I can confirm too that this huge IO activity was not present some days ago, so it must be a recent Hardy change.

[Gleidson Lacerda (gleidsonlm)]

+1

Linux version 2.6.24-16-generic (buildd@palmer) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Thu Apr 10 13:23:42 UTC 2008
Mozilla/5.0 (X11; U; Linux i686; pt-BR; rv:1.9b5) Gecko/2008041514 Firefox/3.0b5

[Rocco (rocco)]

This fix didn't fix the problem I'm seeing in https://bugs.launchpad.net/bugs/224320, which is marked as a dup of this bug.

Removed .mozilla and I still see the same problem as reported at 224320, after install of proposed xulrunner-1.9

[Martin Kaffanke (martin-kaffanke)]

Since upgrade to the xulrunner from hardy-proposed, my firefox crashes after watching videos on www.autsch.de and www.clipfish.de - I can watch one movie, the second crashes firefox.

[jeroenl (jeroenl)]

I didn't experienced any high disk I/O and high CPU since upgrading to the mt-version.

@Rocco and @Martin: Are you sure you closed/killed all firefox processes after upgrading?

@Martin: I can't confirm any crashes on those sites with those flash videos.

[Pjotr12345 (computertip)]

Still testing (over a day now) and still no problems. This update really works. Good job! :-)

[Alexander Sack (asac)]

Rocco, I unduped your bug. You are seeing something else as you are sure that you experience no high IO load.

[Alexander Sack (asac)]

Daniel, afaict you don't need to remove any existing urlclassifier database.

[R (Chandra) Chandrasekhar (chandra)]

I have added to /etc/apt/sources.list the following two lines:

deb http://ppa.launchpad.net/mozillateam/ubuntu hardy main
deb-src http://ppa.launchpad.net/mozillateam/ubuntu hardy main

The currently installed packages are:

firefox-3.0_3.0~b5+nobinonly-0ubuntu3_i386.deb
xulrunner-1.9_1.9~b5+nobinonly-0ubuntu4~8.04.0mt1_i386.deb

The previously encountered problems have not recurred.

I have one question, though. Having ticked both the "suspected attack site" and "suspected forgery site" boxes, I find that I now have only

urlclassifier3.sqlite

in my .mozilla/firefox/<blah> directory, and it is growing at a modest rate. But there is no

urlclassifier2.sqlite

which was one of the files I was advised to delete previously.

Does this have any significance?

Thanks.

[f3a97 (f3a97)]

Hi,
       I upgraded the xulrunner too, up to now the behaviour is ok!

Thanks for the fix.

When it will be available as a standard distribution package?

[Martin Pitt (pitti)]

stek79. The packages are currently in hardy-proposed and should be tested from there (not from the PPA). Once we collected enough test results, they will go into hardy-updates, where all users will get them automatically.

[Alexander Sack (asac)]

yes, please remove the PPA lines from your sources.list ... enable -proposed in System -> Administration -> Software sources ... and run

  sudo apt-get update; sudo apt-get install --reinstall xulrunner-1.9

Then test and submit your test-results.

Thanks!

[Colin Watson (cjwatson)]

This is all rather subjective, as I didn't time it beforehand, but Firefox feels a lot more responsive after this fix, particularly just after entering a URL and when switching tabs.

[Benjamín Valero Espinosa (benjavalero)]

I am using the proposed update and all works fine now (at least till this moment). By the way, I agree Colin, I feel Firefox more responsive. Good work!

[Daniel T Chen (crimsun)]

Using xulrunner-1.9 from hardy-proposed, verified that symptoms are no longer present using:
1) existing user with existing FF 2.0.0.14 profile (gutsy-security) migrated to FF 3.0b5 profile (hardy-proposed);
2) existing user with existing FF 3.0b5 profile (hardy-proposed);
3) new user with new FF 3.0b5 profile (hardy-proposed).

[Martin Pitt (pitti)]

For the record, I have used the new xulrunner for over a day here and didn't notice any regressions. I don't use any fancy extensions, though (just AdBlock plus), and the proprietary flash.

[Pjotr12345 (computertip)]

Today (10 hours ago), I repeated the testing with xulrunner in proposed, instead of xulrunner from upstream (after applying Alexander Sack's instruction: https://bugs.launchpad.net/ubuntu/+bug/215728/comments/94

Again, no problems after a day of intensive web surfing.

*Please* don't wait any longer with spreading this update through the normal channels. This bug is so bad, that reparation is extremely urgent.

[Chris Coulson (chrisccoulson)]

I can't test the version in hardy-proposed, because I can find a mirror that's up-to-date enough, and the connection to Main is patchy at best

[Samuel Lidén Borell (samuellb)]

I can confirm that this update doesn't introduce any regressions (the urlclassifier files are updated, web sites I use to visit work, etc). I've tested on a computer (and Firefox profile) upgraded from Ubuntu 5.04->5.10->6.06->8.04. I did some quick testing with a fresh profile and it seemed to work well too.

[Moshewe (orelweinstock)]

Same here.

[Jruderman (jruderman)]

*** Bug 432444 has been marked as a duplicate of this bug. ***

[Gabriel (gabriel-nobody)]

same problem , same behaviour here. Laptop goes unusable.
I went back to firefox 2 for now.

[R (Chandra) Chandrasekhar (chandra)]

No problems, but two questions.

Before the workaround, I had two files:

4970496 2008-04-26 23:42 urlclassifier2.sqlite
23249920 2008-04-27 17:01 urlclassifier3.sqlite

Now that I have installed xulrunner-1.9 version 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 from hardy-proposed, I do not have the file

urlclassifier2.sqlite

I only have the file urlclassifier3.sqlite so:

33275904 2008-05-07 10:25 urlclassifier3.sqlite

I have enabled both suspected attack sites and suspected forgery sites.

1. Is this absence of urlclassifier2.sqlite normal behaviour?

2. What does it imply?

Thanks.

[Martin Pitt (pitti)]

Verification looks good so far, marking verification-done. I'll copy this once Alex answers to the previous question and it is not a problem.

[Alexander Sack (asac)]

urlclassifier2.sqlite is the old filename and is not used anymore.

The schema/filename policy reads like this:

// The database filename is updated when there is an incompatible
// schema change and we expect both implementations to continue
// accessing the same database (such as between stable versions of the
// platform).
#define DATABASE_FILENAME "urlclassifier3.sqlite"

// The implementation version is updated during development when we
// want to change schema, or to recover from updating bugs. When an
// implementation version change is detected, the database is scrapped
// and we start over.
#define IMPLEMENTATION_VERSION 4

This update only bumped IMPLEMENTATION_VERSION.

[Martin Pitt (pitti)]

Copied to hardy-updates.

[dave87 (david-roth)]

Same Problem with my laptop here. Extremely hard-disk usage with firefox 3 Beta5 and hardy heron makes laptop very very slow.
not very nice ...

[Dan Andreșan (danyer)]

dave87, look at the post above yours (posted 11 hours ago).

Fix was released, use System|Administration|Update Manager to download the latest updates. It will be two updates xulrunner-something.

Enjoy,
Dan.

[Tormod Volden (tormodvolden)]

I installed the 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 from comment 69, but still had this problem. I have now reinstalled from hardy-updates. I see the version name is the same, but can I expect it to work better now?

[Alexander Sack (asac)]

On Wed, May 07, 2008 at 08:45:30PM -0000, Tormod Volden wrote:
> I installed the 1.9~b5+nobinonly-0ubuntu4~8.04.0mt1 from comment 69, but
> still had this problem. I have now reinstalled from hardy-updates. I see
> the version name is the same, but can I expect it to work better now?
>

No, you are most likely seeing an unrelated issue (most likely a
rendering operation or javascript consuming high CPU ... or flash).

 - Alexander

[Tormod Volden (tormodvolden)]

I should have been more clear: I mostly saw high i/o load, not CPU usage, and the .sqlite file was ~30MB. This came several days after installing your package (and deleting the old .sqlite files). Now things run fine with the official update, but I wonder if it can kick in again after some time.

[kulight (kulight)]

I'm also still getting excessive disk I/O after the update.
fire-fox grinds the disk while surfing

[f3a97 (f3a97)]

Are you sure it is Firefox to cause that IO load? Have you measured it? If you want I have a Systemtap script useful to find the IO bound process.

With the proposed fix I've never experienced the issue again.

[kulight (kulight)]

The disk I/O is not as bad or as frequent as before. But it's still there, i have no idea how to measure it or where it comes from if you have a script or instructions on how to do this (measuring) i'll be happy to do that and report.

[unggnu (unggnu)]

I can confirm this. The main problem is gone but it seems partly shifted. Every half an hour or something like that Firefox and nearly the whole system freezes (at least all disk tasks) for around ten seconds. My laptop hard disk isn't so fast so I guess Firefox syncs the the database everytime when it happens. It is now btw. 50 MB. If I disable the options the problem disappears.
The kernel should better be able to scale/queue intensive hard disk ios so not one task is able to halt the whole system (Bug #131094).

[f3a97 (f3a97)]

Hi guys,
     two useful methods to measure IO by process are the following, based on the great SystemTap linux kernel tool.

The disktop.stp script, which can be found attached here, measure the first N processes odered by IO activity - a 'top' for disk resource, we could say. I've found it on the net, it seems to be made by Oracle and it is the porting to SystemTap of the equivalent Dtrace script.

Once found the culprit process, we can also see the response time of each IO request that the process make with the iotime.stp script, which is provided with the systemtap package (/usr/share/doc/systemtap/examples/iotime.stp)

I hope you can find them useful. Let us know if firefox is actually the process that hurts your performance. I've noticed that in Ubuntu there are also other very high disk demanding tool, for example the scrollkeeper-update or the slocate db updater.

I think it should be paid more attention to the disk resource, often its impact on performance is overlooked but actually it is important. If the culprit process is a background tool, you can lower its IO priority with ionice - thanks to the linux fair CFQ scheduler.

Let us know what you find!

[Nick B. (futurepilot)]

Yes, there is still some heavy disk I/O going on. It's no where near as bad as it was before the update. This patch is a HUGE improvement, but it still happens from time to time causing Firefox to become unresponsive for a few seconds.

[Tormod Volden (tormodvolden)]

The easiest way to see which processes are doing i/o is to use iotop, there are packages in my PPA.

[f3a97 (f3a97)]

Hi,
   actually I've just measured some firefox heavy IO activity, but as Nick reported, it is a matter of seconds and in my case it doesn't make the whole system irresponsive as it was before the patch.

Luckily I had disktop running, so here is the output. I can say that the the slowdown that I felt was probably due to the large write of 5.5MB in 5 seconds you can see at 18:28:31 - some random IO activity I guess.

Sun May 11 18:28:11 2008 , Average: 657Kb/sec, Read: 2642Kb, Write: 647Kb

     UID PID PPID CMD DEVICE T BYTES
    1000 5880 1 firefox sda3 R 2605104
    1000 5880 1 firefox sda3 W 646660
    1000 5875 1 transmission sdb2 R 98304
    1000 5875 1 transmission sdb2 W 16384
    1000 5807 1 multiload-apple sda3 R 2049

Sun May 11 18:28:16 2008 , Average:1031Kb/sec, Read: 2965Kb, Write: 2194Kb

     UID PID PPID CMD DEVICE T BYTES
    1000 5880 1 firefox sda3 R 2936832
    1000 5880 1 firefox sda3 W 2246792
    1000 5875 1 transmission sdb2 R 98304
    1000 5807 1 multiload-apple sda3 R 1366

Sun May 11 18:28:21 2008 , Average: 856Kb/sec, Read: 2842Kb, Write: 1438Kb

     UID PID PPID CMD DEVICE T BYTES
    1000 5880 1 firefox sda3 R 2826240
    1000 5880 1 firefox sda3 W 1440504
    1000 5875 1 transmission sdb2 R 81920
    1000 5875 1 transmission sdb2 W 32768
    1000 5807 1 multiload-apple sda3 R 2049

Sun May 11 18:28:26 2008 , Average: 607Kb/sec, Read: 2089Kb, Write: 948Kb

     UID PID PPID CMD DEVICE T BYTES
    1000 5880 1 firefox sda3 R 2023424
    1000 5880 1 firefox sda3 W 971588
    1000 5875 1 transmission sdb2 R 114688
    1000 5807 1 multiload-apple sda3 R 1366
       0 5331 1 preload sda3 W 131

Sun May 11 18:28:31 2008 , Average:1144Kb/sec, Read: 254Kb, Write: 5468Kb

     UID PID PPID CMD DEVICE T BYTES
    1000 5880 1 firefox sda3 W 5599737
    1000 5880 1 firefox sda3 R 131204
    1000 5875 1 transmission sdb2 R 114688
    1000 6157 5970 iostat sda3 R 11652
    1000 5807 1 multiload-apple sda3 R 2049
    1000 6157 5970 ba...

[Pjotr12345 (computertip)]

I can confirm both the improvement because of the patch, and the remaining hugely diminished problem.

Oh well, I have simply disabled the warning for attack websites. Who cares anyway, all they attack is Windows machines..... :-)

[Alexander Sack (asac)]

ok, i have cut out a new bug for the issues left:

https://bugs.edge.launchpad.net/ubuntu/+source/firefox/+bug/229745:
 #229745 - after fix for #215728 - Committing to urlclassifier3.sqlite still causes excessive CPU usage and disk I/O (the 2nd)

if you want to follow there, feel free to subscribe to that bug.

[Tormod Volden (tormodvolden)]

(Just write bug #229745 and you get a correct bug link)