firefox-3.0 breaks with NFS home directory

[Tessa]

Binary package hint: firefox-3.0

My home directory is an NFS mount, and when Firefox-3.0 (beta 5 and RC2 tested on Hardy) doesn't already have a working configuration in ~/.mozilla, it doesn't start correctly. It opens, but the bookmarks menu is missing, and it is impossible to type anything into the location or search bars. I haven't figured out any way to fix it aside from binding a non-NFS directory to ~/.mozilla and then starting it.

As well, even if a working configuration dir is created, extensions like DownloadThemAll break when used off NFS, which seems to indicate a widespread issue with Firefox on NFS.

[Tsteiner (tsteiner)]

Created attachment 303047
Initial browser window after creating a fresh profile while using an AFP home directory.

[Dstillman+bugzilla (dstillman+bugzilla)]

I can confirm this:

Error: [Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]" nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)" location: "JS frame :: file:///Applications/Minefield.app/Contents/MacOS/components/nsBrowserGlue.js :: bg__initPlaces :: line 386" data: no]
Source File: file:///Applications/Minefield.app/Contents/MacOS/components/nsBrowserGlue.js
Line: 386

The error is in initializing Places. I suspect this is due to needing the SQLITE_ENABLE_LOCKING_STYLE SQLite macro uncommented in sqlite.c on OS X. See http://<email address hidden>/msg21123.html for details.

We're seeing this in our mozStorage-backed extension as well, with openDatabase() throwing NS_ERROR_FILE_IS_LOCKED. Latest SQLite command-line client also doesn't work over AFP without that macro.

mozStorage seemed to work over AFP in Firefox 2, so this is a regression.

[Dstillman+bugzilla (dstillman+bugzilla)]

SQLite changelog lists "Get the SQLITE_ENABLE_LOCKING_STYLE macro working again on MacOSX" for 3.5.7, so I don't know what effect it would have in 3.5.4.2 (which appears to be Fx3's version).

[rogerdodd (roger-dodd)]

I can confirm this bug with AFP mounted profile directories. At my place of work our home directories (and therefore firefox profiles) are AFP mounted and Firefox 3 is broken as described by Tim Steiner. This will be a complete show-stopper for us if we upgrade and suddenly everyone's browsers no longer work. As such, I would regard this as absolutely critical to be fixed prior to release.

[Dietrich-mozilla (dietrich-mozilla)]

confirming, from the multiple independent confirmations above.

[Phiw (phiw)]

*** Bug 437313 has been marked as a duplicate of this bug. ***

[Shawn Wilsher (sdwilsh)]

This may be fixable, but might be a bit slower too - see http://<email address hidden>/msg34473.html

[Tessa (unit3)]

Binary package hint: firefox-3.0

My home directory is an NFS mount, and when Firefox-3.0 (beta 5 and RC2 tested on Hardy) doesn't already have a working configuration in ~/.mozilla, it doesn't start correctly. It opens, but the bookmarks menu is missing, and it is impossible to type anything into the location or search bars. I haven't figured out any way to fix it aside from binding a non-NFS directory to ~/.mozilla and then starting it.

As well, even if a working configuration dir is created, extensions like DownloadThemAll break when used off NFS, which seems to indicate a widespread issue with Firefox on NFS.

[Beltzner (beltzner)]

Will relnote this, but won't block the release on it. This is something we need to address in a branch update, though.

[Bshirley (bshirley)]

I'm having the same show-stopping issue for Firefox 3 RC 2, OS X 10.5.3
It makes me sad - we just went to AFP mounted home directories, and I've lost the FireFox i've been running solidly in beta for months.

[Shawn Wilsher (sdwilsh)]

Lovely - the version of sqlite that is in the tree does not compile with the AFP support. We'll need a newer version of sqlite on branch (bug 435414) if we want to try this).

[Shawn Wilsher (sdwilsh)]

Created attachment 324848
v1.0

This enables AFP on the mac. Note, the library on the system that Apple uses also has this enabled.

I also cleaned up the Makefile a bit since I was in the area.

[Beltzner (beltzner)]

(relnote added)

[Tsteiner (tsteiner)]

I created a build from the official 3.0 sources with the patch from this bug and the patches from bug 435414 and the issue seems to be resolved!

[Shawn Wilsher (sdwilsh)]

I hope to push this to mozilla-central tomorrow, and then make an argument for drivers that we should take this for 3.0.1.

[Phiw (phiw)]

*** Bug 439706 has been marked as a duplicate of this bug. ***

[Gsmith-ku (gsmith-ku)]

This same behavior exists when the home folder is a SMB based share. We utilize the ability of the Mac to use our existing Windows Domain with home folders and the release version firefox 3 does not work with the home folders operating off of the domain resource.

[Shawn Wilsher (sdwilsh)]

Pushed to mozilla-central:
http://hg.mozilla.org/mozilla-central/index.cgi/rev/c9b5882bf6f6

(In reply to comment #16)
> This same behavior exists when the home folder is a SMB based share. We utilize
> the ability of the Mac to use our existing Windows Domain with home folders and
> the release version firefox 3 does not work with the home folders operating off
> of the domain resource.
Can you possibly test to see if this patch solves your issue? It'll be in tomorrows nighlies for 3.1.

[Abillings (abillings)]

*** Bug 440044 has been marked as a duplicate of this bug. ***

[Phiw (phiw)]

*** Bug 440244 has been marked as a duplicate of this bug. ***

[Phiw (phiw)]

*** Bug 440159 has been marked as a duplicate of this bug. ***

[rogerdodd (roger-dodd)]

FYI - I have tested the nightly for 080619 and this patch has solved the problems with AFP-mounted profile directories on our systems. Thanks for your help with fixing the issue.

[Gsmith-ku (gsmith-ku)]

FYI, the nightly build seems to have resolved the same issue on systems using smb-mounted profiles as well.

Thanks for the community's quick turn around on this bug. Any ETA on getting it integrated into the current version 3 as a point release?

[Ryanvm (ryanvm)]

Assuming that the SQLite 3.5.9 update sticks, I believe the aim is to get this fixed for Firefox 3.0.1.

[Shawn Wilsher (sdwilsh)]

(In reply to comment #23)
> Assuming that the SQLite 3.5.9 update sticks, I believe the aim is to get this
> fixed for Firefox 3.0.1.
That is the going plan - still waiting on blocking status for sqlite and this bug.

[Shawn Wilsher (sdwilsh)]

verified per comment 21 and comment 22

Thanks guys.

[Htroberts (htroberts)]

*** Bug 440502 has been marked as a duplicate of this bug. ***

[Shawn Wilsher (sdwilsh)]

I'd *really* like to see this get in for 3.0.1. This fix has been landed in mozilla-central for a little more than ten days, with no noticeable performance regressions. The fix has been verified to fix the issues with *both* AFP and smb-mounted profile folders (comment 21 and comment 22 respectively).

I have a red-eye flight tonight, so I don't know how much I'm going to be around tomorrow. If I'm not, can a driver please make sure someone else lands this if I'm not around?

[Matti-mversen (matti-mversen)]

*** Bug 442749 has been marked as a duplicate of this bug. ***

[Beltzner (beltzner)]

Comment on attachment 324848
v1.0

a=beltzner, sorry for losing track of this

[Reed Loden (reed)]

Checking in db/sqlite3/src/Makefile.in;
/cvsroot/mozilla/db/sqlite3/src/Makefile.in,v <-- Makefile.in
new revision: 1.36; previous revision: 1.35
done

[Tchung-mozilla (tchung-mozilla)]

updating status whiteboard to verified per comment 25

[Shawn Wilsher (sdwilsh)]

Had to back this out for bug 442949

[Matti-mversen (matti-mversen)]

*** Bug 444776 has been marked as a duplicate of this bug. ***

[Matti-mversen (matti-mversen)]

*** Bug 445460 has been marked as a duplicate of this bug. ***

[Shawn Wilsher (sdwilsh)]

We can try this again when we upgrade to the latest sqlite...

[Florian von Kurnatowski (florian-scalix)]

Made me back-out the migration to server-based home directories for our Mac network. :-(

[Reverend Darkness (reverend-darkness)]

v3.0.1 release appears to resolve issue on SMB network volumes.

[Ryanvm (ryanvm)]

Yes, the updated SQLite was only backed out of the trunk (Firefox 3.1). It was left on the Firefox 3.0.x branch.

[Catalin Marinas (cmarinas)]

I got the same problem. It might be related to this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=417037

[Hskupin (hskupin)]

(In reply to comment #59)
> Yep this seems to happen to all network users, local accounts are fine... I
> will submit a new bug.

Nathan, have you filed a new bug?

[Nathan-pachal (nathan-pachal)]

Hi, yes I did, I can be found here https://bugzilla.mozilla.org/show_bug.cgi?id=502283

[W-administrator-yockeylaw-com (w-administrator-yockeylaw-com)]

I can tell you that this is NOT affecting my network when the user accounts are stored on a OS X 10.4 Open Directory (AFP) server.

[Hskupin (hskupin)]

Nathan, does the problem also occur with Firefox 3.0.x? Would be nice if you could check Firefox 3.0.11, Firefox 3.0.11.

[Nathan-pachal (nathan-pachal)]

Hi,

Version 3.0.10 works most of the time... 3.0.11 doesn't work at all...

[Hskupin (hskupin)]

Can you please check bug 497792 comment 5? Does it stop working with the 2nd build?

[Nathan-pachal (nathan-pachal)]

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/04/2009-04-17-04-mozilla1.9.0/ ***Works***
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/04/2009-04-18-04-mozilla1.9.0/ ***Don't Work***

[Hskupin (hskupin)]

(In reply to comment #5)
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/04/2009-04-17-04-mozilla1.9.0/
> ***Works***
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/04/2009-04-18-04-mozilla1.9.0/
> ***Don't Work***

That's really bad. So it is another result of the sqlite upgrade. :(

Nathan, you you please check the following build if the problem still persists for you? It should be fixed in the upcoming 3.0.12:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0.12-candidates/build1/firefox-3.0.12.en-US.mac.dmg

Shawn and Samuel, this issue still persists on 1.9.1. I have setup a smb connection and I can see the same behavior as reported on this bug with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.1pre) Gecko/20090709 Shiretoko/3.5.1pre

[Drh (drh)]

What does that error message mean: "Unable to quarantine <filename>"? Is there some kind of virus scanner or other "security" software preventing files from being written?

From the log, it appears that the "unix-dotlock" VFS is being used. With that
VFS in use, SQLite uses "dot-file" locking, which should work over SMB, NFS, or any other filesystem where file locking is known to be broken. So it is unclear what the problem is. Certainly, nothing in that part of SQLite has changed in a long time. Which SQLite upgrade broke the build?

[Samuel Sidler (samuel-sidler)]

(In reply to comment #6)
> Shawn and Samuel, this issue still persists on 1.9.1. I have setup a smb
> connection and I can see the same behavior as reported on this bug with
> Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.1pre)
> Gecko/20090709 Shiretoko/3.5.1pre

And is it fixed in 1.9.0.12? If so, that makes no sense because we should now be using the same sqlite on both branches (that is, 1.9.0.12 uses the same sqlite that 1.9.1.0 uses).

[Nathan-pachal (nathan-pachal)]

(In reply to comment #7)
> What does that error message mean: "Unable to quarantine <filename>"? Is
> there some kind of virus scanner or other "security" software preventing files
> from being written?
>
> From the log, it appears that the "unix-dotlock" VFS is being used. With that
> VFS in use, SQLite uses "dot-file" locking, which should work over SMB, NFS, or
> any other filesystem where file locking is known to be broken. So it is
> unclear what the problem is. Certainly, nothing in that part of SQLite has
> changed in a long time. Which SQLite upgrade broke the build?

 "Unable to quarantine <filename>" does not have anything to do with security software, as there is no software on that system...

[Hskupin (hskupin)]

(In reply to comment #7)
> changed in a long time. Which SQLite upgrade broke the build?

See bug 488710 for the details. It was the upgrade to 3.6.7. Shawn will know more about it.

(In reply to comment #8)
> And is it fixed in 1.9.0.12? If so, that makes no sense because we should now
> be using the same sqlite on both branches (that is, 1.9.0.12 uses the same
> sqlite that 1.9.1.0 uses).

The release candidate of Firefox 3.0.12 doesn't show this problem for me. Nathan can you please x-check? Please see comment 6.

[Samuel Sidler (samuel-sidler)]

For: Shawn

Because you were mean to me on IRC today.

Love,
Sam

[Nathan-pachal (nathan-pachal)]

So, it would appear that 3.0.12 works as well as 3.0.10 (3.0.11 and 3.5 not working at all.) Also, there is still "Unable to quarantine" error messages pointing to .sqlit stuff in my console log, but Firefox runs... Thanks for your help in sorting this out BTW.

[Shawn Wilsher (sdwilsh)]

That's really weird since 3.0.12 and 3.5 both are running the same SQLite with the same compiler options. I can't explain why one works and the other doesn't.

[Nathan-pachal (nathan-pachal)]

(In reply to comment #6)

Did you get a quarantine message in your console log when you did the smb 3.5 test?

[Hskupin (hskupin)]

I have to add that the release candidate of Firefox 3.0.12 also stops working after the profile has been opened with Firefox 3.5. After that 3.0.12 suffers from the same problems. That happens because an empty places.sqlite-journal is lingering around in this profile. Removing this file and starting 3.0.12 again the bookmarks and history systems works fine. 3.0.12 doesn't create a journal file. CC'ing Dietrich and Marco.

[Hskupin (hskupin)]

Same happens on Windows too.

[Shawn Wilsher (sdwilsh)]

I'm a little confused by the recent summary change. 3.5 (or 1.9.1) is using SQLite 3.6.10 (http://mxr.mozilla.org/mozilla1.9.1/source/db/sqlite3/src/sqlite3.h#110). As is 1.9.0.12 (http://mxr.mozilla.org/seamonkey/source/db/sqlite3/src/sqlite3.h#110)

Also, an empty places.sqlite-journal file is completely OK.

[Hskupin (hskupin)]

Yes, should be 3.6.10.

[Micah Gersten (micahg)]

There seems to have been a regression on the 3.0.x branch, but according to this bug, it's been fixed in 3.0.12:
https://bugzilla.mozilla.org/show_bug.cgi?id=497792

[Shawn Wilsher (sdwilsh)]

To clarify (this bug and it's comments are confusing) - what versions of Firefox are experiencing this?

[Hskupin (hskupin)]

Any recent build on 1.9.0 doesn't suffer from that problem anymore. Even the following build works fine: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.12pre) Gecko/2009071004 GranParadiso/3.0.12pre

The issue I have mentioned in comment 15 is still persistent with a recent build of Gran Paradiso (3.0.14pre). So we could cover this on this bug or file a new one?

Nathan, can you verify that removing the empty places.sqlite-journal file from the profile fixes the problem with Firefox 3.0.x?

[Shawn Wilsher (sdwilsh)]

(In reply to comment #20)
> The issue I have mentioned in comment 15 is still persistent with a recent
> build of Gran Paradiso (3.0.14pre). So we could cover this on this bug or file
> a new one?
File a new bug please.

[Nathan-pachal (nathan-pachal)]

3.0.12 works 3.5.2 does not work. Hope this helps.

[Hskupin (hskupin)]

(In reply to comment #21)
> File a new bug please.

I don't think that we should do that. I feel both are the same issue. So after Nathan's comment I checked again with multiple starts of 3.5.3pre:

1. Created a fresh profile with 3.5.3pre on a SMB folder => WFM
2. Restarted 3.5.3pre => Failed
3. Removed places.sqlite-journal and restarted 3.5.3pre => WFM
4. Started 3.0.14pre => Failed
5. Removed places.sqlite-journal and restarted 3.0.14pre => WFM
6. Restarted 3.0.14pre => WFM

At some point the cookies.sqlite-journal also shows the same behavior. It makes me feel that all journal files with a filesize of 0 makes problems.

[Shawn Wilsher (sdwilsh)]

So, the difference between 1.9.0 and 1.9.1 (and later) is that we have the shared cache turned on in 1.9.0, but do not in 1.9.1. I'm wondering if that is the issue here.

[Hskupin (hskupin)]

Could be related to bug 444763. We would have to make a regression test for the builds 2008-08-16 and 2008-08-17.

[Hskupin (hskupin)]

Mmh it's really hard to track down. There are a couple of issues involved here when doing regression tests. I don't think it is worth doing tests of such old builds because they have an older sqlite3 version which will break Firefox/Shiretoko all the time. On 1.9.0 it was first fixed for Firefox 3.0.1 (bug 417037) and broke again with Firefox 3.0.11 and has been fixed for Firefox 3.0.12 (bug 497792). So it was related to the used sqlite3 version.

When I switch to 3.5 I'm a bit afraid if testing against older sqlite3 versions will help. Given that it is impossible for me to run tests with builds from 2008-08-16 and 2008-08-17. I feel using a debugger would be more helpful. I'll add some debug output which will hopefully help to get closer to this problem.

[Hskupin (hskupin)]

Created an attachment (id=392681)
Debug output

This attachment shows some debug output from a Minefield build (090722)

[Deadpan110 (deadpan110)]

I get the same bug in:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.13) Gecko/2009080315 Ubuntu/8.10 (intrepid) Firefox/3.0.13

After reading the above and testing out various bits and pieces I can confirm it is related to sqlite locks over NFS.

I have not tested with NFSv4 and still use NFSv3 primarily on my network.

To reproduce:

1: Log in on a computer with a NFS /home and start Firefox.
2: Reproduce a power fail or a situation where you hard reboot (Power off)
3: Power back on, log in and start Firefox.

Firefox behaves as described above with missing bookmarks, unusable address bar etc.

To recover:

Close Firefox and either reboot your server or restart nfs file locking on the server (this is not something any regular user is able to do)

It is debatable if NFS or Firefox is to blame but in my opinion - Firefox should have better ways of utilising sqlite and its file locking... Firefox really should detect that its not running and that the sqlite database is locked and attempt to do something about it.

[Micah Gersten (micahg)]

@Deadpan110

Your problem might be Bug #343788 which Mozilla said was an nfs issue.

[Hskupin (hskupin)]

Can we get any update on this bug? As given in comment 26 I cannot do a regression test due to broken builds so I did a debug session and have identified the code which is causing this problem:

In mozStorageConnection::Initialize we are able to open the database but fail on the following line:

srv = sqlite3_prepare_v2(mDBConn, "SELECT * FROM sqlite_master", -1, &stmt,
                         NULL);

We get an SQLITE_IOERR which is "Some kind of disk I/O error occurred".

Richard, is that a known or already fixed bug in sqlite? Affected database file in this case is places.sqlite.

[Drh (drh)]

Didn't we previously determine that the error was SQLITE_IOERR_RDLOCK because
SQLite was not allowed to obtain a read lock on the database file?

[Hskupin (hskupin)]

Created an attachment (id=405344)
sqlite3 variable output

I have no idea if that helps but while stepping through the sqlite code the error is raised inside sqlite3RunParser. Variables output is attached.

[Shawn Wilsher (sdwilsh)]

(In reply to comment #29)
> Didn't we previously determine that the error was SQLITE_IOERR_RDLOCK because
> SQLite was not allowed to obtain a read lock on the database file?
Yes, we had I thought, but possibly in a different bug?

[Hskupin (hskupin)]

Checked the same profile via an AFP connection again and it doesn't show this bug. So it's really SMB related.

[Beltzner (beltzner)]

I don't think we'll block Firefox 3.6 on this as it's not a regression from 3.5; would be nice to get a clearer idea of what's going on. I'm assuming this is OSX-only (as all comments relate to that) since I expect we'd have heard more about this if SMB-based profiles stopped working on Windows. If that assumption is false, renominate, please.

[Gervase Markham (gerv-mozilla)]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body contains places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv