MASTER firefox crash [@NSSRWLock_LockRead]

[cement_head]

Binary package hint: firefox

don't know...bug report was automatically generated

ProblemType: Crash
Architecture: i386
Date: Wed Mar 14 19:39:34 2007
DistroRelease: Ubuntu 7.04
ExecutablePath: /usr/lib/firefox/firefox-bin
Package: firefox 2.0.0.2+1-0ubuntu1
PackageArchitecture: i386
ProcCmdline: /usr/lib/firefox/firefox-bin
ProcCwd: /home/andor
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
 LANG=en_US.UTF-8
Signal: 11
SourcePackage: firefox
StacktraceTop:
 __kernel_vsyscall ()
 raise () from /lib/tls/i686/cmov/libpthread.so.0
 ?? ()
 ?? ()
 ?? ()
Uname: Linux oskar3 2.6.20-10-generic #2 SMP Mon Mar 12 00:02:49 UTC 2007 i686 GNU/Linux
UserGroups: adm admin audio cdrom dialout dip fax floppy fuse gdm lpadmin netdev plugdev powerdev root scanner slocate tape users video

From the retraced stacktrace:
...
#3 <signal handler called>
#4 0xb255bf6e in NSSRWLock_LockRead () from /usr/lib/libnss3.so
#5 0xb2534e4d in SECMOD_GetReadLock () from /usr/lib/libnss3.so
#6 0xb2547567 in PK11_GetAllTokens () from /usr/lib/libnss3.so
#7 0xb254789a in PK11_GetBestSlotMultiple () from /usr/lib/libnss3.so
#8 0xb254793c in PK11_GetBestSlot () from /usr/lib/libnss3.so
#9 0xb2534497 in PK11_CreateDigestContext () from /usr/lib/libnss3.so
#10 0xb252630e in md5_NewContext () from /usr/lib/libnss3.so
#11 0xb2526444 in HASH_Create () from /usr/lib/libnss3.so
#12 0xb25ef014 in nsCryptoHash::Init (this=0xb257c748, algorithm=2)
...

[Doug-turner (doug-turner)]

doesn't psm / nss need a profile before it will work correctly?

[Darin-moz (darin-moz)]

Yes, but there is a profile ;-) Firefox selects a profile before starting XPCOM.

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

PSM requires a profile? Why? In profileless situations can't it use the default certs?

[Doug-turner (doug-turner)]

Mozilla requires some profile directory. For nss/psm, they find the secmod, cert, key, db's in this directory. Of course, you can start NSS in readonly mode.

[Christian Biesinger (cbiesinger)]

bug 309877 is about making PSM/NSS work w/o profile

[Darin-moz (darin-moz)]

Stack trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 16384 (LWP 4678)]
0x40767c8c in NSSRWLock_LockRead () from /usr/local/fritz/firefox/libnss3.so
(gdb) bt
#0 0x40767c8c in NSSRWLock_LockRead ()
  from /usr/local/fritz/firefox/libnss3.so
#1 0x407458f7 in SECMOD_GetReadLock ()
  from /usr/local/fritz/firefox/libnss3.so
#2 0x407555b8 in PK11_GetAllTokens () from /usr/local/fritz/firefox/libnss3.so
#3 0x407557bf in PK11_GetBestSlotMultiple ()
  from /usr/local/fritz/firefox/libnss3.so
#4 0x4075598d in PK11_GetBestSlot () from /usr/local/fritz/firefox/libnss3.so
#5 0x407441b9 in PK11_CreateDigestContext ()
  from /usr/local/fritz/firefox/libnss3.so
#6 0x40737eaa in CERT_DecodeCRLDistributionPoints ()
  from /usr/local/fritz/firefox/libnss3.so
#7 0x40738159 in HASH_Create () from /usr/local/fritz/firefox/libnss3.so
#8 0x08676160 in nsXPTCVariant::Init ()
#9 0x4012f60b in XPTC_InvokeByIndex ()
  from /usr/local/fritz/firefox/libxpcom_core.so
#10 0x080a14f8 in nsTHashtable<nsBaseHashtableET<nsDepCharHashKey,
nsAutoPtr<nsINIParser::INIValue> > >::~nsTHashtable ()
#11 0x080a6c74 in nsTHashtable<nsBaseHashtableET<nsDepCharHashKey,
nsAutoPtr<nsINIParser::INIValue> > >::~nsTHashtable ()
#12 0x4004f90d in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#13 0x4005863b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#14 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#15 0x40053e5b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#16 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#17 0x40053e5b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#18 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#19 0x40053e5b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#20 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#21 0x40053e5b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#22 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#23 0x40053e5b in js_Interpret () from /usr/local/fritz/firefox/libmozjs.so
#24 0x4004f9be in js_Invoke () from /usr/local/fritz/firefox/libmozjs.so
#25 0x0809d314 in nsTHashtable<nsBaseHashtableET<nsDepCharHashKey,
nsAutoPtr<nsINIParser::INIValue> > >::~nsTHashtable ()
#26 0x0809b4d4 in nsTHashtable<nsBaseHashtableET<nsDepCharHashKey,
nsAutoPtr<nsINIParser::INIValue> > >::~nsTHashtable ()
#27 0x4012f730 in XPTC_InvokeByIndex ()
  from /usr/local/fritz/firefox/libxpcom_core.so
#28 0x40111efa in NS_CreateServicesFromCategory ()
  from /usr/local/fritz/firefox/libxpcom_core.so
#29 0x400e4e28 in NS_InitXPCOM3_P ()
  from /usr/local/fritz/firefox/libxpcom_core.so

[Kai Engert (kaie)]

Darin: We saw yesterday, that EnsureNSSInitialized does not return a possible failure, and therefore PSM services might get loaded, even if there is a problem with NSS databases.

Did you verify the call to NSS_InitReadWrite in nsNSSComponent::InitializeNSS succeeds?

[Darin-moz (darin-moz)]

> Did you verify the call to NSS_InitReadWrite in nsNSSComponent::InitializeNSS
> succeeds?

Nope, I have not. That would be a good thing to test when I have a chance.

[Gavin Sharp (gavin-sharp)]

*** Bug 366528 has been marked as a duplicate of this bug. ***

[Jruderman (jruderman)]

On Mac, this happens to several out of every thousand Firefox instances (see bug 366528). I think that makes it a topcrash.

[RobertSayre (sayrer)]

Created attachment 256217
JS stack from shutdown crash

[Tony Chang (tony-ponderer)]

It sounds like this happens if NSS hits shutdown before nsUrlClassifierDBService (anti-phishing thread). During xpcom-shutdown, nsUrlClassifierDBService processes pending events, which may include using nsCryptoHash.

I think the fix is to turn all pending events into NOOPs during nsUrlClassifierDBService shutdown. I'll see if I can repro and get a patch.

[cement_head (andorjkiss)]

Binary package hint: firefox

don't know...bug report was automatically generated

ProblemType: Crash
Architecture: i386
Date: Wed Mar 14 19:39:34 2007
DistroRelease: Ubuntu 7.04
ExecutablePath: /usr/lib/firefox/firefox-bin
Package: firefox 2.0.0.2+1-0ubuntu1
PackageArchitecture: i386
ProcCmdline: /usr/lib/firefox/firefox-bin
ProcCwd: /home/andor
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
 LANG=en_US.UTF-8
Signal: 11
SourcePackage: firefox
StacktraceTop:
 __kernel_vsyscall ()
 raise () from /lib/tls/i686/cmov/libpthread.so.0
 ?? ()
 ?? ()
 ?? ()
Uname: Linux oskar3 2.6.20-10-generic #2 SMP Mon Mar 12 00:02:49 UTC 2007 i686 GNU/Linux
UserGroups: adm admin audio cdrom dialout dip fax floppy fuse gdm lpadmin netdev plugdev powerdev root scanner slocate tape users video

[Hilario J. Montoliu (hjmf) (hmontoliu)]

Thank you for your report

[Hilario J. Montoliu (hjmf) (hmontoliu)]

Retrace done.

...
#4 0xb255bf6e in NSSRWLock_LockRead () from /usr/lib/libnss3.so
#5 0xb2534e4d in SECMOD_GetReadLock () from /usr/lib/libnss3.so
#6 0xb2547567 in PK11_GetAllTokens () from /usr/lib/libnss3.so
#7 0xb254789a in PK11_GetBestSlotMultiple () from /usr/lib/libnss3.so
#8 0xb254793c in PK11_GetBestSlot () from /usr/lib/libnss3.so
#9 0xb2534497 in PK11_CreateDigestContext () from /usr/lib/libnss3.so
#10 0xb252630e in md5_NewContext () from /usr/lib/libnss3.so
#11 0xb2526444 in HASH_Create () from /usr/lib/libnss3.so
#12 0xb25ef014 in nsCryptoHash::Init (this=0xb257c748, algorithm=2)
...

Probably a dup of bug #91737. Though need to evaluate with more time. Assigned to me.

[Hilario J. Montoliu (hjmf) (hmontoliu)]

Looks like it's similar to bug #91737 , but I'm assigning to mozilla-bugs for a further analysis.

[Tony Chang (tony-ponderer)]

Created attachment 260933
v1: no js callbacks during shutdown

I wasn't able to repro, but this patch cancels all calls into JS code once the urlclassifier shutdown has started. Maybe someone who can repro can check to see if this fixes the shutdown crash.

[Darin-moz (darin-moz)]

Comment on attachment 260933
v1: no js callbacks during shutdown

It seems like some of these methods should return errors since they are not going to generate callbacks. How about NS_ERROR_NOT_INITIALIZED? Or, NS_ERROR_UNEXPECTED?

-Darin

[Tony Chang (tony-ponderer)]

Created attachment 261031
v2: no js callbacks during shutdown

Updated to return NS_ERROR_NOT_INITIALIZED instead.

[RobertSayre (sayrer)]

We seem to be this bug pretty frequently on the tinderbox page, so I checked this in.

Checking in src/nsUrlClassifierDBService.cpp;
/cvsroot/mozilla/toolkit/components/url-classifier/src/nsUrlClassifierDBService.cpp,v <-- nsUrlClassifierDBService.cpp
new revision: 1.21; previous revision: 1.20
done

[Jeanmichel-reghem (jeanmichel-reghem)]

(In reply to comment #16)

Do you know if this fix could be solve bug 376709 ?

[Jruderman (jruderman)]

I hit this crash three times last night (Mac trunk debug), despite my build being up-to-date. (That's three crashes out of about 1360 shutdowns.)

[Hilario J. Montoliu (hjmf) (hmontoliu)]

Related to upstream's https://bugzilla.mozilla.org/show_bug.cgi?id=321024 ( Crash in nsCryptoHash during shutdown of nsUrlClassifierDBService [@ nsUrlClassifierDBService::Shutdown][@ NSSRWLock_LockRead]) as some of the upstream's duplicates of the marked one (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=366528) show the same stack than this one.

[Rcampbell-mozilla (rcampbell-mozilla)]

Happening much more frequently now on the Firefox tinderbox page, on Mac qm-xserve01. I've copied a crashreport to:

http://pastebin.mozilla.org/127724

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

Created attachment 271852
We should observe a much earlier topic, rev. 1

We should be observing a much earlier observer topic in any case. This will shutdown database activity at profile-before-change, as well as xpcom-shutdown-threads.

The comment in ::Shutdown says that it joins the thread, but I don't think it actually does. Unless I am misreading, we should definitely join that thread properly during shutdown. You aren't allowed to have threads persist past the xpcom-shutdown-threads observer topic.

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

I committed this patch a=robcee to get the tinderbox green, but I'd still like ex-post-facto review. Does this affect the branches at all, or just trunk?

[Adam Guthrie (ispiked)]

Looks like it affects branch, yes. NSSRWLock_LockRead is the #22 topcrash for Firefox 2.0.0.4. http://talkback-public.mozilla.org/reports/firefox/FF2004/FF2004-topcrashers.html

[Tony Chang (tony-ponderer)]

It's also the #7 crasher in Thunderbird:
http://talkback-public.mozilla.org/reports/thunderbird/TB2004/TB2004-topcrashers.html

[Dveditz (dveditz)]

What in the world is Thunderbird 2 doing with the URLClassifier? I can confirm that Thunderbird has a filehandle open to urlclassifier2.sqlite, (which is empty, of course).

filed bug 387840 to use --disable-safe-browsing in tbird

[Tony Chang (tony-ponderer)]

Thunderbird 2 has phishing detection. We want to keep it enabled.
http://mxr.mozilla.org/seamonkey/source/mail/components/phishing/

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

Created attachment 272029
Branch rollup patch

Here's a rollup patch for the 1.8 branch.

[Dveditz (dveditz)]

Comment on attachment 272029
Branch rollup patch

too late for 1.8.1.5

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

v/fixed according to http://crash-stats.mozilla.com/report/list?range_unit=weeks&branch=1.9&range_value=2&signature=NSSRWLock_LockRead

[Dveditz (dveditz)]

Comment on attachment 272029
Branch rollup patch

approved for 1.8.1.7, a=dveditz for release-drivers

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

Fixed on MOZILLA_1_8_BRANCH for what is now I think 1.8.1.8

[Pavel Penaz (pavel-penaz)]

I believe the string "fixed1.8.1.8" should be moved from "URL" to "Keywords" in the bug status summary.

[Tchung-mozilla (tchung-mozilla)]

Can someone provide a STR or testcase? Trying to verify on branch, but need steps.

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

Because this is a thread race there really aren't steps to reproduce. The best way to verify would be to check whether the crashes disappeared in talkback.

[Dveditz (dveditz)]

This patch did not appear to help the NSSRWLock_LockRead crash. According to talkback that's hovering around 1.4% of all crashes in both FF2.0.0.7 and FF2.0.0.8

[Hskupin (hskupin)]

*** Bug 376709 has been marked as a duplicate of this bug. ***

[Alexander Sack (asac)]

fixed as a topcrash upstream in 2.0.0.8