BUG: soft lockup - CPU#0 stuck for 61s!

[Brian J. Murrell]

I've been experiencing lockups on the last few Karmic kernels. My less-than-informed opinion is that it's in NFS4 somewhere, given the stack traces and the soft lockup message at the end of traces:

[ 5436.357006] BUG: soft lockup - CPU#1 stuck for 61s! [10.75.22.3-mana:24899]
[ 5436.360017] Modules linked in: xt_multiport binfmt_misc nvidia(P) agpgart bridge stp bnep tun des_generic cbc autofs4 video output rpcsec_gss_krb5 nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc nf_conntrack_ipv6 xt_hl ipt_LOG xt_limit ipt_REJECT xt_tcpudp xt_state ipt_addrtype ip6table_filter ip6_tables nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables it87 hwmon_vid eeprom lp snd_hda_codec_analog msp3400 tuner_simple tuner_types tea5767 snd_emu10k1_synth snd_hda_intel tuner snd_emux_synth snd_hda_codec bttv snd_seq_virmidi snd_seq_midi_emul ir_common snd_seq_dummy snd_emu10k1 i2c_algo_bit snd_seq_oss snd_ac97_codec v4l2_common videodev snd_seq_midi v4l1_compat snd_bt87x snd_rawmidi ac97_bus videobuf_dma_sg snd_seq_midi_event snd_pcm_oss snd_mixer_oss snd_pcm videobuf_core emu10k1_gp snd_seq snd_util_mem btcx_risc ns558 gameport snd_seq_device snd_timer snd_hwdep tveeprom k8temp asus_atk0110 btusb i2c_nforce2 pcspkr snd soundcore snd_page_alloc ppdev parport_pc parport psmouse serio_raw ohci1394 ieee1394 forcedeth floppy raid10 raid456 raid6_pq async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
[ 5436.360017]
[ 5436.360017] Pid: 24899, comm: 10.75.22.3-mana Tainted: P (2.6.31-2-generic #16-Ubuntu) System Product Name
[ 5436.360017] EIP: 0060:[<fa095c85>] EFLAGS: 00000246 CPU: 1
[ 5436.360017] EIP is at nfs4_state_manager+0x55/0x1d0 [nfs]
[ 5436.360017] EAX: 00000000 EBX: f426d0c8 ECX: 00000000 EDX: 00000040
[ 5436.360017] ESI: f426d000 EDI: 00000000 EBP: f7189fac ESP: f7189f94
[ 5436.360017] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 5436.360017] CR0: 8005003b CR2: b8064000 CR3: 00811000 CR4: 00000690
[ 5436.360017] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 5436.360017] DR6: ffff0ff0 DR7: 00000400
[ 5436.360017] Call Trace:
[ 5436.360017] [<fa095e00>] ? nfs4_run_state_manager+0x0/0x30 [nfs]
[ 5436.360017] [<fa095e00>] ? nfs4_run_state_manager+0x0/0x30 [nfs]
[ 5436.360017] [<fa095e17>] nfs4_run_state_manager+0x17/0x30 [nfs]
[ 5436.360017] [<c0156364>] kthread+0x74/0x80
[ 5436.360017] [<c01562f0>] ? kthread+0x0/0x80
[ 5436.360017] [<c0103f57>] kernel_thread_helper+0x7/0x10

I will attach the output of a sysrq-t while it was locked up.

$ uname -a
Linux pc 2.6.31-2-generic #16-Ubuntu SMP Mon Jul 6 20:38:51 UTC 2009 i686 GNU/Linux

$ cat /proc/version_signature
Ubuntu 2.6.31-2.16-generic

$ sudo lspci -vnvn
[sudo] password for brian:
00:00.0 RAM memory [0500]: nVidia Corporation C51 Host Bridge [10de:02f0] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Capabilities: [44] HyperTransport: Slave or Primary Interface
  Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
  Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
  Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
  Link Control 1: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
  Link Config 1: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
  Revision ID: 1.03
  Link Frequency 0: 1.0GHz
  Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
  Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+ 500MHz+ 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
  Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA- UIDRD-
  Link Frequency 1: 800MHz
  Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
  Link Frequency Capability 1: 200MHz+ 300MHz+ 400MHz+ 500MHz+ 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
  Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
  Prefetchable memory behind bridge Upper: 00-00
  Bus Number: 00
 Capabilities: [e0] HyperTransport: MSI Mapping Enable+ Fixed-
  Mapping Address Base: 00000000fee00000

00:00.1 RAM memory [0500]: nVidia Corporation C51 Memory Controller 0 [10de:02fa] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-

00:00.2 RAM memory [0500]: nVidia Corporation C51 Memory Controller 1 [10de:02fe] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:00.3 RAM memory [0500]: nVidia Corporation C51 Memory Controller 5 [10de:02f8] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:00.4 RAM memory [0500]: nVidia Corporation C51 Memory Controller 4 [10de:02f9] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0

00:00.5 RAM memory [0500]: nVidia Corporation C51 Host Bridge [10de:02ff] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Capabilities: [44] #00 [00fe]
 Capabilities: [fc] #00 [0000]

00:00.6 RAM memory [0500]: nVidia Corporation C51 Memory Controller 3 [10de:027f] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:00.7 RAM memory [0500]: nVidia Corporation C51 Memory Controller 2 [10de:027e] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:02.0 PCI bridge [0604]: nVidia Corporation C51 PCI Express Bridge [10de:02fc] (rev a1)
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 32 bytes
 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
 I/O behind bridge: 0000a000-0000afff
 Memory behind bridge: fd800000-fd8fffff
 Prefetchable memory behind bridge: 00000000fd700000-00000000fd7fffff
 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
 BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
  PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
 Capabilities: [40] Subsystem: nVidia Corporation Device [10de:0000]
 Capabilities: [48] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+
  Address: 00000000fee0300c Data: 4149
 Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
  Mapping Address Base: 00000000fee00000
 Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
  DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
   ExtTag- RBE- FLReset-
  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
   RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
   MaxPayload 128 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
  LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
   ClockPM- Suprise- LLActRep+ BwNot-
  LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
   ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
   Slot # 0, PowerLimit 0.000000; Interlock- NoCompl-
  SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
   Control: AttnInd Off, PwrInd On, Power- Interlock-
  SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
   Changed: MRL- PresDet- LinkState-
  RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
  RootCap: CRSVisible-
  RootSta: PME ReqID 0000, PMEStatus- PMEPending-
 Capabilities: [100] Virtual Channel <?>
 Kernel driver in use: pcieport-driver
 Kernel modules: shpchp

00:03.0 PCI bridge [0604]: nVidia Corporation C51 PCI Express Bridge [10de:02fd] (rev a1)
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 32 bytes
 Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
 I/O behind bridge: 00008000-00008fff
 Memory behind bridge: fde00000-fdefffff
 Prefetchable memory behind bridge: 00000000fdd00000-00000000fddfffff
 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
 BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
  PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
 Capabilities: [40] Subsystem: nVidia Corporation Device [10de:0000]
 Capabilities: [48] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+
  Address: 00000000fee0300c Data: 4151
 Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
  Mapping Address Base: 00000000fee00000
 Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
  DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
   ExtTag- RBE- FLReset-
  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
   RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
   MaxPayload 128 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
  LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
   ClockPM- Suprise- LLActRep+ BwNot-
  LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
   ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
   Slot # 0, PowerLimit 0.000000; Interlock- NoCompl-
  SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
   Control: AttnInd Off, PwrInd On, Power- Interlock-
  SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
   Changed: MRL- PresDet- LinkState-
  RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
  RootCap: CRSVisible-
  RootSta: PME ReqID 0000, PMEStatus- PMEPending-
 Capabilities: [100] Virtual Channel <?>
 Kernel driver in use: pcieport-driver
 Kernel modules: shpchp

00:04.0 PCI bridge [0604]: nVidia Corporation C51 PCI Express Bridge [10de:02fb] (rev a1)
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 32 bytes
 Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
 I/O behind bridge: 0000b000-0000bfff
 Memory behind bridge: fdc00000-fdcfffff
 Prefetchable memory behind bridge: 00000000fd900000-00000000fd9fffff
 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
 BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
  PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
 Capabilities: [40] Subsystem: nVidia Corporation Device [10de:0000]
 Capabilities: [48] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+
  Address: 00000000fee0300c Data: 4159
 Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
  Mapping Address Base: 00000000fee00000
 Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
  DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
   ExtTag- RBE- FLReset-
  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
   RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
   MaxPayload 128 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
  LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
   ClockPM- Suprise- LLActRep+ BwNot-
  LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
   ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
   Slot # 0, PowerLimit 0.000000; Interlock- NoCompl-
  SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
   Control: AttnInd Off, PwrInd On, Power- Interlock-
  SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
   Changed: MRL- PresDet- LinkState-
  RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
  RootCap: CRSVisible-
  RootSta: PME ReqID 0000, PMEStatus- PMEPending-
 Capabilities: [100] Virtual Channel <?>
 Kernel driver in use: pcieport-driver
 Kernel modules: shpchp

00:05.0 VGA compatible controller [0300]: nVidia Corporation C51PV [GeForce 6150] [10de:0240] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81cd]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Interrupt: pin A routed to IRQ 16
 Region 0: Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
 Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
 Region 3: Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
 [virtual] Expansion ROM at c0000000 [disabled] [size=128K]
 Capabilities: [48] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
  Address: 0000000000000000 Data: 0000
 Kernel driver in use: nvidia
 Kernel modules: nvidia, nvidiafb

00:09.0 RAM memory [0500]: nVidia Corporation MCP51 Host Bridge [10de:0270] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Capabilities: [44] HyperTransport: Slave or Primary Interface
  Command: BaseUnitID=9 UnitCnt=15 MastHost- DefDir- DUL-
  Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
  Link Config 0: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
  Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+ <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
  Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
  Revision ID: 1.03
  Link Frequency 0: 800MHz
  Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
  Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+ 500MHz+ 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
  Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA- UIDRD-
  Link Frequency 1: 200MHz
  Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
  Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz- 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
  Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
  Prefetchable memory behind bridge Upper: 00-00
  Bus Number: 00
 Capabilities: [e0] HyperTransport: MSI Mapping Enable- Fixed-
  Mapping Address Base: 00000000fee00000

00:0a.0 ISA bridge [0601]: nVidia Corporation MCP51 LPC Bridge [10de:0260] (rev a3)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0

00:0a.1 SMBus [0c05]: nVidia Corporation MCP51 SMBus [10de:0264] (rev a3)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Interrupt: pin A routed to IRQ 255
 Region 4: I/O ports at 4c00 [size=64]
 Region 5: I/O ports at 4c40 [size=64]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: nForce2_smbus
 Kernel modules: i2c-nforce2

00:0a.2 RAM memory [0500]: nVidia Corporation MCP51 Memory Controller 0 [10de:0272] (rev a3)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:0b.0 USB Controller [0c03]: nVidia Corporation MCP51 USB Controller [10de:026d] (rev a3) (prog-if 10)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (750ns min, 250ns max)
 Interrupt: pin A routed to IRQ 20
 Region 0: Memory at fe02f000 (32-bit, non-prefetchable) [size=4K]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: ohci_hcd

00:0b.1 USB Controller [0c03]: nVidia Corporation MCP51 USB Controller [10de:026e] (rev a3) (prog-if 20)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (750ns min, 250ns max)
 Interrupt: pin B routed to IRQ 21
 Region 0: Memory at fe02e000 (32-bit, non-prefetchable) [size=256]
 Capabilities: [44] Debug port: BAR=1 offset=0098
 Capabilities: [80] Power Management version 2
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME+
 Kernel driver in use: ehci_hcd

00:0d.0 IDE interface [0101]: nVidia Corporation MCP51 IDE [10de:0265] (rev a1) (prog-if 8a [Master SecP PriP])
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (750ns min, 250ns max)
 Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [disabled] [size=8]
 Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [disabled] [size=1]
 Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [disabled] [size=8]
 Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [disabled] [size=1]
 Region 4: I/O ports at f400 [size=16]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: pata_amd

00:0e.0 IDE interface [0101]: nVidia Corporation MCP51 Serial ATA Controller [10de:0266] (rev a1) (prog-if 85 [Master SecO PriO])
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (750ns min, 250ns max)
 Interrupt: pin A routed to IRQ 23
 Region 0: I/O ports at 09f0 [size=8]
 Region 1: I/O ports at 0bf0 [size=4]
 Region 2: I/O ports at 0970 [size=8]
 Region 3: I/O ports at 0b70 [size=4]
 Region 4: I/O ports at e000 [size=16]
 Region 5: Memory at fe02d000 (32-bit, non-prefetchable) [size=4K]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [b0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
  Address: 0000000000000000 Data: 0000
 Capabilities: [cc] HyperTransport: MSI Mapping Enable+ Fixed+
 Kernel driver in use: sata_nv

00:0f.0 IDE interface [0101]: nVidia Corporation MCP51 Serial ATA Controller [10de:0267] (rev a1) (prog-if 85 [Master SecO PriO])
 Subsystem: ASUSTeK Computer Inc. Device [1043:81c0]
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (750ns min, 250ns max)
 Interrupt: pin A routed to IRQ 22
 Region 0: I/O ports at 09e0 [size=8]
 Region 1: I/O ports at 0be0 [size=4]
 Region 2: I/O ports at 0960 [size=8]
 Region 3: I/O ports at 0b60 [size=4]
 Region 4: I/O ports at cc00 [size=16]
 Region 5: Memory at fe02c000 (32-bit, non-prefetchable) [size=4K]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [b0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
  Address: 0000000000000000 Data: 0000
 Capabilities: [cc] HyperTransport: MSI Mapping Enable+ Fixed+
 Kernel driver in use: sata_nv

00:10.0 PCI bridge [0604]: nVidia Corporation MCP51 PCI Bridge [10de:026f] (rev a2) (prog-if 01)
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Bus: primary=00, secondary=04, subordinate=04, sec-latency=128
 I/O behind bridge: 00009000-00009fff
 Memory behind bridge: fdb00000-fdbfffff
 Prefetchable memory behind bridge: fda00000-fdafffff
 Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR+
 BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
  PriDiscTmr- SecDiscTmr+ DiscTmrStat+ DiscTmrSERREn-
 Capabilities: [b8] Subsystem: Gammagraphx, Inc. (or missing ID) Device [0000:0000]
 Capabilities: [8c] HyperTransport: MSI Mapping Enable+ Fixed-
  Mapping Address Base: 00000000fee00000

00:10.1 Audio device [0403]: nVidia Corporation MCP51 High Definition Audio [10de:026c] (rev a2)
 Subsystem: ASUSTeK Computer Inc. Device [1043:81cb]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (500ns min, 1250ns max)
 Interrupt: pin B routed to IRQ 22
 Region 0: Memory at fe024000 (32-bit, non-prefetchable) [size=16K]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0 Enable-
  Address: 0000000000000000 Data: 0000
  Masking: 00000000 Pending: 00000000
 Capabilities: [6c] HyperTransport: MSI Mapping Enable+ Fixed+
 Kernel driver in use: HDA Intel
 Kernel modules: snd-hda-intel

00:14.0 Bridge [0680]: nVidia Corporation MCP51 Ethernet Controller [10de:0269] (rev a3)
 Subsystem: ASUSTeK Computer Inc. Device [1043:816a]
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0 (250ns min, 5000ns max)
 Interrupt: pin A routed to IRQ 23
 Region 0: Memory at fe02b000 (32-bit, non-prefetchable) [size=4K]
 Region 1: I/O ports at c800 [size=8]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
 Kernel driver in use: forcedeth
 Kernel modules: forcedeth

00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration [1022:1100]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Capabilities: [80] HyperTransport: Host or Secondary Interface
  !!! Possibly incomplete decoding
  Command: WarmRst+ DblEnd-
  Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0
  Link Config: MLWI=16bit MLWO=16bit LWI=16bit LWO=16bit
  Revision ID: 1.02

00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map [1022:1101]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller [1022:1102]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control [1022:1103]
 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Capabilities: [f0] Secure device <?>
 Kernel driver in use: k8temp
 Kernel modules: k8temp

04:05.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) [104c:8023] (prog-if 10)
 Subsystem: ASUSTeK Computer Inc. Device [1043:808b]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 32 (500ns min, 1000ns max), Cache Line Size: 32 bytes
 Interrupt: pin A routed to IRQ 19
 Region 0: Memory at fdbff000 (32-bit, non-prefetchable) [size=2K]
 Region 1: Memory at fdbf8000 (32-bit, non-prefetchable) [size=16K]
 Capabilities: [44] Power Management version 2
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME+
 Kernel driver in use: ohci1394
 Kernel modules: firewire-ohci, ohci1394

04:08.0 Multimedia video controller [0400]: Brooktree Corporation Bt878 Video Capture [109e:036e] (rev 11)
 Subsystem: ATI Technologies Inc Device [1002:0001]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 32 (4000ns min, 10000ns max)
 Interrupt: pin A routed to IRQ 16
 Region 0: Memory at fdaff000 (32-bit, prefetchable) [size=4K]
 Capabilities: [44] Vital Product Data <?>
 Capabilities: [4c] Power Management version 2
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: bttv
 Kernel modules: bttv

04:08.1 Multimedia controller [0480]: Brooktree Corporation Bt878 Audio Capture [109e:0878] (rev 11)
 Subsystem: ATI Technologies Inc Device [1002:0001]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 32 (1000ns min, 63750ns max)
 Interrupt: pin A routed to IRQ 16
 Region 0: Memory at fdafe000 (32-bit, prefetchable) [size=4K]
 Capabilities: [44] Vital Product Data <?>
 Capabilities: [4c] Power Management version 2
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: Bt87x
 Kernel modules: snd-bt87x

04:09.0 Multimedia audio controller [0401]: Creative Labs SB Live! EMU10k1 [1102:0002] (rev 07)
 Subsystem: Creative Labs Device [1102:8027]
 Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 32 (500ns min, 5000ns max)
 Interrupt: pin A routed to IRQ 17
 Region 0: I/O ports at 9c00 [size=32]
 Capabilities: [dc] Power Management version 1
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: EMU10K1_Audigy
 Kernel modules: snd-emu10k1

04:09.1 Input device controller [0980]: Creative Labs SB Live! Game Port [1102:7002] (rev 07)
 Subsystem: Creative Labs Device [1102:0020]
 Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 32
 Region 0: I/O ports at 9800 [size=8]
 Capabilities: [dc] Power Management version 1
  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Kernel driver in use: Emu10k1_gameport
 Kernel modules: emu10k1-gp

[Brian J. Murrell (brian-interlinx)]

Just got another one. Will attach the dmesg including stack traces.

[Brian J. Murrell (brian-interlinx)]

This thread seems interesting with regard to this problem: http://marc.info/?l=linux-kernel&m=124677884816794&w=2

[Bardur Arantsson (bardur-arantsson)]

I'm also experiencing this, but have been unable to obtain the dmesg output since my machine locks up so badly that I can't get any commands to run at all (I can't even find it in /var/log after restarting).

I do remember from the dmesg output that my system is also complaining about nfs4_state_manager. This is with an up-to-date karmic (kernel 2.6.31-3-generic on a 64-bit quad core CPU).

[Brian J. Murrell (brian-interlinx)]

Not even a peep from the Ubuntu developers on this bug? Has this even been triaged yet? This is making the Karmic kernel *completely* useless for me.

[Bardur Arantsson (bardur-arantsson)]

Still experiencing this with

Linux version 2.6.31-6-generic (buildd@yellow) (gcc version 4.4.1 (Ubuntu 4.4.1-1ubuntu3) ) #25-Ubuntu SMP Fri Aug 14 16:28:28 UTC 2009

so it hasn't been fixed upstream yet.

I'll attach my kern.log as well.

[Brian J. Murrell (brian-interlinx)]

Have I filed this bug in the wrong place? Why, after all this time has
there not been a single peep from a Ubuntu/Canonical bug triage person?

Is Karmic going to get released with this known defect? Surely wouldn't
be the first DOA release Ubuntu have done lately. Sadly.

[Bardur Arantsson (bardur-arantsson)]

This can probably be closed. At least, I'm not seeing this any more in the final Karmic (for the record: it went away somewhere around the final or penultimate karmic alpha).

[Jeremy Foshee (jeremyfoshee)]

Hi Brian,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 397058

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Brian J. Murrell (brian-interlinx)]

You can probably just close this. I have not seen it in a while and am running 2.6.31-20-generic now.

[Jeremy Foshee (jeremyfoshee)]

Brian,
    Thanks for the follow up. Marking this Fix Released.

~JFo