14e4:165f [Dell PowerEdge R520] BCM95720 Network disconnects and resets

[David Peall]

The server is constantly going on and offline here are messages and hardware Details:
[ 4.394703] tg3 0000:02:00.0: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:0d:d4:aa
[ 4.394713] tg3 0000:02:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[ 4.394720] tg3 0000:02:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[ 4.394726] tg3 0000:02:00.0: eth0: dma_rwctrl[00000001] dma_mask[64-bit]

The dump to dmesg:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1152580/+attachment/3939814/+files/log

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-38-generic 3.2.0-38.61
ProcVersionSignature: Ubuntu 3.2.0-38.61-generic 3.2.37
Uname: Linux 3.2.0-38-generic x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Mar 8 13:26 seq
 crw-rw---T 1 root audio 116, 33 Mar 8 13:26 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Fri Mar 8 14:21:45 2013
HibernationDevice: RESUME=UUID=f9a32355-70b4-462b-a01e-2665b81b2ec0
InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
MachineType: Dell Inc. PowerEdge R520
MarkForUpload: True
PciMultimedia:

ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-38-generic root=UUID=0c0140bd-05eb-4e5c-9596-1cf4dee88ae4 ro
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-38-generic N/A
 linux-backports-modules-3.2.0-38-generic N/A
 linux-firmware 1.79.1
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
StagingDrivers: mei
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/21/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.3.5
dmi.board.name: 051XDX
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.3.5:bd08/21/2012:svnDellInc.:pnPowerEdgeR520:pvr:rvnDellInc.:rn051XDX:rvrA00:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R520
dmi.sys.vendor: Dell Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.9 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.9-rc1-raring/

[David Peall (dkpeall)]

I've updated the bug, It seems to be resolved in the latests kernel.

[David Peall (dkpeall)]

Just wanted to find out what happens now, we have 8 of these machines we would like to put into production, is there any chance we can raise the priority?

[David Peall (dkpeall)]

I've installed the latest kernel for precise on the server, I see the changelog for version linux-image-3.2.0-39-generic includes the following:

 * tg3: Avoid null pointer dereference in tg3_interrupt in netconsole mode
    - LP: #1131218
 * tg3: Fix crc errors on jumbo frame receive

Can anyone confirm that this is the fix for the reported bug?

[David Peall (dkpeall)]

To answer my own question, this update does not fix the issue.

[Joris S'heeren (jsheeren)]

We have the same issue with one of our servers, a Dell R720.

Ubuntu 12.04.2 LTS
kernel 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Hardware:
Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe, firmware 7.4.8
[ 1.481061] tg3 0000:01:00.1: eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d4:ae:52:89:64:36
[ 1.481074] tg3 0000:01:00.1: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[ 1.481083] tg3 0000:01:00.1: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[ 1.481091] tg3 0000:01:00.1: eth1: dma_rwctrl[00000001] dma_mask[64-bit]

dmesg dump:
 [ 14.994691] NETDEV WATCHDOG: eth1 (tg3): transmit queue 3 timed out
[ 14.994930] tg3 0000:01:00.1: eth1: transmit timed out, resetting
[ 3.890604] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[ 16.251575] tg3 0000:01:00.1: eth1: 0x00000010: 0xd91d000c, 0x00000000, 0xd91e000c, 0x00000000
[ 16.251862] tg3 0000:01:00.1: eth1: 0x00000020: 0xd91f000c, 0x00000000, 0x00000000, 0x1f5b1028
[ 16.252151] tg3 0000:01:00.1: eth1: 0x00000030: 0xdc800000, 0x00000048, 0x00000000, 0x0000020e
[ 16.252730] tg3 0000:01:00.1: eth1: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
[ 16.252730] tg3 0000:01:00.1: eth1: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
[ 16.253026] tg3 0000:01:00.1: eth1: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
[ 16.253308] tg3 0000:01:00.1: eth1: 0x00000070: 0x000710b0, 0xffd4b73d, 0x0001421c, 0x00000000
[ 16.253604] tg3 0000:01:00.1: eth1: 0x00000080: 0x00000001, 0xc00000d0, 0x00000000, 0x00000001
[ 16.253886] tg3 0000:01:00.1: eth1: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x000000ca
[ 16.254180] tg3 0000:01:00.1: eth1: 0x000000a0: 0x8010ac11, 0x00000004, 0x00001004, 0x00020010
[ 16.254454] tg3 0000:01:00.1: eth1: 0x000000b0: 0x10008d81, 0x0010242e, 0x0004cc22, 0x10120040
[ 16.254724] tg3 0000:01:00.1: eth1: 0x000000d0: 0x0000001f, 0x00000006, 0x00000000, 0x00000000
[ 16.255004] tg3 0000:01:00.1: eth1: 0x000000f0: 0x00000000, 0x05720000, 0x00000000, 0x00000000
[ 16.255293] tg3 0000:01:00.1: eth1: 0x00000100: 0x13c10001, 0x00000000, 0x00018000, 0x000e7030
[ 16.255582] tg3 0000:01:00.1: eth1: 0x00000110: 0x00002000, 0x000031c0, 0x000000a0, 0x00000000
[ 16.255870] tg3 0000:01:00.1: eth1: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[ 16.256160] tg3 0000:01:00.1: eth1: 0x00000140: 0x52896436, 0x0000d4ae, 0x00000000, 0x00000000
[ 16.256449] tg3 0000:01:00.1: eth1: 0x00000150: 0x16010004, 0x00000000, 0x0007811b, 0x00000001
[ 16.256738] tg3 0000:01:00.1: eth1: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
[ 16.257027] tg3 0000:01:00.1: eth1: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[ 16.257317] tg3 0000:01:00.1: eth1: 0x00000200: 0x00000000, 0x03000000, 0x00000000, 0x02000000
[ 16.257607] tg3 0000:01:00.1: eth1: 0x00000210: 0x00000000, 0x01000000, 0x00000000, 0x01000000
[ 16.257896] tg3 0000:01:00.1: eth1: 0x00000220: 0x00000000, 0x02000000, 0x00000000, 0x00000000
[ 16.258189] tg3 0000:01:00.1: eth1: 0x0000...

[Joris S'heeren (jsheeren)]

installed kernel 3.9.0-030900-generic #201304291257 SMP Mon Apr 29 16:58:15 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

still the same issue

[Dmitry Sherman (dmitry-b)]

Same here, system Dell 720xd, BCM5720, tg3 drive, Ubuntu 12.10.
Using bonding of 4 ports, one port constantly flapping.
Dells hardware has latest firmwares.

[Nithin Nayak Sujir (nithinsujir)]

Please try this upstream patch.

commit 9bc297ea0622bb2a6b3abfa2fa84f0a3b86ef8c8
Author: Nithin Sujir <email address hidden>
Date: Mon Jun 3 09:19:34 2013 +0000

    tg3: Add read dma workaround for 5720

    Commit 091f0ea30074bc43f9250961b3247af713024bc6 "tg3: Add New 5719 Read
    DMA workaround" added a workaround for TX DMA stall on the 5719. This
    workaround needs to be applied to the 5720 as well.

[Bryan Quigley (bryanquigley)]

The patch identified is built-in to the following kernels:
Ubuntu Raring (13.04)
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13.3-raring/

For 3.5-based kernels (12.10 Quantal):
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.7.15-quantal/

For 3.2 (12.04 LTS):
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.47-precise/

If you are having this issue please test them and see if it fixed the issue.

[Dmitry Sherman (dmitry-b)]

Thanks, are those patches stable? Can I upgrade production environment ?

Dmitry Sherman
Interhost Networks Ltd
Email: <email address hidden>
web: http://www.interhost.co.il/
fb: http://www.facebook.com/InterhostIL
t: ([phone] f: ([phone]

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Bryan Quigley
Sent: Wednesday, June 26, 2013 5:43 PM
To: Dmitry Sherman
Subject: [Bug 1152580] Re: BCM95720 Network disconnects and resets with new Dell R520 server

The patch identified is built-in to the following kernels:
Ubuntu Raring (13.04)
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13.3-raring/

For 3.5-based kernels (12.10 Quantal):
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.7.15-quantal/

For 3.2 (12.04 LTS):
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.47-precise/

If you are having this issue please test them and see if it fixed the issue.

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1152580

Title:
  BCM95720 Network disconnects and resets with new Dell R520 server

Status in The Dell PowerEdge project:
  New
Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  The server is constantly going on and offline here are messages and
  details

  Linux xxxxxxxxxxxxxxxxxx 3.2.0-38-generic #61-Ubuntu SMP Tue Feb 19
  12:18:21 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=12.04
  DISTRIB_CODENAME=precise
  DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

  Hardware Details:
  [ 4.394703] tg3 0000:02:00.0: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:0d:d4:aa
  [ 4.394713] tg3 0000:02:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
  [ 4.394720] tg3 0000:02:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
  [ 4.394726] tg3 0000:02:00.0: eth0: dma_rwctrl[00000001] dma_mask[64-bit]

  The dump to dmesg:
  [ 1836.538377] tg3 0000:02:00.0: eth0: transmit timed out, resetting
  [ 1837.804044] tg3 0000:02:00.0: eth0: 0x00000000: 0x165f14e4, 0x00100406, 0x02000000, 0x00800010
  [ 1837.812650] tg3 0000:02:00.0: eth0: 0x00000010: 0xd90a000c, 0x00000000, 0xd90b000c, 0x00000000
  [ 1837.821272] tg3 0000:02:00.0: eth0: 0x00000020: 0xd90c000c, 0x00000000, 0x00000000, 0x04f61028
  [ 1837.830323] tg3 0000:02:00.0: eth0: 0x00000030: 0xdd000000, 0x00000048, 0x00000000, 0x0000010f
  [ 1837.839735] tg3 0000:02:00.0: eth0: 0x00000040: 0x00000000, 0x03000000, 0xc8035001, 0x64002008
  [ 1837.849390] tg3 0000:02:00.0: eth0: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
  [ 1837.859425] tg3 0000:02:00.0: eth0: 0x00000060: 0x00000000, 0x00000000, 0xf0000298, 0x00380081
  [ 1837.869992] tg3 0000:02:00.0: eth0: 0x00000070: 0x000710b0, 0xff401a00, 0x0001421c, 0x00000000
  [ 1837.880661] tg3 0000:02:00.0: eth0: 0x00000080: 0x00000001, 0x800000f8, 0x00000000, 0x00000a64
  [ 1837.891604] tg3 0000:02:00.0: eth0: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x00000232
  [ 1837.903145] tg3 0000:02:00.0: eth0: 0...

[Kent Baxley (kentb)]

We had tried some of the following to reproduce the issue, but, were unsuccessful:

1) Generate a large file (40GB) and copy it from some systems with 5720 adapters to another system via scp. There were 4 adapters in each machine and we tested each adapter individually. The systems were a PowerEdge R820 and an R620.

2) Next, we ran netperf against each machine for 12 hours and were not able to reproduce any TX hangs.

The systems were running 13.04 server with the 64-bit 3.5.0-28 kernels.

[Kent Baxley (kentb)]

correction on the kernels...it was 3.8.0-25. sorry.

[empty (empty-chai)]

Test each adapter separately, are good, but both configure two adapters, the second adapter will error。

The systems were running 12.04.2 server with the 64-bit 3.5.0-23 and 3.5.0-34 kernels.

[Kent Baxley (kentb)]

@empty, what type of testing did you do to reproduce? When you mean both adapters configured, were they in a bond or anything like that or was it just both adapters up and running at the same time with no special configuration?

[empty (empty-chai)]

I'm sorry, yesterday did not complete the test , fully tested today, only eth1 adapter problems,Replace the adapter chipset with intel, now normal

[Jeff Armstrong (n-launchpa7-f)]

I noted similar behaviour immediately following a wheezy upgrade on one in five Dell servers with bonded eths.
I found that Debian package: firmware-linux-nonfree was NOT installed, and this mentions tg3 firmware.

I tried:

sudo apt-get install firmware-linux-nonfree
sudo reboot

The error now appears to be resolved on the one host that was emitting warnings every minute.
Hope this helps.

Regards,
Jeff

[penalvch (penalvch)]

Cut from Bug Description.

[penalvch (penalvch)]

David Peall, as per your https://launchpadlibrarian.net/133336555/BootDmesg.txt :
[ 0.026147] ------------[ cut here ]------------
[ 0.026153] WARNING: at /build/buildd/linux-3.2.0/drivers/iommu/intr_remapping.c:558 enable_intr_remapping+0x77/0x1ab()
[ 0.026157] Hardware name: PowerEdge R520
[ 0.026160] Your BIOS is broken and requested that x2apic be disabled
[ 0.026160] This will leave your machine vulnerable to irq-injection attacks
[ 0.026161] Use 'intremap=no_x2apic_optout' to override BIOS request
...
[ 2.724831] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored

Hence, as per www.dell.com/support/drivers/us/en/19/Product/poweredge-m520 an update is available for your BIOS (2.0.23). During your maintenance window, if you update to this following https://help.ubuntu.com/community/BiosUpdate , does it change anything? If it doesn't, could you please both specify what happened, and just provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

For more on BIOS updates and linux, please see https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette .

Thank you for your understanding.

[wonko (bernard-0)]

Hi,

we have a similar, or the same, problem with Dell PE machines, having the Broadcom chip mentioned above. We have this problem on generation 12 systems, across different models (R420, R620), with several combinations of bios firmwares, lifecycle firmwares, etc... We see this on several versions of the linux kernel, ranging from 3.2.x up tot 3.11, with several versions of the tg3 driver, including a manually compiled latest version (3.133d) loaded in a 3.11. The latest machine, where we can reproduce the problem has Ubuntu Precise installed, but we also see this behavior on Debian machines. We run Xen on it, running HVM hosts on it. Storage is handled over iSCSI (and it is the iSCSI interface we can trigger this bug on in a reproducible way, while we have the impression it also happens on other interfaces, but there we don't have a solid case where we have e reproducible setup.

All this info actually points into the direction of the tg3 driver and hardware below it not handling certain datastreams or data patterns correctly, and finally crashing the system. It seems unrelated to the version of kernel running, xen-version running, amount of VM's running, firmwares and revisions running, etc...

We have been trying to pinpoint this for over a year now, being unable to actually create a scenario where we could reproduce this. As of this week, we finally found a specific setup where we could trigger the error.

The error is triggered by running a certain VM, and inside that VM, importing a mysqldump in a running mysql on that VM. The VM has it's traffic on an iSCSI volume, so this effectually generates a datastream over the eth1 interface of the machine. Within a short amount of time, the system will crash in 2 steps. We first see a timeout on the tg3 driver, and finally, the machine freezes and reboots.

Kernel boot:

[ 0.000000] Linux version 3.11.0-19-generic (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #33~precise1-Ubuntu SMP Wed Mar 12 21:16:27 UTC 2014 (Ubuntu 3.11.0-19.33~precise1-generic 3.11.10.5)
[ 0.000000] Command line: placeholder root=UUID=bbc71780-90bf-4647-b579-e48d5d8c2bce ro vga=0x317 crashkernel=348M-2G:64M,2G-:128M intremap=off
...

The intremap=off was a guess it had something to do with the interrupts being handled wrongly, but this didn't matter.

Our tg3 hardware/driver init:

[ 64.596574] tg3.c:v3.133d (August 07, 2013)
...
[ 64.609739] pcieport 0000:00:1c.4: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address f0:1f:af:e9:33:da
[ 64.609905] pcieport 0000:00:1c.4: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[ 64.610071] pcieport 0000:00:1c.4: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[ 64.610244] pcieport 0000:00:1c.4: eth0: dma_rwctrl[00000001] dma_mask[64-bit]
...
[ 64.621595] pcieport 0000:00:1c.4: eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address f0:1f:af:e9:33:db
[ 64.621602] pcieport 0000:00:1c.4: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[ 64.621605] pcieport 0000:00:1c.4: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[ 64.62...

[penalvch (penalvch)]

wonko, thank you for your comment. So your hardware and problem may be tracked, could you please file a new report with Ubuntu by executing the following in a terminal while booted into the default Ubuntu kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

[Frank Louwers (frank-openminds)]

[Llinking to other bug report, for future reference/googlers/...] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331513

[Angel Genchev (i-liketowin)]

Well i have the same problem with the tg3 driver @ Broadcom 5719 NIC and HP ProLiant DL385 G8 server. Under Ubuntu 14.04, kernel 3.13.x. I get the transmit timed out, resetting and the long table with hex codes in DMESG.
But it worked well with 3.8 kernels on Ubuntu 12.x
I'm going to try changing the clock source (TSC selected by the kernel). I'll try forcing PIT or HPET.

[penalvch (penalvch)]

Angel Genchev, thank you for your comment. So your hardware and problem may be tracked, could you please file a new report with Ubuntu by executing the following in a terminal while booted into the default Ubuntu kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
https://wiki.ubuntu.com/Kernel/Policies/DuplicateBugs
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

As well, please do not announce in this report you created a new bug report.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs