eth0: Dumping tx registers

[GSMD]

Sorry for a lame description but it's the best I could figure out.
I'm running Gutsy i386 on a ASUS M2NPV-MX <GeForce 6150> board. It's been running w/o a notch until today, when I found integrated LAN interface non-functional and syslog flooded with messages like
--
Dec 7 07:50:32 s kernel: [313077.970000] eth0: Got tx_timeout. irq: 00000070
Dec 7 07:50:32 s kernel: [313077.970000] eth0: Ring at 2180000
Dec 7 07:50:32 s kernel: [313077.970000] eth0: Dumping tx registers
Dec 7 07:50:32 s kernel: [313077.970000] 0: 00000070 000000ff 00000003 00f903
ca 00000000 00000000 00000000 00000000
Dec 7 07:50:32 s kernel: [313077.970000] 20: 00000000 f0000000 00000000 000000
00 00000000 00000000 00000000 00000000
Dec 7 07:50:32 s kernel: [313077.970000] 40: 0420e20e 0000a455 00002e20 000000
00 00000000 00000000 00000000 00000000
Dec 7 07:50:32 s kernel: [313077.970000] 60: 00000000 00000000 00000000 0000ff
ff 0000ffff 0000ffff 0000ffff 00000000
Dec 7 07:50:32 s kernel: [313077.970000] 80: 003b0f3c 00000001 00000000 007f00
88 0000061c 00000001 00000000 00007f18
Dec 7 07:50:32 s kernel: [313077.970000] a0: 0014050f 00000016 5ff31800 00002b
b0 00000001 00000000 00000000 00000000
Dec 7 07:50:32 s kernel: [313077.970000] c0: 10000002 00000001 00000001 000000
01 00000001 00000001 00000001 00000001
Dec 7 07:50:32 s kernel: [313077.970000] e0: 00000001 00000001 00000001 000000
01 00000001 00000001 00000001 00000001
Dec 7 07:50:32 s kernel: [313077.970000] eth0: Dumping tx ring
Dec 7 07:50:32 s kernel: [313077.970000] 000: 00000000 1778e14a 2000006d // 00000000 2d9650da 20000040 // 00000000 2d9654da 20000040 // 00000000 2d9658da 20000040
Dec 7 07:50:32 s kernel: [313077.970000] 004: 00000000 2d965cda 20000040 // 00000000 24d2a0da 20000040 // 00000000 24d2a4da 20000040 // 00000000 24d2a8da 20000040
Dec 7 07:51:02 s kernel: [313107.970000] NETDEV WATCHDOG: eth0: transmit timed
out
Dec 7 07:51:02 s kernel: [313107.970000] eth0: Got tx_timeout. irq: 00000070
Dec 7 07:51:02 s kernel: [313107.970000] eth0: Ring at 2180000
Dec 7 07:51:02 s kernel: [313107.970000] eth0: Dumping tx registers
Dec 7 07:51:02 s kernel: [313107.970000] 0: 00000070 000000ff 00000003 000e03
ca 00000000 00000000 00000000 00000000
--
Just rebooted the system and it works fine again. I don't know what state it was left in, so it's probably a security vulnerability.

[Che Guevara (che-guevara-3)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately we can't fix it, because your description does not yet have enough information.

Please include the following additional information, if you have not already done so (pay attention to lspci's additional options), as required by the Ubuntu Kernel Team:
1. Please include the output of the command "uname -a" in your next response. It should be one, long line of text which includes the exact kernel version you're running, as well as the CPU architecture.
2. Please run the command "dmesg > dmesg.log" after a fresh boot and attach the resulting file "dmesg.log" to this bug report.
3. Please run the command "sudo lspci -vvnn > lspci-vvnn.log" and attach the resulting file "lspci-vvnn.log" to this bug report.

For your reference, the full description of procedures for kernel-related bug reports is available at [WWW] http://wiki.ubuntu.com/KernelTeamBugPolicies. Thanks in advance!

[GSMD (gsmdib)]

Thanks for your reply.
uname -a:
Linux host.name.skipped 2.6.22-14-server #1 SMP Sun Oct 14 23:34:23 GMT 2007 i686 GNU/Linux

The other two logs attached.

[Che Guevara (che-guevara-3)]

Thank you, changing package to linux-source-2.6.22 and setting status back to new

[Che Guevara (che-guevara-3)]

By the way can you check if there are any BIOS updates for your motherboard please?

[GSMD (gsmdib)]

Running the latest BIOS available.

[Leann Ogasawara (leannogasawara)]

Hi GSMD,

I'm just curious if this was something that just occurred once or have you been able to reproduce it consistently? Also, just for future reference, if you could attach your log files individually rather than as a tarball it would be much appreciated. Thanks!

[Leann Ogasawara (leannogasawara)]

Also, the Hardy Heron Alpha2 release will be coming out soon (around Dec 20). It will have an updated version of the kernel. It would be great if you could test with this new release and verify if this issue still exists. I'll be sure to update this report when Alpha2 is available. Thanks!

[GSMD (gsmdib)]

It occurred twice already, so basically it's reproducible.
I'm not sure I'll be able to update to Hardy ASAP, but as I do, I'll definitely report the results here.

[Cindy Ames (cma42)]

I'm having this problem too. Recently upgraded my thin client server from Dapper to Gutsy, interface worked fine in Dapper. Using 2.6.22-14-server #1 SMP Sun Oct 14 23:34:23 GMT 2007 i686 GNU/Linux. Interface uses forcedeth. This is the interface serving the thin clients' network, so all my thin clients screeched to a halt! Rebooting resolved the problem for now, but definitely need a real fix.

Dec 21 12:45:23 ltsp kernel: [141586.910000] NETDEV WATCHDOG: eth3: transmit timed out
Dec 21 12:45:23 ltsp kernel: [141586.910000] eth3: Got tx_timeout. irq: 00000032
Dec 21 12:45:23 ltsp kernel: [141586.910000] eth3: Ring at 1fec8000
Dec 21 12:45:23 ltsp kernel: [141586.910000] eth3: Dumping tx registers
Dec 21 12:45:23 ltsp kernel: [141586.910000] 0: 00002032 000000ff 00000003 00bd03ca 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 40: 0420e20e 0000a855 00002e20 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 60: 00000000 00000000 00000000 0000ffff 0000ffff 0000ffff 0000ffff 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 80: 003b0f3d 40000001 00000000 007f0028 0000061c 00000001 00000000 00007f8a
Dec 21 12:45:23 ltsp kernel: [141586.910000] a0: 0014050f 00000016 e5d1a000 00007a58 005e0001 00000100 ffffffff 0000ffff
Dec 21 12:45:23 ltsp kernel: [141586.910000] c0: 10000002 00000001 00000001 00000001 00000001 00000001 00000001 00000001
Dec 21 12:45:23 ltsp kernel: [141586.910000] e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
Dec 21 12:45:23 ltsp kernel: [141586.910000] 100: 1fec8800 1fec8000 007f00ff 00008000 00010032 00000000 00000022 1fec8c90
Dec 21 12:45:23 ltsp kernel: [141586.910000] 120: 1fec83a0 0abf2d00 a000ffd3 00000000 00000000 1fec8c9c 1fec83a0 0fe08000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 140: 00304120 80c02600 00000000 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 160: 00000000 00000000 00000000 00000000 00c00030 0000c000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 180: 00000016 00000008 0294796d 00008103 0000004a 00007c00 00000080 0000fd83
Dec 21 12:45:23 ltsp kernel: [141586.910000] 1a0: 00000016 00000008 0294796d 00008103 0000004a 00007c00 0000059e 0000fd83
Dec 21 12:45:23 ltsp kernel: [141586.910000] 1c0: 00000016 00000008 0294796d 00008103 0000004a 00007c00 00000597 0000fd83
Dec 21 12:45:23 ltsp kernel: [141586.910000] 1e0: 00000016 00000008 0294796d 00008103 0000004a 00007c00 00000080 0000fda3
Dec 21 12:45:23 ltsp kernel: [141586.910000] 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 21 12:45:23 ltsp kernel: [141586.910000] 260: 00000000 00000000 fe027001 00000100 00000011 000000a3 fe027011 000001a3
Dec 21 12:45:23 ltsp kernel: [141586.910000] 28...

[Cindy Ames (cma42)]

I'm wondering if this also has something to do with this, which I'm seeing repeatedly in /var/log/debug & /var/log/kern.log:

eth3: too many iterations (6) in nv_nic_irq.

I found a thread about that here: http://lists.debian.org/debian-amd64/2006/08/msg00274.html

[GSMD (gsmdib)]

What's incomplete about this bug?

[Leann Ogasawara (leannogasawara)]

Hardy Heron Alpha2 was recently released. It contains an updated version of the kernel. You can download and try the new Hardy Heron Alpha2 release from http://cdimage.ubuntu.com/releases/hardy/alpha-2/ . You should be able to then test the new kernel via the LiveCD. Please verify if this bug still exists or not and report back your results. Thanks!

[GSMD (gsmdib)]

Just had this issue once again.
Unfortunately, I can't install Hardy as VMware server doesn't work under 2.6.24 yet. Besides, LiveCD won't help either as I face this issue approx. once a month.

[Leann Ogasawara (leannogasawara)]

This report will remain open against the actively developed kernel and will be closed against linux-source-2.6.22. Thanks!

[Roy Olsen (royolsen)]

I'm seeing the same issue on Ubuntu 7.10. The problem is reproducable by generating a significant samba network load with both incoming and outgoing traffic.

* Using winrar on a client to uncompress a large archive on a samba share (source and destination both on the share). The network interface on the server usually goes down within a few minutes.

* Running bittorrent on a client machine, downloading to a samba share. Running at 10 Mpbs both incoming and outgoing with hundreds of active connecions seems to triggers the failure within the hour.

The kernel is 2.6.22-14-server, unfortunately I'm not sure what motherboard/chipset is on that server - I'll be able to check this later though.

I suppose I could try the Hardy Heron beta release and report back.

[Roy Olsen (royolsen)]

I upgraded to Hardy Heron beta yesterday, and have not been able to crash forcedeth after 20 hours of bashing the network. The too many iterations message still floods the log though.

[aggieml (michael-linder)]

Having the same problem, but I'm running Mepis 2.6.22-1-mepis-smp. I am only using Etch's stable repositories. Does anyone know if upgrading to the "testing" repositories would fix this problem?

[sebrock (sebrock)]

I also have this problem on a 7.10 server edition. Anything new in the matter?

Basically happens 2-3 times a month and usually when there is heavy load on the net.

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Michele Mangili (mangilimic)]

This bug report is being closed due to your last comment regarding this being fixed with an update. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status . Thank you again for taking the time to report this bug and helping to make Ubuntu better. Feel free to submit any future bugs you may find.

[Stefano Rivera (stefanor)]

This bug still seems to be alive and well in hardy (LTS).

I've just had it trigger on a machine with max_interrupt_work=10, as suggested by https://bugzilla.redhat.com/show_bug.cgi?id=179422 although that did seem to give us a few months reliable service before this event. Unfortunately it doesn't seem to be easy to trigger.

There are a pile of likely suspects on the kernel bugzilla.

[Michael van der Kolff (mvanderkolff)]

This bug has been fixed in the upstream sources - commit 8f955d7f042e4ac44891a400d5000928f8db9f58 in the torvalds/linux-2.6.git tree. The fix was committed on Mon, 27 Apr 2009 09:40:51 +0000

Is it possible to take the forcedeth.c from the current git tree? The driver is next to useless without it...

Cheers,

Michael

[Leann Ogasawara (leannogasawara)]

The git commit mentioned in the previous comment is already in the upcoming Karmic kernel. ISO CD images are available at http://cdimage.ubuntu.com/releases/karmic/ . Marking this Fix Released.

ogasawara@emiko:~/ubuntu-karmic$ git show 8f955d7f042e4ac44891a400d5000928f8db9f58
commit 8f955d7f042e4ac44891a400d5000928f8db9f58
Author: Ayaz Abdulla <email address hidden>
Date: Sat Apr 25 09:17:56 2009 +0000

    forcedeth: tx timeout fix

    This patch fixes the tx_timeout() to properly handle the clean up of the
    tx ring. It also sets the tx put pointer back to the correct position to
    be in sync with HW.

    Signed-off-by: Ayaz Abdulla <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>