RTL8192SE DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0

[Da Xue]

Two onboard NIC R8169. One PCIE RTL8192SE. If the module is loaded for rtl8192se, I will get SW-IOMMU errors after an hour or two. All of the network interface cards would stop working. Kernel log is filled with SW-IOMMU errors.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-19-generic 3.2.0-19.31
ProcVersionSignature: Ubuntu 3.2.0-19.30-generic 3.2.11
Uname: Linux 3.2.0-19-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 1.94.1-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: dx 12595 F.... pulseaudio
 /dev/snd/pcmC0D0c: dx 12595 F...m pulseaudio
 /dev/snd/pcmC0D7p: dx 12595 F...m pulseaudio
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xfe600000 irq 49'
   Mixer name : 'Intel CougarPoint HDMI'
   Components : 'HDA:10ec0892,10193162,00100302 HDA:80862805,80862805,00100000'
   Controls : 52
   Simple ctrls : 21
Date: Wed Mar 21 16:31:18 2012
HibernationDevice: RESUME=UUID=c8f4b4fb-3a2e-42f8-a29d-4266e7ee55e8
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Beta amd64 (20120229)
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
 Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
 Bus 001 Device 003: ID 099a:2515 Zippy Technology Corp.
MachineType: ECS H67H2-M
ProcEnviron:
 TERM=linux
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-19-generic root=UUID=5a235b5f-702c-4f2a-8020-7b6a5a1a5dff ro
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-19-generic N/A
 linux-backports-modules-3.2.0-19-generic N/A
 linux-firmware 1.71
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/07/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 4.6.4
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: H67H2-M
dmi.board.vendor: ECS
dmi.board.version: 1.0
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: ECS
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr4.6.4:bd04/07/2011:svnECS:pnH67H2-M:pvr1.0:rvnECS:rnH67H2-M:rvr1.0:cvnECS:ct3:cvr1.0:
dmi.product.name: H67H2-M
dmi.product.version: 1.0
dmi.sys.vendor: ECS

[Da Xue (da-t)]

The SW-IOMMU errors only occur when rtl8192se module is loaded.

Mar 20 09:30:04 server kernel: [ 2892.930146] DMA: Out of SW-IOMMU space for 54 bytes at device 0000:05:00.0
Mar 20 09:30:04 server kernel: [ 2892.930153] r8169 0000:05:00.0: eth1: Failed to map TX DMA!
Mar 20 09:30:05 server kernel: [ 2893.970118] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.019589] DMA: Out of SW-IOMMU space for 91 bytes at device 0000:05:00.0
Mar 20 09:30:05 server kernel: [ 2894.019597] r8169 0000:05:00.0: eth1: Failed to map TX DMA!
Mar 20 09:30:05 server kernel: [ 2894.072448] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.161438] DMA: Out of SW-IOMMU space for 172 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.174824] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.277052] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.319623] DMA: Out of SW-IOMMU space for 91 bytes at device 0000:05:00.0
Mar 20 09:30:05 server kernel: [ 2894.319633] r8169 0000:05:00.0: eth1: Failed to map TX DMA!
Mar 20 09:30:05 server kernel: [ 2894.379557] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:05 server kernel: [ 2894.482047] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:06 server kernel: [ 2894.584356] DMA: Out of SW-IOMMU space for 178 bytes at device 0000:07:00.0
Mar 20 09:30:06 server kernel: [ 2894.603783] DMA: Out of SW-IOMMU space for 305 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.603790] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.605141] DMA: Out of SW-IOMMU space for 314 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.605147] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.607031] DMA: Out of SW-IOMMU space for 375 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.607037] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.614537] DMA: Out of SW-IOMMU space for 371 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.614545] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.626655] DMA: Out of SW-IOMMU space for 314 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.626661] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.628034] DMA: Out of SW-IOMMU space for 357 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.628040] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.637217] DMA: Out of SW-IOMMU space for 371 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.637223] r8169 0000:04:00.0: eth0: Failed to map TX DMA!
Mar 20 09:30:06 server kernel: [ 2894.644417] DMA: Out of SW-IOMMU space for 359 bytes at device 0000:04:00.0
Mar 20 09:30:06 server kernel: [ 2894.652100] DMA: Out of SW-IOMMU space for 357 bytes at device 0000:04:00....

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get dist-upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

[Joseph Salisbury (jsalisbury)]

This bug seems very similar to a bug fixed with upstream commit:
43f88d530ec656d9b556fb0d127b30757b1c3d35

[Joseph Salisbury (jsalisbury)]

Can you try booting with the following boot options:
iommu=off mem=4g

[Joseph Salisbury (jsalisbury)]

I have one additional request, would it also be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.3 kernel[1] (Not a kernel in the daily directory).

Thanks in advance.

[1] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-precise/

[Da Xue (da-t)]

I upgrade to the latest (3.2.0-19.31) and the network interface freeze continued to occur.
I installed a mainline 3.3.0-030300-generic and it did the same thing.

If I rmmod the rtl8192se module, all the other interfaces become active again.

I am currently testing the 3.2.0 kernel with both boot options and will report back the findings. Do you want me to test them separately also?

[Da Xue (da-t)]

it doesn't seem to be related to amount of traffic. more like 1-2 hours will cause the sw-iommu issue.

I used Iperf to transfer more than 10GB without freezing. I am using the device as an access point in master mode.

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get dist-upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

[Da Xue (da-t)]

kernel with iommu=off and mem=4g

this time it didn't bring the other two bridged network interfaces down but the wifi did stop working after slightly over an hour.

Mar 21 21:39:20 server kernel: [ 8.003649] rtl8192se 0000:07:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 21 21:39:20 server kernel: [ 8.003659] rtl8192se 0000:07:00.0: setting latency timer to 64
Mar 21 21:39:20 server kernel: [ 8.012438] rtl8192se: rtl8192ce: FW Power Save off (module option)
Mar 21 21:39:20 server kernel: [ 8.013006] rtl8192se: Driver for Realtek RTL8192SE/RTL8191SE
Mar 21 21:39:20 server kernel: [ 8.013007] Loading firmware rtlwifi/rtl8192sefw.bin
Mar 21 21:39:20 server kernel: [ 8.182259] ieee80211 phy0: Selected rate control algorithm 'rtl_rc'
Mar 21 21:47:02 server kernel: [ 7.608257] rtl8192se 0000:07:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 21 21:47:02 server kernel: [ 7.608265] rtl8192se 0000:07:00.0: setting latency timer to 64
Mar 21 21:47:02 server kernel: [ 7.619263] rtl8192se: rtl8192ce: FW Power Save off (module option)
Mar 21 21:47:02 server kernel: [ 7.621292] rtl8192se: Driver for Realtek RTL8192SE/RTL8191SE
Mar 21 21:47:02 server kernel: [ 7.621294] Loading firmware rtlwifi/rtl8192sefw.bin
Mar 21 21:47:02 server kernel: [ 7.783603] ieee80211 phy0: Selected rate control algorithm 'rtl_rc'
Mar 21 21:59:31 server kernel: [ 7.769093] rtl8192se 0000:07:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 21 21:59:31 server kernel: [ 7.769101] rtl8192se 0000:07:00.0: setting latency timer to 64
Mar 21 21:59:31 server kernel: [ 7.782120] rtl8192se: rtl8192ce: FW Power Save off (module option)
Mar 21 21:59:31 server kernel: [ 7.782563] rtl8192se: Driver for Realtek RTL8192SE/RTL8191SE
Mar 21 21:59:31 server kernel: [ 7.782564] Loading firmware rtlwifi/rtl8192sefw.bin
Mar 21 21:59:31 server kernel: [ 7.918741] ieee80211 phy0: Selected rate control algorithm 'rtl_rc'
Mar 21 22:02:22 server kernel: [ 7.885624] rtl8192se 0000:07:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 21 22:02:22 server kernel: [ 7.885632] rtl8192se 0000:07:00.0: setting latency timer to 64
Mar 21 22:02:22 server kernel: [ 7.899202] rtl8192se: rtl8192ce: FW Power Save off (module option)
Mar 21 22:02:22 server kernel: [ 7.899263] rtl8192se: Driver for Realtek RTL8192SE/RTL8191SE
Mar 21 22:02:22 server kernel: [ 7.899264] Loading firmware rtlwifi/rtl8192sefw.bin
Mar 21 22:02:22 server kernel: [ 8.331898] ieee80211 phy0: Selected rate control algorithm 'rtl_rc'
Mar 21 22:07:57 server kernel: [ 7.633145] rtl8192se 0000:07:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 21 22:07:57 server kernel: [ 7.633154] rtl8192se 0000:07:00.0: setting latency timer to 64
Mar 21 22:07:57 server kernel: [ 7.645813] rtl8192se: rtl8192ce: FW Power Save off (module option)
Mar 21 22:07:57 server kernel: [ 7.646783] rtl8192se: Driver for Realtek RTL8192SE/RTL8191SE
Mar 21 22:07:57 server kernel: [ 7.646785] Loading firmware rtlwifi/rtl8192sefw.bin
Mar 21 22:07:57 server kernel: [ 7.786693] ieee80211 phy0: Selected rate control algorithm 'rtl_rc'
Mar 21 22:08:27 server kernel: [ 48.7...

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report at bugzilla.kernel.org [1]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

If you are comfortable with opening a bug upstream, It would be great if you can report back the upstream bug number in this bug report. That will allow us to link this bug to the upstream report.

[1] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Da Xue (da-t)]

https://bugzilla.kernel.org/show_bug.cgi?id=42976

[Da Xue (da-t)]

Upstream fixed it relatively quickly. The patch is here: https://bugzilla.kernel.org/attachment.cgi?id=72688. I tested the patch against the 3.2 kernel and problem solved.

[Joseph Salisbury (jsalisbury)]

Thanks for the udpate, Da Xue.

It looks like the patch may not have completely solved the issue according to the upstream bug. Is that correct?

[Da Xue (da-t)]

It solved one issue with the IOMMU problems. The second issue I'm having may or may not be related. After I applied the patch I left iperf on to transfer 100GB over wireless to stress test. When I woke up the next morning, it exhibited the same problem of all interfaces not functioning (but restored when rmmod rtl8192se) but there were no IOMMU errors or any errors really besides cryptic messages in kern.log:
Mar 23 10:28:56 server kernel: [23319.482507] rtlwifi: &&&&&&&&&del entry 5
Mar 23 10:32:32 server kernel: [23535.368946] rtlwifi: &&&&&&&&&del entry 4
I've since been testing whether it's time or transfers that cause the new issue as the old issue would pop up an hour or so because it was beacon count that triggered it. So far, I've left the interface on and idle for several days and it hasn't crashed. I'm retesting transfer tonight and will get some results tomorrow as to what may be causing the second issue.

[Maarten Bezemer (veger)]

Thanks for taking the time to report this bug in the upstream bug tracking system this is a tremendous help. Launchpad has the ability to watch lots of upstream bug trackers and this can be done by following the procedure documented at https://wiki.ubuntu.com/Bugs/Watches. I've added the bug watch for this bug report.

[Joseph Salisbury (jsalisbury)]

It might be best to open a separate bug for the performance issue, so we can focus on each bug individually.

[Da Xue (da-t)]

I concur with Joseph. I haven't come across the second issue again after testing for the last few days. The wireless has been pretty stable. It may be safe to assume that it was a random glitch.

[Da Xue (da-t)]

Was this pushed to the Ubuntu repos? I'm still getting IOMMU errors from linux-image-3.2.0-22.

[Joseph Salisbury (jsalisbury)]

I searched LKML and I don't see Larry's patch submitted. I'll ping the upstream bug and see if Larry plans on submitting his patch upstream.

[Da Xue (daxue)]

Will the Canonical kernel team add this patch to the 3.2 kernel before 12.04 ships? Thanks.

[Joseph Salisbury (jsalisbury)]

Since it's not in the upstream kernels as of yet, probably not.

You may be wondering why the kernel team doesn't just apply the patch and fix it. The reason is that the kernel team is reluctant (not opposed) to apply any patch to a stable kernel that is not from upstream. Applying patches that don't come from upstream add greatly to the support of the kernel as other upstream patches may touch the same area as the non-upstream patch and may prevent them from applying cleanly.

[Da Xue (daxue)]

I assumed as such. I already scripted the module recompile so it's no biggy.

[Joseph Salisbury (jsalisbury)]

It looks like the patch to fix this bug has been added to v3.2-rc3:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-rc3-precise/

[Joseph Salisbury (jsalisbury)]

Typo,
The path has been merged into v3.4-rc3

[Luis Alvarado (luisalvarado)]

This also applies in 12.04.1 to the BCM4321 802.11b/g/n driver card pci@0000:03:00.0) and the RT2800 802.11n PCI driver card (pci@0000:03:02.0). Both of which are giving the same error.

[Luis Alvarado (luisalvarado)]

Any idea if this will be applied to the 3.2.x kernel of Ubuntu 12.04.x. Right now, the PC crashes/freezes everytime I do intensive network stuff like torrent downloading, http downloading or even watching videos in any site.

[Luis Alvarado (luisalvarado)]

This is still affecting me on 12.10 Beta2:

Oct 1 18:57:34 cyrex kernel: [21915.262347] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0
Oct 1 18:57:34 cyrex kernel: [21915.262443] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0
Oct 1 18:57:34 cyrex kernel: [21915.262755] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0
Oct 1 18:57:34 cyrex kernel: [21915.263042] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0
Oct 1 18:57:34 cyrex kernel: [21915.269121] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0
Oct 1 18:57:34 cyrex kernel: [21915.288175] DMA: Out of SW-IOMMU space for 3856 bytes at device 0000:03:02.0

[Matthew MacDonald (mvmacd)]

Hello I am affected by this with using RTL8192CE driver both using Ad-hoc network, and hostapd Infrastructure mode.

I have Ubuntu 12.10 and 3.5.0-21-generic kernel.

Does anybody know why it hasn't been fixed?

[WindPlume (etandrew)]

I also have this issue with Ubuntu 12.10 and kernel 3.5.0-25-generic x86_64

The driver causes this issue seems to be dmfe for Davicom ethernet card.

[JaSauders (jasauders)]

Just ran into this with the latest daily build of Ubuntu 13.04 64 bit beta.

[Julian B. (julian8x)]

Experiencing the same with my Atheros network card that uses atl1c on 13.04 64bit beta.

[penalvch (penalvch)]

Da Xue the upstream is available in Precise+ as per http://kernel.ubuntu.com/git?p=ubuntu%2Fubuntu-precise.git&a=search&h=HEAD&st=commit&s=rtlwifi%3A+Add+missing+DMA+buffer+unmapping+for+PCI+drivers . Hence, this bug report is being closed due to your last comment https://bugs.launchpad.net/ubuntu/+source/linux/+bug/961618/comments/18 regarding this being fixed with the patch. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. If you have any further issues, please file a new report via a terminal:
ubuntu-bug linux

Thank you again for taking the time to report this bug and helping to make Ubuntu better. Please submit any future bugs you may find.