10ec:8821 [Gigabyte GB-BXi7-4470R] Kernel panic in rtl8821ae

[David Coles]

After updating the system to linux-image-3.13.0-30-generic (trusty-updates), the encounters a kernel panic during boot in the rtl8821ae module. This prevents the system from booting.

This appears to be an issue relating to the mPCI WiFi+BT module (Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter) included with the system.

A workaround is to boot using the older linux-image-3.13.0-24-generic (trusty).

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-30-generic 3.13.0-30.55
ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: dcoles 2324 F.... pulseaudio
 /dev/snd/pcmC1D0p: dcoles 2324 F...m pulseaudio
 /dev/snd/controlC0: dcoles 2324 F.... pulseaudio
CurrentDesktop: GNOME
Date: Sun Jul 13 20:35:13 2014
HibernationDevice: RESUME=UUID=489d6aeb-add6-485d-8056-de392a8591b7
InstallationDate: Installed on 2014-03-14 (120 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140310)
MachineType: GIGABYTE M4HM87P-00
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-24-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro quiet splash
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-24-generic N/A
 linux-backports-modules-3.13.0-24-generic N/A
 linux-firmware 1.127.4
SourcePackage: linux
StagingDrivers: rtl8821ae
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/06/2014
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F3
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: M4HM87P-00
dmi.board.vendor: GIGABYTE
dmi.board.version: 1.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF3:bd02/06/2014:svnGIGABYTE:pnM4HM87P-00:pvr1.x:rvnGIGABYTE:rnM4HM87P-00:rvr1.x:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: M4HM87P-00
dmi.product.version: 1.x
dmi.sys.vendor: GIGABYTE

[David Coles (dcoles)]

Attached screenshot of kernel panic.

[David Coles (dcoles)]

This looks quite similar to bug 1327741.

[Brad Figg (brad-figg)]

This change was made by a bot.

[David Coles (dcoles)]

Hi Christopher,

Looks like that did the trick. I can successfully boot the trusty-updates kernel:

$ uname -a
Linux dcoles-brix 3.13.0-30-generic #55-Ubuntu SMP Fri Jul 4 21:40:53 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date
F5
06/23/2014

Little tricky to do the BIOS update since the image is now too big for a FDD image. Best bet seems to be putting a FreeDOS image on a USB stick, copying over the BIOS updater files and running it off that.

Thanks for your assistance.

[David Coles (dcoles)]

Hi Christopher,

I"m reopening this ticket as it appears that the updating the bios did not resolve the issue. As soon as wireless is enabled in Network Manager the system kernel panics with the same rtl8821ae_rx_query_desc+0x1d5/0xa50 RIP.

Please let me know I can assist in any way.

[penalvch (penalvch)]

David Coles, could you please test the latest upstream kernel available from the very top line at the top of the page (not the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-3.16-rc6

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description.

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

[David Coles (dcoles)]

Thanks for your assistance. Just tested with latest upstream kernel and can successfully use WiFi without a Kernel Panic.

linux-image-3.16.0-031600rc6-generic_3.16.0-031600rc6.201407210035_amd64.deb

$uname -a
Linux dcoles-brix 3.16.0-031600rc6-generic #201407210035 SMP Mon Jul 21 04:36:11 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[penalvch (penalvch)]

David Coles, the next step is to fully reverse commit bisect the kernel in order to identify the fix commit. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection#How_do_I_reverse_bisect_the_upstream_kernel.3F ?

[TJ (tj)]

The latest kernel 3.16 driver doesn't Panic but it still has driver failures that render the driver and device unable to scan and find networks.

No need for a bisect since this driver is in staging, and is a work-in-progress. For Trusty 3.13 kernel it may be necessary to disable building the module since there is nothing we can do downstream to make it work.

[ 13.738475] rtl8821ae: module is from the staging directory, the quality is unknown, you have been warned.

-- v3.16 kernel --
[ 14.019892] WARNING: CPU: 0 PID: 321 at /home/apw/COD/linux/net/wireless/reg.c:1555 wiphy_apply_custom_regulatory+0xbc/0xe0 [cfg80211]()
[ 14.019894] wiphy should have REGULATORY_CUSTOM_REG
[ 14.019896] Modules linked in: asus_nb_wmi(+) intel_soc_dts_thermal(-) intel_powerclamp asus_wmi sparse_keymap coretemp rtl8821ae(C+) kvm mac80211 crct10dif_pclmul btusb crc32_pclmul ghash_clmulni_intel bluetooth uvcvideo cryptd cfg80211 videobuf2_vmalloc joydev videobuf2_memops videobuf2_core snd_soc_rt5640 v4l2_common snd_hda_intel(+) snd_soc_rl6231 serio_raw snd_hda_controller videodev snd_soc_core media snd_hda_codec snd_compress snd_pcm_dmaengine rtsx_pci_ms snd_seq_midi snd_seq_midi_event snd_rawmidi snd_hwdep 6lowpan_iphc lpc_ich(+) memstick snd_pcm snd_seq wmi i915(+) snd_seq_device snd_timer snd i2c_hid hid soundcore dw_dmac video dw_dmac_core drm_kms_helper drm i2c_designware_platform i2c_algo_bit i2c_designware_core parport_pc spi_pxa2xx_platform pwm_lpss ppdev mac_hid lp 8250_dw mei_txe parport mei snd_soc_sst_acpi iosf_mbi rtsx_pci_sdmmc psmouse r8169 mii ahci rtsx_pci libahci sdhci_acpi sdhci
[ 14.019972] CPU: 0 PID: 321 Comm: systemd-udevd Tainted: G C 3.16.0-031600-generic #201408031935
[ 14.019975] Hardware name: ASUSTeK COMPUTER INC. X551MA/X551MA, BIOS X551MA.402 02/08/2014
[ 14.019978] 0000000000000613 ffff8800b1633a18 ffffffff82786525 0000000000003f8a
[ 14.019984] ffff8800b1633a68 ffff8800b1633a58 ffffffff8207207c ffffffffc0a9bfb8
[ 14.019988] ffffffffc0a8d1e0 ffff8800aff00260 ffffffffc0a48ab0 ffff8800aff01840
[ 14.019993] Call Trace:
[ 14.020003] [<ffffffff82786525>] dump_stack+0x46/0x58
[ 14.020010] [<ffffffff8207207c>] warn_slowpath_common+0x8c/0xc0
[ 14.020031] [<ffffffffc0a48ab0>] ? rtl_regd_init+0x310/0x310 [rtl8821ae]
[ 14.020036] [<ffffffff82072166>] warn_slowpath_fmt+0x46/0x50
[ 14.020053] [<ffffffffc078327c>] wiphy_apply_custom_regulatory+0xbc/0xe0 [cfg80211]
[ 14.020067] [<ffffffffc0a48ab0>] ? rtl_regd_init+0x310/0x310 [rtl8821ae]
[ 14.020080] [<ffffffffc0a488f2>] rtl_regd_init+0x152/0x310 [rtl8821ae]
[ 14.020095] [<ffffffffc0a43c81>] rtl_init_core+0x41/0x170 [rtl8821ae]
[ 14.020110] [<ffffffffc0a57cd9>] rtl_pci_probe+0x289/0x910 [rtl8821ae]
[ 14.020116] [<ffffffff823d4c0e>] local_pci_probe+0x4e/0xa0
[ 14.020121] [<ffffffff823d5d25>] ? pci_match_device+0xe5/0x110
[ 14.020126] [<ffffffff823d5e49>] __pci_device_probe+0xb9/0xf0
[ 14.020131] [<ffffffff823d5eba>] pci_device_probe+0x3a/0x60
[ 14.020137] [<ffffffff824c4091>] really_probe+0x91/0x380
[ 14.020141] [<ffffffff824c4507>] driver_probe_device+0x47/0xa0
[ 14.020146] [<ffffffff824c460b>] __driver_attach+0xab/0x...

[penalvch (penalvch)]

TJ, thank you for your comment. Unfortunately, this is scoped to the original reporter David Coles, not you, your hardware, or your problem. Hence, please do not adjust the Bug Title, Tags, or Status of this report. If you are having a problem in Ubuntu, so your hardware and problem may be tracked, could you please file a new report with Ubuntu by executing the following in a terminal while booted into the default Ubuntu kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

[TJ (tj)]

On 08/08/14 11:00, Christopher M. Penalver wrote:
> TJ, thank you for your comment. Unfortunately, this is scoped to the original reporter David Coles, not you...

Christopher. I was triaging the bug as it affects other users, it is not specific to the original reporters motherboard. I worked on this in detail yesterday supporting another user with the identical
failure on Asus X551MA.

As a kernel hacker I investigated the bug in detail and diagnosed the cause. My additions to this report are in light of my findings.

RIP [<ffffffffa042ffe5>] rtl8821ae_rx_query_desc+0x1d5/0xa50 [rtl8821ae]

No changes were introduced in the rtl8821ae module between 3.13.0-24 and 3.13.0-30. The only changes were in mac80211, which rtl8821ae depends on (along with cfg80211):

# check rtl8821ae
$ gitlog Ubuntu-3.13.0-24.47..Ubuntu-3.13.0-30.55 -- drivers/staging/rtl8821ae
# check mac80211
$ gitlog Ubuntu-3.13.0-24.47..Ubuntu-3.13.0-30.55 -- net/mac80211
7049ad3 Mon May 19 18:45:30 2014 +0100 Michael Braun mac80211: fix WPA with VLAN on AP side with ps-sta again
5d31275 Mon May 19 18:45:30 2014 +0100 Johannes Berg mac80211: fix suspend vs. authentication race
56f2ea4 Mon May 19 18:45:29 2014 +0100 Johannes Berg mac80211: fix potential use-after-free
22bf70f Tue Apr 15 15:27:46 2014 +0100 Johannes Berg mac80211: add length check in ieee80211_is_robust_mgmt_frame()
# check mac80211
$ gitlog Ubuntu-3.13.0-24.47..Ubuntu-3.13.0-30.55 -- net/wireless/
$

The faulting location is in function rx_query_desc() at offset 0x1d5.

$ objdump -d /lib/modules/3.13.0-30-generic/kernel/drivers/staging/rtl8821ae/rtl8821ae.ko

0000000000033e40 <rtl8821ae_rx_query_desc>:

Faulting instruction is at 0x33e40 + 0x1d5 = 0x34015

Now I examine the debug-symbols of the module with:

$ gdb -d drivers/staging/rtl8821ae -d drivers/staging/rtl8821ae/rtl8821ae /usr/lib/debug/modules/3.13.0-30-generic/kernel/drivers/staging/rtl8821ae/rtl8821ae.dbgsym.ko

(gdb) info line rtl8821ae_rx_query_desc
Line 539 of "/build/buildd/linux-3.13.0/drivers/staging/rtl8821ae/rtl8821ae/trx.c" starts at address 0x33e40 <rtl8821ae_rx_query_desc>
    and ends at 0x33e65 <rtl8821ae_rx_query_desc+37>.
(gdb) x/i 0x34015
    0x34015 <rtl8821ae_rx_query_desc+469>: movzwl (%rdi),%esi
(gdb) disas rtl8821ae_rx_query_desc
...
    0x0000000000033ffe <+446>: je 0x34641 <rtl8821ae_rx_query_desc+2049>
    0x0000000000034004 <+452>: cmpl $0x18,0x68(%rdx)
    0x0000000000034008 <+456>: jbe 0x34268 <rtl8821ae_rx_query_desc+1064>
    0x000000000003400e <+462>: mov 0xd8(%rdx),%rdi /* hdr->frame_control */
    0x0000000000034015 <+469>: movzwl (%rdi),%esi /* FAULT %rdi invalid */
    0x0000000000034018 <+472>: mov %esi,%ecx
    0x000000000003401a <+474>: and $0xfc,%cx
    0x000000000003401f <+479>: cmp $0xa0,%cx
    0x0000000000034024 <+484>: je 0x34068 <rtl8821ae_rx_query_desc+552>
...
(gdb) info line *0x34015
Line 2194 of "/build/buildd/linux-3.13.0/include/linux/ieee80211.h" starts at address 0x34015 <rtl8821ae_rx_query_desc+469>
    and ends at 0x34018 <rtl8821ae_rx_query_desc+472>.

---- include/linux/ieee80211.h -----
/**
  * _ieee80211_is_robust_mgmt_frame - ch...

[penalvch (penalvch)]

TJ, regarding your comments:
>"The latest kernel 3.16 driver doesn't Panic but it still has driver failures that render the driver and device unable to scan and find networks."

As noted in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341275/comments/11 what you are finding with your/someone else's hardware is in direct contradiction with this report.

>"No need for a bisect since this driver is in staging, and is a work-in-progress."

The fact it's in staging has nothing to do with if a bisect is required or not.

>"Christopher. I was triaging the bug as it affects other users,"

Ok, but creating a frivolous new bug report and then marking this a duplicate of it isn't helping at all. This will be unduplicated and you are free to mark your bug report a duplicate of this one.

>"...it is not specific to the original reporters motherboard."

Nobody said anything about how a motherboard has anything to do with this issue.

>"I worked on this in detail yesterday supporting another user with the identical failure on Asus X551MA."

Exactly my point, another user, not the original reporter of this report. It's great that you are helping others in the Ubuntu Community address their issue. However, let us keep this report focused on this original reporter, his problem, and the facts surrounding it.

>"As a kernel hacker I investigated the bug in detail and diagnosed the cause. My additions to this report are in light of my findings."

Ok. Upon review of the mainline kernel commits your proposed patch change has already been implemented upstream 5 months ago:
Author: John W. Linville <email address hidden> 2014-03-13 11:53:52
Committer: John W. Linville <email address hidden> 2014-03-13 13:05:32
Parent: e46215fe678a9271c4eb98645187ef048d04e15f (Revert "Revert "Staging: rtl8812ae: remove modules field of rate_control_ops"")
Child: 3ead0d2e220ea7ced14027336bb168bafa01b7af (wlan-ng: fixup staging driver for removal of ieee80211_dsss_chan_to_freq)
Branches: master, remotes/origin/master
Follows: v3.14-rc2
Precedes: v3.15-rc1

    rtl8821ae: fixup staging driver for revised ieee80211_is_robust_mgmt_frame

    Commit d8ca16db6bb2 ("mac80211: add length check in
    ieee80211_is_robust_mgmt_frame()") changed that API to take an skb,
    and added "_ieee80211_is_robust_mgmt_frame" as a direct replacement
    for the older API. This is the same fix that was applied to the other
    rtlwifi drivers in that commit.

    Cc: Johannes Berg <email address hidden>
    Signed-off-by: John W. Linville <email address hidden>

------------------ drivers/staging/rtl8821ae/rtl8821ae/trx.c ------------------
index 75ae438..963b55f 100644
@@ -616,7 +616,7 @@ bool rtl8821ae_rx_query_desc(struct ieee80211_hw *hw,
     return false;
   }

- if ((ieee80211_is_robust_mgmt_frame(hdr)) &&
+ if ((_ieee80211_is_robust_mgmt_frame(hdr)) &&
    (ieee80211_has_protected(hdr->frame_control)))
    rx_status->flag &= ~RX_FLAG_DECRYPTED;
   else

So, the next step is for David Coles to test to this commit to see if this would fix his issue.

David Coles, could you please build up to this commit and advise if you don't have the panic following https://wiki.ubu...

[TJ (tj)]

This bug has now been fixed via bug #1354469

[penalvch (penalvch)]

Not Fix Committed as outlined in https://wiki.ubuntu.com/Bugs/Bug%20statuses .