WARNING: CPU: 0 PID: 1529 at /build/buildd/linux-3.11.0/kernel/softirq.c:159 local_bh_enable+0x60/0x90()

[Dirk Kraft]

After updating my system from raring to saucy it was crashing during boot. After some digging I realised that it is connected to the network interfaces. After removing the auto line for the two network interfaces in /etc/network/interfaces the system boots successfully.

If one then tries to bring up the network interface using the via_velocity driver (using ifup) the system crashes and the following message appears in syslog:

Sep 15 02:25:24 nas dhclient: Internet Systems Consortium DHCP Client 4.2.4
Sep 15 02:25:24 nas dhclient: Copyright 2004-2012 Internet Systems Consortium.
Sep 15 02:25:24 nas dhclient: All rights reserved.
Sep 15 02:25:24 nas dhclient: For info, please visit https://www.isc.org/software/dhcp/
Sep 15 02:25:24 nas dhclient:
Sep 15 02:25:25 nas kernel: [ 81.953639] Velocity is AUTO mode
Sep 15 02:25:25 nas dhclient: Listening on LPF/eth0/00:40:63:f5:15:76
Sep 15 02:25:25 nas dhclient: Sending on LPF/eth0/00:40:63:f5:15:76
Sep 15 02:25:25 nas dhclient: Sending on Socket/fallback
Sep 15 02:25:25 nas dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x4ff0e6ad)
Sep 15 02:25:26 nas kernel: [ 83.462216] eth0: Link auto-negotiation speed 100M bps full duplex
Sep 15 02:25:28 nas dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x4ff0e6ad)
Sep 15 02:25:29 nas kernel: [ 86.675203] ------------[ cut here ]------------
Sep 15 02:25:29 nas kernel: [ 86.675227] WARNING: CPU: 0 PID: 1529 at /build/buildd/linux-3.11.0/kernel/softirq.c:159 local_bh_enable+0x60/0x90()
Sep 15 02:25:29 nas kernel: [ 86.675231] Modules linked in: snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) snd_seq_midi(F) snd_seq_midi_event(F) via_cputemp snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) i2c_viapro psmouse(F) serio_raw(F) snd(F) soundcore(F) shpchp mac_hid hwmon_vid lp(F) parport(F) binfmt_misc(F) raid10(F) raid1(F) raid0(F) multipath(F) linear(F) dm_crypt(F) via_rng raid456(F) async_raid6_recov(F) async_memcpy(F) async_pq(F) async_xor(F) async_tx(F) xor(F) raid6_pq(F) pata_acpi padlock_sha padlock_aes pata_via via_velocity via_rhine mii(F) ahci(F) libahci(F) crc_ccitt(F)
Sep 15 02:25:29 nas kernel: [ 86.675315] CPU: 0 PID: 1529 Comm: SpiderOak Tainted: GF 3.11.0-7-generic #13-Ubuntu
Sep 15 02:25:29 nas kernel: [ 86.675320] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To be filled by O.E.M., BIOS 080014 12/18/2007
Sep 15 02:25:29 nas kernel: [ 86.675324] 00000000 00000000 ee803ce8 c16313c4 00000000 ee803d18 c105268e c1805af4
Sep 15 02:25:29 nas kernel: [ 86.675336] 00000000 000005f9 c1805dc8 0000009f c1056b50 c1056b50 c19566c0 efbafb80
Sep 15 02:25:29 nas kernel: [ 86.675347] 00000000 ee803d28 c1052752 00000009 00000000 ee803d30 c1056b50 ee803d4c
Sep 15 02:25:29 nas kernel: [ 86.675359] Call Trace:
Sep 15 02:25:29 nas kernel: [ 86.675371] [<c16313c4>] dump_stack+0x41/0x52
Sep 15 02:25:29 nas kernel: [ 86.675379] [<c105268e>] warn_slowpath_common+0x7e/0xa0
Sep 15 02:25:29 nas kernel: [ 86.675387] [<c1056b50>] ? local_bh_enable+0x60/0x90
Sep 15 02:25:29 nas kernel: [ 86.675394] [<c1056b50>] ? local_bh_enable+0x60/0x90
Sep 15 02:25:29 nas kernel: [ 86.675401] [<c1052752>] warn_slowpath_null+0x22/0x30
Sep 15 02:25:29 nas kernel: [ 86.675409] [<c1056b50>] local_bh_enable+0x60/0x90
Sep 15 02:25:29 nas kernel: [ 86.675421] [<c155f1ea>] dst_alloc+0x12a/0x140
Sep 15 02:25:29 nas kernel: [ 86.675431] [<c158189d>] rt_dst_alloc+0x4d/0x60
Sep 15 02:25:29 nas kernel: [ 86.675440] [<c1583fdc>] ip_route_input_noref+0x2bc/0xa30
Sep 15 02:25:29 nas kernel: [ 86.675448] [<c107c504>] ? resched_task+0x24/0x70
Sep 15 02:25:29 nas kernel: [ 86.675456] [<c107cfd8>] ? ttwu_do_wakeup+0x18/0x100
Sep 15 02:25:29 nas kernel: [ 86.675464] [<c1585b4b>] ip_rcv_finish+0xbb/0x340
Sep 15 02:25:29 nas kernel: [ 86.675471] [<c1586414>] ip_rcv+0x254/0x3c0
Sep 15 02:25:29 nas kernel: [ 86.675479] [<c1558217>] __netif_receive_skb_core+0x557/0x730
Sep 15 02:25:29 nas kernel: [ 86.675487] [<c1558406>] __netif_receive_skb+0x16/0x60
Sep 15 02:25:29 nas kernel: [ 86.675494] [<c155846f>] netif_receive_skb+0x1f/0x80
Sep 15 02:25:29 nas kernel: [ 86.675516] [<f845e7ab>] velocity_rx_srv+0x2ab/0x410 [via_velocity]
Sep 15 02:25:29 nas kernel: [ 86.675525] [<c13dfa19>] ? __mix_pool_bytes+0x39/0x80
Sep 15 02:25:29 nas kernel: [ 86.675534] [<c1014790>] ? alternatives_enable_smp+0x110/0x110
Sep 15 02:25:29 nas kernel: [ 86.675547] [<f84608ac>] velocity_poll+0x3c/0x90 [via_velocity]
Sep 15 02:25:29 nas kernel: [ 86.675554] [<c1558725>] net_rx_action+0xf5/0x1f0
Sep 15 02:25:29 nas kernel: [ 86.675562] [<c10573c1>] __do_softirq+0xc1/0x1d0
Sep 15 02:25:29 nas kernel: [ 86.675574] [<c103cd90>] ? ack_apic_level+0x60/0x100
Sep 15 02:25:29 nas kernel: [ 86.675584] [<c10d7100>] ? handle_irq_event_percpu+0x150/0x1a0
Sep 15 02:25:29 nas kernel: [ 86.675591] [<c1057635>] irq_exit+0x95/0xa0
Sep 15 02:25:29 nas kernel: [ 86.675602] [<c163f1f5>] do_IRQ+0x45/0xb0
Sep 15 02:25:29 nas kernel: [ 86.675610] [<c163efb3>] common_interrupt+0x33/0x38
Sep 15 02:25:29 nas kernel: [ 86.675616] ---[ end trace 00a91aa3f264230a ]---

No crash occurs when bringing up the second interface in the machine (using the via_rhine driver).

Testing with the mainline kernels from here http://kernel.ubuntu.com/~kernel-ppa/mainline/ the problem does:
not appear with v3.11-rc5-saucy
appear with v3.11-rc6-saucy
appear with v3.11-saucy (same hang, no crash message in syslog)
appear with v3.11.1-saucy (same hang, no crash message in syslog)

The problem is not present using the ubuntu kernel 3.8.0-29-generic from raring.

The machine in question has the via epia sn18000g motherboard with both mentioned network interfaces onboard.

I suspect the changes here https://github.com/torvalds/linux/commit/2fdac010bdcf10a30711b6924612dfc40daf19b8 could be involved.

The problem appears reliably. So, I am happy to test other kernels (versions, patched, ...).

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: linux-image-3.11.0-7-generic 3.11.0-7.13
ProcVersionSignature: Ubuntu 3.11.0-7.13-generic 3.11.0
Uname: Linux 3.11.0-7-generic i686
ApportVersion: 2.12.1-0ubuntu4
Architecture: i386
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Sun Sep 15 03:01:37 2013
HibernationDevice: RESUME=UUID=505d751d-1bf4-4f44-bb59-b4c1547800f9
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 eth1 no wireless extensions.
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.11.0-7-generic root=UUID=486c7989-4b56-43de-95fd-27ac5a386777 ro console=tty0 console=ttyS0,115200n8 crashkernel=384M-2G:64M,2G-:128M --verbose
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-7-generic N/A
 linux-backports-modules-3.11.0-7-generic N/A
 linux-firmware 1.114
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to saucy on 2013-09-02 (12 days ago)
dmi.bios.date: 12/18/2007
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 080014
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: To be filled by O.E.M.
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: To be filled by O.E.M.
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr080014:bd12/18/2007:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnTobefilledbyO.E.M.:rnTobefilledbyO.E.M.:rvrTobefilledbyO.E.M.:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.

[Brad Figg (brad-figg)]

This change was made by a bot.

[penalvch (penalvch)]

Dirk Kraft, the next step is to bisect between v3.11-rc5-saucy and v3.11-rc6-saucy in order to identify the offending commit. Could you please do this via https://wiki.ubuntu.com/Kernel/KernelBisection ?

[Joseph Salisbury (jsalisbury)]

Before starting a bisect, can you see if this is fixed in the v3.12-rc1 kernel, which can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-rc1-saucy/

[Dirk Kraft (dirk-kraft)]

v3.12-rc1 still has the bug

[Dirk Kraft (dirk-kraft)]

I worked on a bisect, but the problem is that within that range, a lot of revisions can not be compiled because of another bug that affects compilation for i386.

The second bug is discussed here: https://lkml.org/lkml/2013/8/3/40

Working on the bisect I marked version that showed this second bug as skip. I was able to bring down the range of commits but I right now see no way how to bring it down any further (because of the second bug). I am very happy to hear how I could approach this.

I attach the bisect log I created.

[Dirk Kraft (dirk-kraft)]

To work around in the second bug mentioned in my last message I applied the patch mentioned here http://lists.freedesktop.org/archives/dri-devel/2013-August/042668.html to the versions that would not compile properly.

With this change (where necessary) I was able to fully bisect the problem. The output is:

2fdac010bdcf10a30711b6924612dfc40daf19b8 is the first bad commit

[penalvch (penalvch)]

Dirk Kraft, the issue you are reporting is an upstream one. Could you please report this problem through the appropriate channel by following the instructions _verbatim_ at https://wiki.ubuntu.com/Bugs/Upstream/kernel#KernelTeam.2BAC8-KernelTeamBugPolicies.Overview_on_Reporting_Bugs_Upstream ?

Thank you for your understanding.

[Dirk Kraft (dirk-kraft)]

I reported the problem on the linux-netdev mailing list. http://marc.info/?l=linux-netdev&m=137987093901512&w=2

[Dirk Kraft (dirk-kraft)]

Seems like the fedora people followed up on this nicely and there is now a patch submitted upstream.

https://bugzilla.redhat.com/show_bug.cgi?id=1022733#c14

http://article.gmane.org/gmane.linux.network/292095/match=

[Dirk Kraft (dirk-kraft)]

The patch now ended up in linus' repo
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=bc9627e7e918a85e906c1a3f6d01d9b8ef911a96
and is included in 3.13-rc3

[Tim Gardner (timg-tpi)]

Sent a note to the stable maintainers.

[Dirk Kraft (dirk-kraft)]

OK, the patch now also ended up in 3.12.4 (and later). 3.11 is EOL (with 3.11.10) so it won't end up there. (It also is in 3.13-rc3 and later as mentioned before.)

Is there now a way to get this change into the kernels for saucy and trusty?

[penalvch (penalvch)]

Dirk Kraft, this patch is already available in latest Trusty Ubuntu-3.12.0-7.15. If you test this via Trusty does it change anything for you?