“BUG: scheduling while atomic: nc/3622/0x00000101”

[Anil Shashikumar Belur]

"BUG: scheduling while atomic" ==> IMU this error possibly indicates that a process/kernel trying an operation which is attempting to sleep while in an un-interruptible state, which could result in a dead-lock condition. This issue is seen NOT frequently when loading a custom netfilter hook (.ko) on my box.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-32-generic 3.13.0-32.57
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: abelur 2054 F.... pulseaudio
 /dev/snd/controlC0: abelur 2054 F.... pulseaudio
CurrentDesktop: Unity
Date: Fri Oct 17 09:59:09 2014
InstallationDate: Installed on 2014-05-07 (162 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
MachineType: Dell Inc. Inspiron 5537
ProcFB:
 0 inteldrmfb
 1 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=UUID=7be3bd48-774e-4ef9-aab1-69781ae2ad9f ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-32-generic N/A
 linux-backports-modules-3.13.0-32-generic N/A
 linux-firmware 1.127.7
SourcePackage: linux
StagingDrivers: rts5139
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 11/12/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A07
dmi.board.name: 0FTDJN
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: A07
dmi.modalias: dmi:bvnDellInc.:bvrA07:bd11/12/2013:svnDellInc.:pnInspiron5537:pvrA07:rvnDellInc.:rn0FTDJN:rvrA00:cvnDellInc.:ct8:cvrA07:
dmi.product.name: Inspiron 5537
dmi.product.version: A07
dmi.sys.vendor: Dell Inc.

[Anil Shashikumar Belur (askb23)]

Changed the package to linux-image-3.13.0-32-generic

[Brad Figg (brad-figg)]

This change was made by a bot.

[Anil Shashikumar Belur (askb23)]

Updating latest stack trace from dmesg:
[ 132.216761] BUG: scheduling while atomic: nc/2911/0x00000101
[ 132.216766] Modules linked in: netfilter(POF) ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) ctr ccm bnep rfcomm binfmt_misc snd_hda_codec_realtek snd_hda_codec_hdmi dell_wmi sparse_keymap rts5139(C) uvcvideo hid_multitouch videobuf2_vmalloc videobuf2_memops videobuf2_core videodev ath3k btusb bluetooth dell_laptop dcdbas snd_hda_intel snd_hda_codec snd_hwdep snd_pcm intel_rapl x86_pkg_temp_thermal snd_seq_midi intel_powerclamp coretemp snd_seq_midi_event kvm_intel arc4 kvm snd_rawmidi crct10dif_pclmul snd_page_alloc crc32_pclmul ghash_clmulni_intel ath9k aesni_intel aes_x86_64 ath9k_common ath9k_hw lrw gf128mul glue_helper ath ablk_helper cryptd mac80211 snd_seq radeon joydev serio_raw snd_seq_device snd_timer lpc_ich ttm i915 cfg80211 drm_kms_helper snd drm mei_me mei i2c_algo_bit soundcore wmi parport_pc video mac_hid ppdev lp parport hid_generic usbhid hid psmouse ahci libahci r8169 mii
[ 132.217088] CPU: 1 PID: 2911 Comm: nc Tainted: PF C O 3.13.0-32-generic #57-Ubuntu
[ 132.217092] Hardware name: Dell Inc. Inspiron 5537/0FTDJN, BIOS A07 11/12/2013
[ 132.217096] ffff88025f254440 ffff88025f243a20 ffffffff8171bcb4 ffff88025f243bc0
[ 132.217115] ffff88025f243a30 ffffffff817154c1 ffff88025f243a90 ffffffff8171ff7e
[ 132.217137] ffff880085ab8000 ffff88021c8a9fd8 0000000000014440 0000000000014440
[ 132.217159] Call Trace:
[ 132.217166] <IRQ> [<ffffffff8171bcb4>] dump_stack+0x45/0x56
[ 132.217197] [<ffffffff817154c1>] __schedule_bug+0x4c/0x5a
[ 132.217211] [<ffffffff8171ff7e>] __schedule+0x69e/0x7d0
[ 132.217231] [<ffffffff817200d9>] schedule+0x29/0x70
[ 132.217242] [<ffffffff8171f329>] schedule_timeout+0x239/0x2d0
[ 132.217256] [<ffffffff81121468>] ? trace_function+0x68/0x90
[ 132.217277] [<ffffffff810817fb>] ? insert_work+0x6b/0xb0
[ 132.217293] [<ffffffff8172137e>] wait_for_completion_killable+0xbe/0x1d0
[ 132.217309] [<ffffffff8109a800>] ? wake_up_state+0x20/0x20
[ 132.217327] [<ffffffff8107fe47>] call_usermodehelper_exec+0xe7/0x150
[ 132.217344] [<ffffffff810802a3>] __request_module+0x1b3/0x2a0
[ 132.217362] [<ffffffff8111a5ad>] ? ring_buffer_unlock_commit+0x2d/0x250
[ 132.217387] [<ffffffff81385566>] textsearch_prepare+0xf6/0x170
[ 132.217397] [<ffffffff81655f30>] ? inet_del_offload+0x40/0x40
[ 132.217405] [<ffffffffa09a3043>] in_hook_func+0x43/0x150 [netfilter]
[ 132.217413] [<ffffffffa09a3005>] ? in_hook_func+0x5/0x150 [netfilter]
[ 132.217422] [<ffffffff8164faba>] nf_iterate+0x9a/0xb0
[ 132.217430] [<ffffffff81655f30>] ? inet_del_offload+0x40/0x40
[ 132.217437] [<ffffffff8164fb44>] nf_hook_slow+0x74/0x130
[ 132.217443] [<ffffffff81655f30>] ? inet_del_offload+0x40/0x40
[ 132.217452] [<ffffffff81656958>] ip_rcv+0x2f8/0x3d0
[ 132.217461] [<ffffffff81620366>] __netif_receive_skb_core+0x666/0x840
[ 132.217471] [...

[Anil Shashikumar Belur (askb23)]

I have enabled the function call trace and attached the trace.bz2 file.

[Anil Shashikumar Belur (askb23)]

Updating sample code, test script for reproducer (included in ubug_1382340.tar.gz), if this helps. Here are the steps to run / reproduce the issue:

1. untar ubug_1382340.tar.gz and cd ./ubug_1382340
2. make
2. sudo ./test.sh ===> run this heavy network load.

[Anil Shashikumar Belur (askb23)]

Looks like the issue is in the implementation of the netfilter hook kernel module, as its not recommended to use GFP_KERNEL, and cannot sleep in netfilter hooks, since the hookos are run in interrupt context (when the packet is received) by the net_rx_action softirq. Instead this issue is resolved using GFP_ATOMIC. I think this BUG can be closed.