systemd-udevd high CPU usage

[Sergio Benjamim]

I got high CPU usage with systemd-udevd, 100% in the 8 threads. It happened 2 times.

Xubuntu 14.04, kernel 3.14.52-031452-generic.

KiB Swap: 4194300 total, 218436 used, 3975864 free. 2361304 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21485 root 20 0 51780 1148 516 R 97,7 0,0 2:53.79 systemd-udevd
21534 root 20 0 52704 1532 168 R 95,4 0,0 0:16.19 systemd-udevd
21496 root 20 0 52176 1000 168 R 33,2 0,0 2:11.88 systemd-udevd
21495 root 20 0 52176 1136 300 R 32,9 0,0 2:19.32 systemd-udevd
21536 root 20 0 52704 1532 168 R 32,6 0,0 0:06.07 systemd-udevd
21525 root 20 0 52704 1532 168 R 29,9 0,0 0:27.81 systemd-udevd
21529 root 20 0 52704 1532 168 R 29,6 0,0 0:19.85 systemd-udevd
21487 root 20 0 51780 820 308 R 28,6 0,0 2:51.56 systemd-udevd
21506 root 20 0 52704 1532 168 R 28,6 0,0 0:53.19 systemd-udevd
21533 root 20 0 52704 1532 168 R 27,9 0,0 0:08.06 systemd-udevd
21500 root 20 0 52572 1364 168 R 26,6 0,0 1:31.77 systemd-udevd
21505 root 20 0 52704 1532 168 R 26,6 0,0 1:00.96 systemd-udevd
21528 root 20 0 52704 1532 168 R 26,3 0,0 0:25.09 systemd-udevd
21499 root 20 0 52572 1364 168 R 24,9 0,0 1:15.30 systemd-udevd
21501 root 20 0 52572 1364 168 R 24,9 0,0 1:41.89 systemd-udevd
21507 root 20 0 52704 1532 168 R 24,9 0,0 0:49.58 systemd-udevd
21535 root 20 0 52704 1532 168 R 24,9 0,0 0:08.21 systemd-udevd
21498 root 20 0 52572 1364 168 R 24,6 0,0 1:51.32 systemd-udevd
21508 root 20 0 52704 1532 168 R 24,6 0,0 1:15.18 systemd-udevd
21522 root 20 0 52704 1784 340 R 24,6 0,0 0:59.88 systemd-udevd
 4533 sergio 20 0 2129052 604312 65552 S 3,0 7,6 16:20.87 firefox
24962 sergio 20 0 815400 29356 15628 S 1,7 0,4 66:42.11 gnome-system-mo
 1514 root 20 0 611140 145692 91372 S 1,3 1,8 121:58.81 Xorg

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: udev 204-5ubuntu20.13
Uname: Linux 3.14.52-031452-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.13
Architecture: amd64
CustomUdevRuleFiles: 99-evdev.rules
Date: Mon Sep 21 12:38:09 2015
InstallationDate: Installed on 2014-05-26 (483 days ago)
InstallationMedia: Xubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
MachineType: ASUSTeK COMPUTER INC. X750JB
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.14.52-031452-generic root=UUID=1661f37a-c72d-46a6-a746-d0160ec6e57c ro quiet splash vt.handoff=7
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/14/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: X750JB.208
dmi.board.asset.tag: ATN12345678901234567
dmi.board.name: X750JB
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: 1.0
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: ASUSTeK COMPUTER INC.
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrX750JB.208:bd08/14/2013:svnASUSTeKCOMPUTERINC.:pnX750JB:pvr1.0:rvnASUSTeKCOMPUTERINC.:rnX750JB:rvr1.0:cvnASUSTeKCOMPUTERINC.:ct10:cvr1.0:
dmi.product.name: X750JB
dmi.product.version: 1.0
dmi.sys.vendor: ASUSTeK COMPUTER INC.

[Martin Pitt (pitti)]

Dmesg has tons of kernel errors, which are likely related. When this happens again, can you please run "udevadm monitor -k"? Do you get lots of events there? I. e. is udev merely trying to process a bazillion events that the kernel throws at it, or is it busy with itself?

[Sergio Benjamim (sergio-br2)]

I get nothing with "udevadm monitor -k"

In dmesg I'm getting lot of this:

[21389.065280] BUG: soft lockup - CPU#2 stuck for 24s! [systemd-udevd:16162]
[21389.065284] Modules linked in: hid_sony ff_memless cdc_acm bbswitch(OF) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) rfcomm bnep bluetooth 6lowpan_iphc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic binfmt_misc joydev uvcvideo hid_generic videobuf2_vmalloc videobuf2_memops videobuf2_core ums_realtek usbhid usb_storage videodev hid nls_iso8859_1 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm asus_nb_wmi asus_wmi sparse_keymap mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm arc4 snd_seq_midi snd_seq_midi_event crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_rawmidi ath9k ath9k_common aesni_intel ath9k_hw aes_x86_64 lrw ath gf128mul glue_helper ablk_helper cryptd i915 mac80211 snd_seq microcode snd_seq_device snd_timer cfg80211 snd serio_raw drm_kms_helper drm mei_me mei i2c_algo_bit soundcore lpc_ich video shpchp mac_hid wmi parport_pc ppdev msr lp parport ahci psmouse r8169 libahci mii [last unloaded: nvidia]
[21389.065334] CPU: 2 PID: 16162 Comm: systemd-udevd Tainted: PF O 3.14.52-031452-generic #201509131726
[21389.065336] Hardware name: ASUSTeK COMPUTER INC. X750JB/X750JB, BIOS X750JB.208 08/14/2013
[21389.065338] task: ffff88020fab4b30 ti: ffff8801ddcca000 task.ti: ffff8801ddcca000
[21389.065339] RIP: 0010:[<ffffffff81728762>] [<ffffffff81728762>] _raw_spin_lock+0x12/0x50
[21389.065345] RSP: 0018:ffff8801ddccbb60 EFLAGS: 00000202
[21389.065346] RAX: 0000000041804180 RBX: 0000000000000000 RCX: 00000000000039e2
[21389.065348] RDX: ffff8800b9a9e440 RSI: ffff8801ddccbba8 RDI: ffff8801f72f3718
[21389.065349] RBP: ffff8801ddccbb60 R08: 0000000000000000 R09: 0000000000000246
[21389.065350] R10: ffffffff8123beae R11: ffffea000312d8c0 R12: 00000000f028f026
[21389.065351] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000246
[21389.065353] FS: 00007f88b48e5880(0000) GS:ffff88022fa80000(0000) knlGS:0000000000000000
[21389.065354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21389.065355] CR2: 00007f04a9832000 CR3: 0000000212fac000 CR4: 00000000001407e0
[21389.065357] Stack:
[21389.065358] ffff8801ddccbb90 ffffffff811dbe1a ffff8801ddccbba8 ffff8800b9a9e180
[21389.065361] 0000000000000024 ffff8801ddccbdd0 ffff8801ddccbbd8 ffffffff811dce54
[21389.065363] ffff8800b9a9e180 ffff8800b9a9e2c0 ffff8801f72f3740 0000000000000024
[21389.065366] Call Trace:
[21389.065371] [<ffffffff811dbe1a>] shrink_dentry_list+0x2a/0x110
[21389.065374] [<ffffffff811dce54>] check_submounts_and_drop+0x74/0xa0
[21389.065378] [<ffffffff8123baec>] kernfs_dop_revalidate+0x5c/0xd0
[21389.065380] [<ffffffff811cfe5d>] lookup_fast+0x26d/0x2c0
[21389.065383] [<ffffffff812df85c>] ? security_inode_permission+0x1c/0x30
[21389.065385] [<ffffffff811d08b2>] ? __inode_permission+0x52/0xc0
[21389.065388] [<ffffffff811d0f29>] link_path_walk+0x1d9/0x8b0
[21389.065390] [<ffffffff811dae35>] ? __d_free+0x35/0x40
[21389.065392] [<ffffffff811dbbd5>] ? dentry_kill+0x255/0x2e0
[21389.065395] [<ffffffff811e46a4>] ? mntput+0x24/0x40
[21389.065397] [<ff...

[Martin Pitt (pitti)]

Thanks. Reassigning to the kernel then.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Guillaume Morin (canard)]

I think this is due to the backporting of the 3 commits in this thread: https://groups.google.com/forum/#!msg/linux.kernel/e4tcY4cvHGg/tLZi97S02a0J

3.14 needs the fixes from this thread as well