System freeze after in6_dev_finish_destroy errors

Bug #1702910 reported by Jinn Ko
58
This bug affects 10 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Triaged
High
Unassigned
Xenial
Triaged
High
Unassigned

Bug Description

After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host.

In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related.

As a possible workaround we have disabled IPv6 for now.

```
kernel: [337352.274907] vethf317e0d: renamed from eth0
kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state
kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state
kernel: [337352.395408] device veth160ed4a left promiscuous mode
kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state
kernel: [337359.301576] vethee2599d: renamed from eth0
kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state
kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state
kernel: [337359.344987] device veth3100aab left promiscuous mode
kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state
kernel: [337475.398358] ------------[ cut here ]------------
kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f

kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab
comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil
filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul
input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4

kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu
kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
kernel: [337475.398423] Workqueue: netns cleanup_net
kernel: [337475.398424] 0000000000000000 ffff8801b197fbb8 ffffffff813ddefc 0000000000000000
kernel: [337475.398427] ffffffff81d7aa10 ffff8801b197fbf0 ffffffff8107ea36 ffff88016f97d400
kernel: [337475.398428] ffff88016f97e800 0000000000000006 ffff8801b197fcc0 0000000000000000
kernel: [337475.398430] Call Trace:
kernel: [337475.398435] [<ffffffff813ddefc>] dump_stack+0x63/0x87
kernel: [337475.398440] [<ffffffff8107ea36>] warn_slowpath_common+0x86/0xc0
kernel: [337475.398442] [<ffffffff8107eb2a>] warn_slowpath_null+0x1a/0x20
kernel: [337475.398444] [<ffffffff817e886b>] in6_dev_finish_destroy+0x6b/0xb0
kernel: [337475.398447] [<ffffffff817be120>] ip6_route_dev_notify+0x110/0x130
kernel: [337475.398450] [<ffffffff8109dd89>] notifier_call_chain+0x49/0x70
kernel: [337475.398452] [<ffffffff8109dea6>] raw_notifier_call_chain+0x16/0x20
kernel: [337475.398455] [<ffffffff81701d25>] call_netdevice_notifiers_info+0x35/0x60
kernel: [337475.398458] [<ffffffff8170c017>] netdev_run_todo+0x157/0x300
kernel: [337475.398461] [<ffffffff81703a3e>] ? rollback_registered_many+0x22e/0x2e0
kernel: [337475.398463] [<ffffffff817172be>] rtnl_unlock+0xe/0x10
kernel: [337475.398464] [<ffffffff81704d88>] default_device_exit_batch+0x138/0x150
kernel: [337475.398468] [<ffffffff810bf4e0>] ? __wake_up_sync+0x20/0x20
kernel: [337475.398470] [<ffffffff816fd242>] ops_exit_list.isra.4+0x52/0x60
kernel: [337475.398472] [<ffffffff816fe213>] cleanup_net+0x1b3/0x280
kernel: [337475.398476] [<ffffffff81096d30>] process_one_work+0x150/0x3f0
kernel: [337475.398478] [<ffffffff810974aa>] worker_thread+0x11a/0x470
kernel: [337475.398481] [<ffffffff818089c9>] ? __schedule+0x359/0x980
kernel: [337475.398483] [<ffffffff81097390>] ? rescuer_thread+0x310/0x310
kernel: [337475.398484] [<ffffffff8109cd66>] kthread+0xd6/0xf0
kernel: [337475.398486] [<ffffffff8109cc90>] ? kthread_park+0x60/0x60
kernel: [337475.398488] [<ffffffff8180cc0f>] ret_from_fork+0x3f/0x70
kernel: [337475.398490] [<ffffffff8109cc90>] ? kthread_park+0x60/0x60
kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]---
kernel: [337475.398492] ------------[ cut here ]------------

```

ProblemType: Bug
ApportVersion: 2.14.1-0ubuntu3.24
Architecture: amd64
Date: Fri Jul 7 13:59:36 2017
Dependencies:
 adduser 3.113+nmu3ubuntu3
 apt-utils 1.0.1ubuntu2.17
 base-passwd 3.5.33
 busybox-initramfs 1:1.21.0-1ubuntu1
 coreutils 8.21-1ubuntu5.4
 cpio 2.11+dfsg-1ubuntu1.2
 dbus 1.6.18-0ubuntu4.5
 debconf 1.5.51ubuntu2
 debconf-i18n 1.5.51ubuntu2
 debianutils 4.4
 dmsetup 2:1.02.77-6ubuntu2
 dpkg 1.17.5ubuntu5.7
 e2fslibs 1.42.9-3ubuntu1.3
 e2fsprogs 1.42.9-3ubuntu1.3
 findutils 4.4.2-7
 gcc-4.8-base 4.8.4-2ubuntu1~14.04.3
 gcc-4.9-base 4.9.3-0ubuntu4
 gettext-base 0.18.3.1-1ubuntu3
 grub-common 2.02~beta2-9ubuntu1.12
 grub-gfxpayload-lists 0.6
 grub-pc 2.02~beta2-9ubuntu1.12
 grub-pc-bin 2.02~beta2-9ubuntu1.12
 grub2-common 2.02~beta2-9ubuntu1.12
 ifupdown 0.7.47.2ubuntu4.4
 initramfs-tools 0.103ubuntu4.7
 initramfs-tools-bin 0.103ubuntu4.7
 initscripts 2.88dsf-41ubuntu6.3
 insserv 1.14.0-5ubuntu2
 iproute2 3.12.0-2ubuntu1
 isc-dhcp-client 4.2.4-7ubuntu12.10
 isc-dhcp-common 4.2.4-7ubuntu12.10
 klibc-utils 2.0.3-0ubuntu1.14.04.2
 kmod 15-0ubuntu6
 libacl1 2.2.52-1
 libapparmor1 2.10.95-0ubuntu2.6~14.04.1
 libapt-inst1.5 1.0.1ubuntu2.17
 libapt-pkg4.12 1.0.1ubuntu2.17
 libasprintf0c2 0.18.3.1-1ubuntu3
 libattr1 1:2.4.47-1ubuntu1
 libaudit-common 1:2.3.2-2ubuntu1
 libaudit1 1:2.3.2-2ubuntu1
 libblkid1 2.20.1-5.1ubuntu20.9
 libbz2-1.0 1.0.6-5
 libc6 2.19-0ubuntu6.13
 libcap2 1:2.24-0ubuntu2
 libcgmanager0 0.24-0ubuntu7.5
 libcomerr2 1.42.9-3ubuntu1.3
 libdb5.3 5.3.28-3ubuntu3
 libdbus-1-3 1.6.18-0ubuntu4.5
 libdebconfclient0 0.187ubuntu1
 libdevmapper1.02.1 2:1.02.77-6ubuntu2
 libdrm2 2.4.67-1ubuntu0.14.04.1
 libexpat1 2.1.0-4ubuntu1.3
 libffi6 3.1~rc1+r3.0.13-12ubuntu0.1
 libfreetype6 2.5.2-1ubuntu2.8
 libfuse2 2.9.2-4ubuntu4.14.04.1
 libgcc1 1:4.9.3-0ubuntu4
 libglib2.0-0 2.40.2-0ubuntu1
 libglib2.0-data 2.40.2-0ubuntu1
 libgpm2 1.20.4-6.1
 libjson-c2 0.11-3ubuntu1.2
 libjson0 0.11-3ubuntu1.2
 libklibc 2.0.3-0ubuntu1.14.04.2
 libkmod2 15-0ubuntu6
 liblocale-gettext-perl 1.05-7build3
 liblzma5 5.1.1alpha+20120614-2ubuntu2
 libmount1 2.20.1-5.1ubuntu20.9
 libncurses5 5.9+20140118-1ubuntu1
 libncursesw5 5.9+20140118-1ubuntu1
 libnih-dbus1 1.0.3-4ubuntu25
 libnih1 1.0.3-4ubuntu25
 libpam-modules 1.1.8-1ubuntu2.2
 libpam-modules-bin 1.1.8-1ubuntu2.2
 libpam-runtime 1.1.8-1ubuntu2.2
 libpam-systemd 204-5ubuntu20.24
 libpam0g 1.1.8-1ubuntu2.2
 libpcre3 1:8.31-2ubuntu2.3
 libplymouth2 0.8.8-0ubuntu17.1
 libpng12-0 1.2.50-1ubuntu2.14.04.2
 libprocps3 1:3.3.9-1ubuntu2.2
 libselinux1 2.2.2-1ubuntu0.1
 libsemanage-common 2.2-1
 libsemanage1 2.2-1
 libsepol1 2.2-1ubuntu0.1
 libslang2 2.2.4-15ubuntu1
 libss2 1.42.9-3ubuntu1.3
 libstdc++6 4.8.4-2ubuntu1~14.04.3
 libsystemd-daemon0 204-5ubuntu20.24
 libsystemd-login0 204-5ubuntu20.24
 libtext-charwidth-perl 0.04-7build3
 libtext-iconv-perl 1.7-5build2
 libtext-wrapi18n-perl 0.06-7
 libtinfo5 5.9+20140118-1ubuntu1
 libudev1 204-5ubuntu20.24
 libustr-1.0-1 1.0.4-3ubuntu2
 libuuid1 2.20.1-5.1ubuntu20.9
 libxml2 2.9.1+dfsg1-3ubuntu4.9
 libxtables10 1.4.21-1ubuntu1
 lsb-base 4.1+Debian11ubuntu6.2
 makedev 2.3.1-93ubuntu2~ubuntu14.04.1
 module-init-tools 15-0ubuntu6
 mount 2.20.1-5.1ubuntu20.9
 mountall 2.53
 multiarch-support 2.19-0ubuntu6.13
 netbase 5.2
 os-prober 1.63ubuntu1.1
 passwd 1:4.1.5.1-1ubuntu9.5
 perl-base 5.18.2-2ubuntu1.1
 plymouth 0.8.8-0ubuntu17.1
 procps 1:3.3.9-1ubuntu2.2
 psmisc 22.20-1ubuntu2
 sed 4.2.2-4ubuntu1
 sensible-utils 0.0.9
 sgml-base 1.26+nmu4ubuntu1
 shared-mime-info 1.2-0ubuntu3
 systemd-services 204-5ubuntu20.24
 systemd-shim 6-2bzr1
 sysv-rc 2.88dsf-41ubuntu6.3
 sysvinit-utils 2.88dsf-41ubuntu6.3
 tar 1.27.1-1ubuntu0.1
 tzdata 2016j-0ubuntu0.14.04
 ucf 3.0027+nmu1
 udev 204-5ubuntu20.24
 upstart 1.12.1-0ubuntu4.2
 util-linux 2.20.1-5.1ubuntu20.9
 xml-core 0.13+nmu2
 zlib1g 1:1.2.8.dfsg-1ubuntu1
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2015-08-20 (686 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
Package: linux-image-4.4.0-83-generic 4.4.0-83.106~14.04.1
PackageArchitecture: amd64
ProcCpuinfoMinimal:
 processor : 3
 vendor_id : GenuineIntel
 cpu family : 6
 model : 45
 model name : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
 stepping : 2
 microcode : 0x710
 cpu MHz : 2700.000
 cache size : 20480 KB
 physical id : 6
 siblings : 1
 core id : 0
 cpu cores : 1
 apicid : 6
 initial apicid : 6
 fpu : yes
 fpu_exception : yes
 cpuid level : 13
 wp : yes
 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts
 bugs :
 bogomips : 5400.00
 clflush size : 64
 cache_alignment : 64
 address sizes : 40 bits physical, 48 bits virtual
 power management:
ProcEnviron:
 LANGUAGE=en_GB:
 LC_CTYPE=C
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/zsh
ProcVersionSignature: Ubuntu 4.4.0-83.106~14.04.1-generic 4.4.70
SourcePackage: linux-lts-xenial
Tags: trusty
Uname: Linux 4.4.0-83-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1702910

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: xenial
Revision history for this message
Jinn Ko (jinnko) wrote :

Logs are inline. If anything else is needed I'll capture and attach.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Jinn Ko (jinnko) wrote :

With the sysctl net.ipv6.conf.all.disable_ipv6=1 we have still seen another kernel error as follows:

[97989.224382] ------------[ cut here ]------------
[97989.224395] WARNING: CPU: 1 PID: 19570 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xb0()
[97989.224456] CPU: 1 PID: 19570 Comm: kworker/u8:3 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu
[97989.224457] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[97989.224462] Workqueue: netns cleanup_net
[97989.224463] 0000000000000000 ffff88011b75bbb8 ffffffff813ddefc 0000000000000000
[97989.224466] ffffffff81d7aa10 ffff88011b75bbf0 ffffffff8107ea36 ffff8800960e7c00
[97989.224467] ffff8800960e6c00 0000000000000006 ffff88011b75bcc0 0000000000000000
[97989.224469] Call Trace:
[97989.224474] [<ffffffff813ddefc>] dump_stack+0x63/0x87
[97989.224478] [<ffffffff8107ea36>] warn_slowpath_common+0x86/0xc0
[97989.224479] [<ffffffff8107eb2a>] warn_slowpath_null+0x1a/0x20
[97989.224481] [<ffffffff817e886b>] in6_dev_finish_destroy+0x6b/0xb0
[97989.224487] [<ffffffff817be120>] ip6_route_dev_notify+0x110/0x130
[97989.224490] [<ffffffff8109dd89>] notifier_call_chain+0x49/0x70
[97989.224492] [<ffffffff8109dea6>] raw_notifier_call_chain+0x16/0x20
[97989.224495] [<ffffffff81701d25>] call_netdevice_notifiers_info+0x35/0x60
[97989.224498] [<ffffffff8170c017>] netdev_run_todo+0x157/0x300
[97989.224500] [<ffffffff817172be>] rtnl_unlock+0xe/0x10
[97989.224502] [<ffffffff81704d88>] default_device_exit_batch+0x138/0x150
[97989.224507] [<ffffffff810bf4e0>] ? __wake_up_sync+0x20/0x20
[97989.224509] [<ffffffff816fd242>] ops_exit_list.isra.4+0x52/0x60
[97989.224510] [<ffffffff816fe213>] cleanup_net+0x1b3/0x280
[97989.224514] [<ffffffff81096d30>] process_one_work+0x150/0x3f0
[97989.224516] [<ffffffff810974aa>] worker_thread+0x11a/0x470
[97989.224518] [<ffffffff81097390>] ? rescuer_thread+0x310/0x310
[97989.224520] [<ffffffff8109cd66>] kthread+0xd6/0xf0
[97989.224523] [<ffffffff81081865>] ? do_exit+0x655/0xae0
[97989.224525] [<ffffffff8109cc90>] ? kthread_park+0x60/0x60
[97989.224527] [<ffffffff8180cc0f>] ret_from_fork+0x3f/0x70
[97989.224529] [<ffffffff8109cc90>] ? kthread_park+0x60/0x60
[97989.224530] ---[ end trace dfb865796f5b5681 ]---
[97989.224531] ------------[ cut here ]------------

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12

Changed in linux (Ubuntu):
importance: Undecided → High
status: Confirmed → Incomplete
tags: added: kernel-da-key
Revision history for this message
Jinn Ko (jinnko) wrote :

We're aiming to reproduce this issue on another instance, then will switch to the mainline kernel to gather the info you need.

Revision history for this message
Steffen Pingel (stp) wrote :
Revision history for this message
Marin Nedea (marin-n) wrote :
Download full text (11.9 KiB)

I can confirm the bug:
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320092] ------------[ cut here ]------------
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320100] WARNING: CPU: 1 PID: 42381 at /build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xc0()
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320102] Modules linked in: cmac arc4 md4 nls_utf8 cifs fscache xt_statistic sch_htb ebt_ip ebtable_filter xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_mark xt_comment veth ebtable_nat ebtables ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay i2c_piix4 8250_fintek input_leds joydev mac_hid serio_raw ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hv_netvsc hv_utils hid_hyperv hv_storvsc hid scsi_transport_fc hyperv_keyboard aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper hyperv_fb cryptd psmouse pata_acpi hv_vmbus floppy fjes
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320149] CPU: 1 PID: 42381 Comm: kworker/u128:3 Not tainted 4.4.0-83-generic #106-Ubuntu
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320151] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320155] Workqueue: netns cleanup_net
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320156] 0000000000000286 00000000174b4ae5 ffff880686863ba0 ffffffff813f9513
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320158] 0000000000000000 ffffffff81d75940 ffff880686863bd8 ffffffff81081322
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320160] ffff88048ff82800 ffff88048ff80000 0000000000000006 ffff880686863ca8
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320162] Call Trace:
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320166] [<ffffffff813f9513>] dump_stack+0x63/0x90
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320170] [<ffffffff81081322>] warn_slowpath_common+0x82/0xc0
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320172] [<ffffffff8108146a>] warn_slowpath_null+0x1a/0x20
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320173] [<ffffffff8181ba0b>] in6_dev_finish_destroy+0x6b/0xc0
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320176] [<ffffffff817f17e6>] ip6_route_dev_notify+0x116/0x130
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320178] [<ffffffff810a1b1a>] notifier_call_chain+0x4a/0x70
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320180] [<ffffffff810a1c96>] raw_notifier_call_chain+0x16/0x20
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320182] [<ffffffff8172f655>] call_netdevice_notifiers_info+0x35/0x60
Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320185] [<fffff...

Revision history for this message
Jinn Ko (jinnko) wrote :
Download full text (8.2 KiB)

We've successfully reproduced the error on a newly built instance. We'll proceed to run the mainline kernel on the same instance.

Worth noting that it took 441529 seconds (5.1 days) since boot for this to manifest in this case. On our previous instance the time to failure was in the order of a hours to days.

[441529.300238] ------------[ cut here ]------------
[441529.300288] WARNING: CPU: 1 PID: 10409 at /build/linux-lts-xenial-N0BD4p/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xb0()
[441529.300359] CPU: 1 PID: 10409 Comm: kworker/u4:0 Not tainted 4.4.0-87-generic #110~14.04.1-Ubuntu
[441529.300361] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[441529.300374] Workqueue: netns cleanup_net
[441529.300376] 0000000000000000 ffff880088ddbbb8 ffffffff813de2dc 0000000000000000
[441529.300378] ffffffff81d7aa60 ffff880088ddbbf0 ffffffff8107ea46 ffff880139e5a400
[441529.300380] 0000000000000000 0000000000000006 ffff880088ddbcc0 0000000000000000
[441529.300382] Call Trace:
[441529.300398] [<ffffffff813de2dc>] dump_stack+0x63/0x87
[441529.300413] [<ffffffff8107ea46>] warn_slowpath_common+0x86/0xc0
[441529.300415] [<ffffffff8107eb3a>] warn_slowpath_null+0x1a/0x20
[441529.300417] [<ffffffff817e8b9b>] in6_dev_finish_destroy+0x6b/0xb0
[441529.300421] [<ffffffff817be360>] ip6_route_dev_notify+0x110/0x130
[441529.300425] [<ffffffff8109dde9>] notifier_call_chain+0x49/0x70
[441529.300427] [<ffffffff8109df06>] raw_notifier_call_chain+0x16/0x20
[441529.300429] [<ffffffff81701d35>] call_netdevice_notifiers_info+0x35/0x60
[441529.300432] [<ffffffff8170c007>] netdev_run_todo+0x157/0x300
[441529.300436] [<ffffffff81703a4e>] ? rollback_registered_many+0x22e/0x2e0
[441529.300440] [<ffffffff817172fe>] rtnl_unlock+0xe/0x10
[441529.300442] [<ffffffff81704d98>] default_device_exit_batch+0x138/0x150
[441529.300452] [<ffffffff810bf540>] ? __wake_up_sync+0x20/0x20
[441529.300454] [<ffffffff816fd252>] ops_exit_list.isra.4+0x52/0x60
[441529.300456] [<ffffffff816fe223>] cleanup_net+0x1b3/0x280
[441529.300461] [<ffffffff81096d90>] process_one_work+0x150/0x3f0
[441529.300463] [<ffffffff8109750a>] worker_thread+0x11a/0x470
[441529.300471] [<ffffffff81808d99>] ? __schedule+0x359/0x980
[441529.300473] [<ffffffff810973f0>] ? rescuer_thread+0x310/0x310
[441529.300475] [<ffffffff8109cdc6>] kthread+0xd6/0xf0
[441529.300477] [<ffffffff81081875>] ? do_exit+0x655/0xae0
[441529.300479] [<ffffffff8109ccf0>] ? kthread_park+0x60/0x60
[441529.300481] [<ffffffff8180d00f>] ret_from_fork+0x3f/0x70
[441529.300483] [<ffffffff8109ccf0>] ? kthread_park+0x60/0x60
[441529.300484] ---[ end trace 5e09761f8674a0af ]---
[441529.300485] ------------[ cut here ]------------
[441529.300487] WARNING: CPU: 1 PID: 10409 at /build/linux-lts-xenial-N0BD4p/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:160 in6_dev_finish_destroy+0xa4/0xb0()
[441529.300510] CPU: 1 PID: 10409 Comm: kworker/u4:0 Tainted: G W 4.4.0-87-generic #110~14.04.1-Ubuntu
[441529.300511] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[441529.30051...

Read more...

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Can you also see if this bug happens with the latest upstream stable 4.4 kernel? It is available from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.85/

Revision history for this message
Victor Palma (victor-palma) wrote :

I can confirm that this also happens with 4.4.0-87-generic.

Revision history for this message
Victor Palma (victor-palma) wrote :
Download full text (4.2 KiB)

Here is additional debug information:

      KERNEL: /usr/lib/debug/boot/vmlinux-4.4.0-87-generic
    DUMPFILE: dump.201709151609 [PARTIAL DUMP]
        CPUS: 12
        DATE: Fri Sep 15 16:05:34 2017
      UPTIME: 1 days, 17:34:46
LOAD AVERAGE: 4.17, 5.02, 4.48
       TASKS: 2572
    NODENAME: 543231-infra02
     RELEASE: 4.4.0-87-generic
     VERSION: #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017
     MACHINE: x86_64 (3000 Mhz)
      MEMORY: 32 GB
       PANIC: "BUG: unable to handle kernel paging request at 0000000000400000"
         PID: 4189
     COMMAND: "dnsmasq"
        TASK: ffff8808299dd400 [THREAD_INFO: ffff88042502c000]
         CPU: 11
       STATE: TASK_RUNNING (PANIC)

[149668.845048] WARNING: CPU: 0 PID: 11227 at /build/linux-5EyXrQ/linux-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xc0()
[149668.845050] Modules linked in: nf_conntrack_netlink ebt_arp ebt_among xt_CT xt_mac xt_physdev xt_set ip_set_hash_net ip_set nfnetlink xt_REDIRECT nf_nat_redirect nf_conntrack_ipv6 nf_defrag_ipv6 xt_nat xt_conntrack xt_mark xt_connmark tcp_diag udp_diag inet_diag unix_diag ebtable_filter xt_comment ip6table_raw ip6table_mangle sunrpc binfmt_misc veth xt_CHECKSUM xt_tcpudp iptable_raw ipmi_ssif kvm_amd ib_iser kvm amd64_edac_mod rdma_cm iw_cm 8250_fintek edac_mce_amd irqbypass ib_cm fam15h_power edac_core ib_sa serio_raw ib_mad i2c_piix4 ipmi_si ipmi_msghandler k10temp hpilo ib_core shpchp ib_addr vhost_net vhost macvtap macvlan nbd mac_hid iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_mangle iptable_filter ipt_REJECT nf_reject_ipv4
[149668.845092] ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack ip_tables ip6table_filter ip6_tables ebtables x_tables dm_snapshot dm_bufio dm_multipath br_netfilter bridge 8021q garp mrp stp llc ixgbe mdio tg3 bonding autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en vxlan ip6_udp_tunnel udp_tunnel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul syscopyarea glue_helper sysfillrect ablk_helper sysimgblt fb_sys_fops cryptd drm psmouse pata_acpi igb pata_atiixp hpsa dca ahci mlx4_core ptp libahci scsi_transport_sas pps_core i2c_algo_bit fjes
[149668.845132] CPU: 0 PID: 11227 Comm: kworker/u24:1 Not tainted 4.4.0-87-generic #110-Ubuntu
[149668.845134] Hardware name: HP ProLiant SE4255e /, BIOS A31 01/05/2013
[149668.845138] Workqueue: netns cleanup_net
[149668.845139] 0000000000000286 e218996f8a60a1fa ffff8802f3637ba0 ffffffff813f9903
[149668.845142] 0000000000000000 ffffffff81d75978 ffff8802f3637bd8 ffffffff81081332
[149668.845144] ffff8806dd999400 ffff8806dd99e800 0000000000000006 ffff8802f3637ca8
[149668.845146] Call Trace:
[149668.845151] [<ffffffff813f9903>] dump_stack+0x63/0x90
[149668.845154] [<ffffffff81081332>] warn_slowpath_common+0x82/0xc0
[149668.845156] [<ffffffff8108147a>] warn_slowpath_null+0x1a/0x20
[149668.845158] [<ffffffff8181cceb>] in6_dev_finish_destroy+0x6b/0xc0
[149668.845161] [<ffffffff817f29e6>] ip6_route_dev_notify+0x...

Read more...

Revision history for this message
Matthew Thode (prometheanfire) wrote :

Were you able to reproduce on a 4.12 kernel? Maybe even a 4.14 kernel (even though it's just the first rc that's been cut, it is the next lts). 4.9 is another lts kernel series that could be tested.

Revision history for this message
Major Hayden (rackerhacker) wrote :

There's a discussion here possibly about the same problem:

  https://patchwork.ozlabs.org/patch/801533/

And they referred to this patch:

  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=242d3a49a2a1a71d8eb9f953db1bcaa9d698ce00

The changes look like they line up with the call trace order, but I could be wrong.

Changed in linux (Ubuntu):
status: Incomplete → Triaged
Changed in linux (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Can folks affected by this bug test the kernel located at:
http://kernel.ubuntu.com/~jsalisbury/lp1702910/

To test the kernel, please install both the linux-image and linux-image-extra .deb packages.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.