[4.4.0-144 regression] kernel BUG at [...]/linux-lts-xenial-4.4.0/net/core/skbuff.c:1207

[Tore Anderson]

Following a kernel upgrade from linux-image-4.4.0-116-generic to linux-image-4.4.0-144-generic, our IPTables-based firewall have become unstable and have crashed twice with identical-looking backtraces after a short uptime.

When running linux-image-4.4.0-116-generic the firewall had been up and running stable with an uptime of well over a year. Therefore, I highly suspect that this is a bug that has been introduced between the two versions mentioned.

These are the backtraces printed to the serial console when it crashed.

Crash #1 (after approx. 3 days of uptime):

[258363.286572] ------------[ cut here ]------------
[258363.311248] kernel BUG at /build/linux-lts-xenial-YWfqtJ/linux-lts-xenial-4.4.0/net/core/skbuff.c:1207!
[258363.362309] invalid opcode: 0000 [#1] SMP
[258363.385411] Modules linked in: ip6table_raw ip6table_mangle ip6t_REJECT nf_reject_ipv6 iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_mark iptable_mangle xt_comment ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_limit xt_tcpudp xt_iprange ip_set_hash_net hpwdt nfnetlink_log ip6table_filter ip6_tables xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter ip_tables x_tables dm_crypt intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul nf_conntrack_irc nf_conntrack_tftp nf_conntrack_ftp crc32_pclmul ghash_clmulni_intel hpilo aesni_intel aes_x86_64 nf_conntrack_ipv6 acpi_power_meter mac_hid lrw 8250_fintek nf_defrag_ipv6 shpchp gpio_ich gf128mul serio_raw nf_conntrack_ipv4 i7core_edac glue_helper nf_defrag_ipv4 edac_core nf_conntrack ablk_helper cryptd lpc_ich ipmi_ssif ipmi_devintf ipmi_si ipmi_msghandler ipip tunnel4 ip_tunnel 8021q garp mrp stp llc bonding raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon ixgbe i2c_algo_bit ttm dca drm_kms_helper vxlan syscopyarea ip6_udp_tunnel sysfillrect udp_tunnel sysimgblt ptp fb_sys_fops hpsa pps_core psmouse drm mdio bnx2 scsi_transport_sas fjes
[258363.967279] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.0-144-generic #170~14.04.1-Ubuntu
[258364.011349] Hardware name: HP ProLiant DL360 G7, BIOS P68 07/02/2013
[258364.045068] task: ffffffff81e15500 ti: ffffffff81e00000 task.ti: ffffffff81e00000
[258364.085793] RIP: 0010:[<ffffffff81711d07>] [<ffffffff81711d07>] pskb_expand_head+0x227/0x260
[258364.132139] RSP: 0018:ffff88011ba03b90 EFLAGS: 00010202
[258364.162253] RAX: 0000000000000002 RBX: ffff8800bff20e00 RCX: 0000000002080020
[258364.201311] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff8800bff20e00
[258364.240352] RBP: ffff88011ba03bc8 R08: 0000000000000001 R09: ffff8800d8720000
[258364.279499] R10: ffffffffffffffff R11: 0000000000000040 R12: 0000000000000000
[258364.319220] R13: ffff8800bff20e00 R14: 0000000000000001 R15: ffff8800d327fdce
[258364.358038] FS: 0000000000000000(0000) GS:ffff88011ba00000(0000) knlGS:0000000000000000
[258364.401198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[258364.432486] CR2: 00007fcdc0672543 CR3: 0000000001e0c000 CR4: 0000000000020670
[258364.470918] Stack:
[258364.481920] ffff88011ba03de0 ffff88011ba03de0 ffff8800bff20e00 ffff88011ba03ca0
[258364.521426] 0000000000000001 0000000000000001 ffff8800d327fdce ffff88011ba03c10
[258364.561323] ffffffff81713aa0 ffff8802195c3b28 0000000100000005 ffff8800bff20e00
[258364.601724] Call Trace:
[258364.615234] <IRQ>
[258364.626239] [<ffffffff81713aa0>] __pskb_pull_tail+0x50/0x350
[258364.658220] [<ffffffff818051b6>] _decode_session6+0x2e6/0x490
[258364.690802] [<ffffffff817b9653>] __xfrm_decode_session+0x33/0x50
[258364.724086] [<ffffffff817ef8e4>] icmpv6_route_lookup+0xf4/0x1d0
[258364.756501] [<ffffffff817f0341>] icmp6_send+0x5f1/0x920
[258364.786304] [<ffffffff810dfa3a>] ? handle_edge_irq+0xba/0x180
[258364.818506] [<ffffffffc056e230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[258364.856875] [<ffffffff81808fe1>] icmpv6_send+0x21/0x30
[258364.884947] [<ffffffff817f7250>] ip6_expire_frag_queue+0xe0/0x120
[258364.918555] [<ffffffffc056e24f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6]
[258364.957884] [<ffffffff810edd87>] call_timer_fn+0x37/0x140
[258364.988097] [<ffffffffc056e230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[258365.027226] [<ffffffff810ee133>] run_timer_softirq+0x213/0x320
[258365.065587] [<ffffffff81087625>] __do_softirq+0xe5/0x2b0
[258365.100276] [<ffffffff81087a26>] irq_exit+0x96/0xa0
[258365.132826] [<ffffffff81830c40>] smp_apic_timer_interrupt+0x50/0x70
[258365.172450] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[258365.210589] <EOI>
[258365.221544] [<ffffffff816c2339>] ? cpuidle_enter_state+0xc9/0x280
[258365.266485] [<ffffffff816c232b>] ? cpuidle_enter_state+0xbb/0x280
[258365.305657] [<ffffffff816c2527>] cpuidle_enter+0x17/0x20
[258365.339899] [<ffffffff810c5752>] call_cpuidle+0x32/0x60
[258365.372832] [<ffffffff816c2509>] ? cpuidle_select+0x19/0x20
[258365.407697] [<ffffffff810c5a10>] cpu_startup_entry+0x290/0x350
[258365.443997] [<ffffffff81820f0c>] rest_init+0x7c/0x80
[258365.476503] [<ffffffff81f6b0f7>] start_kernel+0x4a7/0x4b4
[258365.511376] [<ffffffff81f6aa3b>] ? set_init_arg+0x55/0x55
[258365.546509] [<ffffffff81f6a120>] ? early_idt_handler_array+0x120/0x120
[258365.585481] [<ffffffff81f6a5ee>] x86_64_start_reservations+0x2a/0x2c
[258365.623803] [<ffffffff81f6a71d>] x86_64_start_kernel+0x12d/0x13a
[258365.660778] Code: ff ff ff 90 e9 54 ff ff ff 48 89 d7 48 89 55 c8 e8 0f b1 a8 ff 84 c0 48 8b 55 c8 75 97 eb 91 41 81 cf 00 20 00 00 e9 2a fe ff ff <0f> 0b 0f 0b 44 89 fe 4c 89 ef e8 7a ec ff ff 85 c0 74 12 48 89
[258365.772214] RIP [<ffffffff81711d07>] pskb_expand_head+0x227/0x260
[258365.811291] RSP <ffff88011ba03b90>
[258365.850327] ---[ end trace 4625108fd52df30d ]---
[258365.879532] Kernel panic - not syncing: Fatal exception in interrupt
[258365.918544] Kernel Offset: disabled
[258365.940516] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[258365.983012] ------------[ cut here ]------------
[258366.012138] WARNING: CPU: 0 PID: 0 at /build/linux-lts-xenial-YWfqtJ/linux-lts-xenial-4.4.0/arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x63/0x70()
[258366.095628] Modules linked in: ip6table_raw ip6table_mangle ip6t_REJECT nf_reject_ipv6 iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_mark iptable_mangle xt_comment ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_limit xt_tcpudp xt_iprange ip_set_hash_net hpwdt nfnetlink_log ip6table_filter ip6_tables xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter ip_tables x_tables dm_crypt intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul nf_conntrack_irc nf_conntrack_tftp nf_conntrack_ftp crc32_pclmul ghash_clmulni_intel hpilo aesni_intel aes_x86_64 nf_conntrack_ipv6 acpi_power_meter mac_hid lrw 8250_fintek nf_defrag_ipv6 shpchp gpio_ich gf128mul serio_raw nf_conntrack_ipv4 i7core_edac glue_helper nf_defrag_ipv4 edac_core nf_conntrack ablk_helper cryptd lpc_ich ipmi_ssif ipmi_devintf ipmi_si ipmi_msghandler ipip tunnel4 ip_tunnel 8021q garp mrp stp llc bonding raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon ixgbe i2c_algo_bit ttm dca drm_kms_helper vxlan syscopyarea ip6_udp_tunnel sysfillrect udp_tunnel sysimgblt ptp fb_sys_fops hpsa pps_core psmouse drm mdio bnx2 scsi_transport_sas fjes
[258366.718283] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D 4.4.0-144-generic #170~14.04.1-Ubuntu
[258366.775794] Hardware name: HP ProLiant DL360 G7, BIOS P68 07/02/2013
[258366.815370] 0000000000000000 ffff88011ba03630 ffffffff813ee977 0000000000000000
[258366.860330] ffffffff81cb16a0 ffff88011ba03668 ffffffff810829c6 0000000000000003
[258366.905824] ffff88021f857300 0000000000000000 0000000000000000 00000000000103a8
[258366.950305] Call Trace:
[258366.968301] <IRQ> [<ffffffff813ee977>] dump_stack+0x63/0x8c
[258367.003999] [<ffffffff810829c6>] warn_slowpath_common+0x86/0xc0
[258367.040181] [<ffffffff81082aba>] warn_slowpath_null+0x1a/0x20
[258367.075756] [<ffffffff8104f903>] native_smp_send_reschedule+0x63/0x70
[258367.117247] [<ffffffff810c0239>] trigger_load_balance+0x149/0x1f0
[258367.156319] [<ffffffff810afb8a>] scheduler_tick+0xaa/0xe0
[258367.191036] [<ffffffff810ffb60>] ? tick_sched_handle.isra.15+0x60/0x60
[258367.231378] [<ffffffff810f03e1>] update_process_times+0x51/0x60
[258367.269277] [<ffffffff810ffb25>] tick_sched_handle.isra.15+0x25/0x60
[258367.310291] [<ffffffff810ffb9d>] tick_sched_timer+0x3d/0x70
[258367.345839] [<ffffffff810f0fa3>] __hrtimer_run_queues+0xf3/0x270
[258367.385068] [<ffffffff810f1458>] hrtimer_interrupt+0xa8/0x1a0
[258367.422105] [<ffffffff8105264b>] local_apic_timer_interrupt+0x3b/0x60
[258367.462399] [<ffffffff81830c3b>] smp_apic_timer_interrupt+0x4b/0x70
[258367.501999] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[258367.539699] [<ffffffff8118c558>] ? panic+0x1d7/0x221
[258367.572401] [<ffffffff81030b7f>] oops_end+0xff/0x100
[258367.604987] [<ffffffff810310cb>] die+0x4b/0x70
[258367.633915] [<ffffffff8102e02d>] do_trap+0x13d/0x150
[258367.665701] [<ffffffff8102e397>] do_error_trap+0x77/0xe0
[258367.699734] [<ffffffff81711d07>] ? pskb_expand_head+0x227/0x260
[258367.737141] [<ffffffff8182cb80>] ? _raw_read_unlock_bh+0x20/0x30
[258367.775096] [<ffffffff817de13d>] ? ip6_pol_route.isra.39+0x2dd/0x5e0
[258367.815744] [<ffffffff8102e7f0>] do_invalid_op+0x20/0x30
[258367.850598] [<ffffffff8182f89e>] invalid_op+0x1e/0x30
[258367.883143] [<ffffffff81711d07>] ? pskb_expand_head+0x227/0x260
[258367.920118] [<ffffffff81713aa0>] __pskb_pull_tail+0x50/0x350
[258367.956178] [<ffffffff818051b6>] _decode_session6+0x2e6/0x490
[258367.992355] [<ffffffff817b9653>] __xfrm_decode_session+0x33/0x50
[258368.030623] [<ffffffff817ef8e4>] icmpv6_route_lookup+0xf4/0x1d0
[258368.066652] [<ffffffff817f0341>] icmp6_send+0x5f1/0x920
[258368.099028] [<ffffffff810dfa3a>] ? handle_edge_irq+0xba/0x180
[258368.135157] [<ffffffffc056e230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[258368.178097] [<ffffffff81808fe1>] icmpv6_send+0x21/0x30
[258368.210498] [<ffffffff817f7250>] ip6_expire_frag_queue+0xe0/0x120
[258368.247215] [<ffffffffc056e24f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6]
[258368.291091] [<ffffffff810edd87>] call_timer_fn+0x37/0x140
[258368.323741] [<ffffffffc056e230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[258368.366009] [<ffffffff810ee133>] run_timer_softirq+0x213/0x320
[258368.402165] [<ffffffff81087625>] __do_softirq+0xe5/0x2b0
[258368.436015] [<ffffffff81087a26>] irq_exit+0x96/0xa0
[258368.465881] [<ffffffff81830c40>] smp_apic_timer_interrupt+0x50/0x70
[258368.503227] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[258368.539120] <EOI> [<ffffffff816c2339>] ? cpuidle_enter_state+0xc9/0x280
[258368.579263] [<ffffffff816c232b>] ? cpuidle_enter_state+0xbb/0x280
[258368.616444] [<ffffffff816c2527>] cpuidle_enter+0x17/0x20
[258368.648794] [<ffffffff810c5752>] call_cpuidle+0x32/0x60
[258368.680904] [<ffffffff816c2509>] ? cpuidle_select+0x19/0x20
[258368.715039] [<ffffffff810c5a10>] cpu_startup_entry+0x290/0x350
[258368.750114] [<ffffffff81820f0c>] rest_init+0x7c/0x80
[258368.781110] [<ffffffff81f6b0f7>] start_kernel+0x4a7/0x4b4
[258368.814687] [<ffffffff81f6aa3b>] ? set_init_arg+0x55/0x55
[258368.847884] [<ffffffff81f6a120>] ? early_idt_handler_array+0x120/0x120
[258368.887214] [<ffffffff81f6a5ee>] x86_64_start_reservations+0x2a/0x2c
[258368.925863] [<ffffffff81f6a71d>] x86_64_start_kernel+0x12d/0x13a
[258368.962150] ---[ end trace 4625108fd52df30e ]---

Crash #2 (after approx 10.5 days of uptime):

[904742.422354] ------------[ cut here ]------------
[904742.446596] kernel BUG at /build/linux-lts-xenial-YWfqtJ/linux-lts-xenial-4.4.0/net/core/skbuff.c:1207!
[904742.496670] invalid opcode: 0000 [#1] SMP
[904742.519107] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid ip6table_raw ip6table_mangle ip6t_REJECT nf_reject_ipv6 iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_mark iptable_mangle xt_comment ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_limit xt_tcpudp xt_iprange ip_set_hash_net hpwdt nfnetlink_log ip6table_filter ip6_tables xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter ip_tables x_tables dm_crypt nf_conntrack_irc nf_conntrack_tftp nf_conntrack_ftp nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipmi_devintf ipmi_ssif intel_powerclamp coretemp gpio_ich kvm_intel kvm hpilo irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel 8250_fintek aes_x86_64 lrw gf128mul i7core_edac glue_helper ablk_helper cryptd serio_raw shpchp edac_core lpc_ich acpi_power_meter mac_hid ipmi_si ipmi_msghandler ipip tunnel4 ip_tunnel 8021q garp mrp stp llc bonding raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon ixgbe i2c_algo_bit ttm dca drm_kms_helper vxlan syscopyarea ip6_udp_tunnel sysfillrect udp_tunnel sysimgblt ptp fb_sys_fops pps_core hpsa psmouse drm bnx2 mdio scsi_transport_sas fjes
[904743.129561] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.0-144-generic #170~14.04.1-Ubuntu
[904743.174542] Hardware name: HP ProLiant DL360 G7, BIOS P68 07/02/2013
[904743.208594] task: ffff88011a8ff2c0 ti: ffff88011a900000 task.ti: ffff88011a900000
[904743.248704] RIP: 0010:[<ffffffff81711d07>] [<ffffffff81711d07>] pskb_expand_head+0x227/0x260
[904743.295812] RSP: 0018:ffff88011ba43b90 EFLAGS: 00010202
[904743.325174] RAX: 0000000000000002 RBX: ffff88001556d700 RCX: 0000000002080020
[904743.364584] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff88001556d700
[904743.403388] RBP: ffff88011ba43bc8 R08: 0000000000000001 R09: ffff88011a720cc0
[904743.443277] R10: ffffffffffffffff R11: 0000000000000040 R12: 0000000000000000
[904743.483003] R13: ffff88001556d700 R14: 0000000000000001 R15: ffff880081b041ce
[904743.522145] FS: 0000000000000000(0000) GS:ffff88011ba40000(0000) knlGS:0000000000000000
[904743.567296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[904743.599236] CR2: 00007f421b3fb151 CR3: 0000000001e0c000 CR4: 0000000000020670
[904743.639170] Stack:
[904743.650908] ffff88011ba43de0 ffff88011ba43de0 ffff88001556d700 ffff88011ba43ca0
[904743.692211] 0000000000000001 0000000000000001 ffff880081b041ce ffff88011ba43c10
[904743.732787] ffffffff81713aa0 ffff880034673328 0000000100000005 ffff88001556d700
[904743.773609] Call Trace:
[904743.787351] <IRQ>
[904743.797682] [<ffffffff81713aa0>] __pskb_pull_tail+0x50/0x350
[904743.829594] [<ffffffff818051b6>] _decode_session6+0x2e6/0x490
[904743.861962] [<ffffffff817b9653>] __xfrm_decode_session+0x33/0x50
[904743.895682] [<ffffffff817ef8e4>] icmpv6_route_lookup+0xf4/0x1d0
[904743.929316] [<ffffffff817f0341>] icmp6_send+0x5f1/0x920
[904743.958366] [<ffffffff81724853>] ? netif_receive_skb_internal+0x23/0x80
[904743.995988] [<ffffffff811ebabd>] ? kmem_cache_free+0x1dd/0x200
[904744.028305] [<ffffffffc048b230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[904744.067020] [<ffffffff81808fe1>] icmpv6_send+0x21/0x30
[904744.095285] [<ffffffff817f7250>] ip6_expire_frag_queue+0xe0/0x120
[904744.128497] [<ffffffffc048b24f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6]
[904744.168251] [<ffffffff810edd87>] call_timer_fn+0x37/0x140
[904744.204546] [<ffffffffc048b230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[904744.248459] [<ffffffff810ee133>] run_timer_softirq+0x213/0x320
[904744.285473] [<ffffffff81087625>] __do_softirq+0xe5/0x2b0
[904744.320390] [<ffffffff81087a26>] irq_exit+0x96/0xa0
[904744.352580] [<ffffffff81830c40>] smp_apic_timer_interrupt+0x50/0x70
[904744.391918] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[904744.429091] <EOI>
[904744.440129] [<ffffffff816c2339>] ? cpuidle_enter_state+0xc9/0x280
[904744.484424] [<ffffffff816c232b>] ? cpuidle_enter_state+0xbb/0x280
[904744.522613] [<ffffffff816c2527>] cpuidle_enter+0x17/0x20
[904744.557681] [<ffffffff810c5752>] call_cpuidle+0x32/0x60
[904744.591840] [<ffffffff816c2509>] ? cpuidle_select+0x19/0x20
[904744.627778] [<ffffffff810c5a10>] cpu_startup_entry+0x290/0x350
[904744.664245] [<ffffffff810508cc>] start_secondary+0x16c/0x190
[904744.700506] Code: ff ff ff 90 e9 54 ff ff ff 48 89 d7 48 89 55 c8 e8 0f b1 a8 ff 84 c0 48 8b 55 c8 75 97 eb 91 41 81 cf 00 20 00 00 e9 2a fe ff ff <0f> 0b 0f 0b 44 89 fe 4c 89 ef e8 7a ec ff ff 85 c0 74 12 48 89
[904744.813128] RIP [<ffffffff81711d07>] pskb_expand_head+0x227/0x260
[904744.853577] RSP <ffff88011ba43b90>
[904744.894718] ---[ end trace b25a46ce8b731c75 ]---
[904744.925568] Kernel panic - not syncing: Fatal exception in interrupt
[904744.963823] Kernel Offset: disabled
[904744.986232] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[904745.029237] ------------[ cut here ]------------
[904745.058307] WARNING: CPU: 2 PID: 0 at /build/linux-lts-xenial-YWfqtJ/linux-lts-xenial-4.4.0/arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x63/0x70()
[904745.141586] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid ip6table_raw ip6table_mangle ip6t_REJECT nf_reject_ipv6 iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_mark iptable_mangle xt_comment ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_limit xt_tcpudp xt_iprange ip_set_hash_net hpwdt nfnetlink_log ip6table_filter ip6_tables xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter ip_tables x_tables dm_crypt nf_conntrack_irc nf_conntrack_tftp nf_conntrack_ftp nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipmi_devintf ipmi_ssif intel_powerclamp coretemp gpio_ich kvm_intel kvm hpilo irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel 8250_fintek aes_x86_64 lrw gf128mul i7core_edac glue_helper ablk_helper cryptd serio_raw shpchp edac_core lpc_ich acpi_power_meter mac_hid ipmi_si ipmi_msghandler ipip tunnel4 ip_tunnel 8021q garp mrp stp llc bonding raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon ixgbe i2c_algo_bit ttm dca drm_kms_helper vxlan syscopyarea ip6_udp_tunnel sysfillrect udp_tunnel sysimgblt ptp fb_sys_fops pps_core hpsa psmouse drm bnx2 mdio scsi_transport_sas fjes
[904745.792265] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 4.4.0-144-generic #170~14.04.1-Ubuntu
[904745.849608] Hardware name: HP ProLiant DL360 G7, BIOS P68 07/02/2013
[904745.889008] 0000000000000000 ffff88011ba43630 ffffffff813ee977 0000000000000000
[904745.934005] ffffffff81cb16a0 ffff88011ba43668 ffffffff810829c6 000000000000000b
[904745.978889] ffff88021f957300 0000000000000002 0000000000000002 00000000000103a8
[904746.023956] Call Trace:
[904746.042665] <IRQ> [<ffffffff813ee977>] dump_stack+0x63/0x8c
[904746.079135] [<ffffffff810829c6>] warn_slowpath_common+0x86/0xc0
[904746.117250] [<ffffffff81082aba>] warn_slowpath_null+0x1a/0x20
[904746.153396] [<ffffffff8104f903>] native_smp_send_reschedule+0x63/0x70
[904746.193309] [<ffffffff810c0239>] trigger_load_balance+0x149/0x1f0
[904746.232264] [<ffffffff810afb8a>] scheduler_tick+0xaa/0xe0
[904746.266577] [<ffffffff810ffb60>] ? tick_sched_handle.isra.15+0x60/0x60
[904746.307624] [<ffffffff810f03e1>] update_process_times+0x51/0x60
[904746.344548] [<ffffffff810ffb25>] tick_sched_handle.isra.15+0x25/0x60
[904746.383846] [<ffffffff810ffb9d>] tick_sched_timer+0x3d/0x70
[904746.419496] [<ffffffff810f0fa3>] __hrtimer_run_queues+0xf3/0x270
[904746.456808] [<ffffffff810f1458>] hrtimer_interrupt+0xa8/0x1a0
[904746.493033] [<ffffffff8105264b>] local_apic_timer_interrupt+0x3b/0x60
[904746.533936] [<ffffffff81830c3b>] smp_apic_timer_interrupt+0x4b/0x70
[904746.573266] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[904746.610493] [<ffffffff8118c558>] ? panic+0x1d7/0x221
[904746.642355] [<ffffffff81030b7f>] oops_end+0xff/0x100
[904746.674523] [<ffffffff810310cb>] die+0x4b/0x70
[904746.703901] [<ffffffff8102e02d>] do_trap+0x13d/0x150
[904746.736623] [<ffffffff8102e397>] do_error_trap+0x77/0xe0
[904746.770575] [<ffffffff81711d07>] ? pskb_expand_head+0x227/0x260
[904746.807344] [<ffffffff8182cb80>] ? _raw_read_unlock_bh+0x20/0x30
[904746.845076] [<ffffffff817de13d>] ? ip6_pol_route.isra.39+0x2dd/0x5e0
[904746.884317] [<ffffffff8102e7f0>] do_invalid_op+0x20/0x30
[904746.918813] [<ffffffff8182f89e>] invalid_op+0x1e/0x30
[904746.952198] [<ffffffff81711d07>] ? pskb_expand_head+0x227/0x260
[904746.989190] [<ffffffff81713aa0>] __pskb_pull_tail+0x50/0x350
[904747.024669] [<ffffffff818051b6>] _decode_session6+0x2e6/0x490
[904747.061384] [<ffffffff817b9653>] __xfrm_decode_session+0x33/0x50
[904747.099629] [<ffffffff817ef8e4>] icmpv6_route_lookup+0xf4/0x1d0
[904747.136625] [<ffffffff817f0341>] icmp6_send+0x5f1/0x920
[904747.170345] [<ffffffff81724853>] ? netif_receive_skb_internal+0x23/0x80
[904747.211048] [<ffffffff811ebabd>] ? kmem_cache_free+0x1dd/0x200
[904747.246771] [<ffffffffc048b230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[904747.290157] [<ffffffff81808fe1>] icmpv6_send+0x21/0x30
[904747.322350] [<ffffffff817f7250>] ip6_expire_frag_queue+0xe0/0x120
[904747.360290] [<ffffffffc048b24f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6]
[904747.403474] [<ffffffff810edd87>] call_timer_fn+0x37/0x140
[904747.436740] [<ffffffffc048b230>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6]
[904747.479154] [<ffffffff810ee133>] run_timer_softirq+0x213/0x320
[904747.515067] [<ffffffff81087625>] __do_softirq+0xe5/0x2b0
[904747.547965] [<ffffffff81087a26>] irq_exit+0x96/0xa0
[904747.578018] [<ffffffff81830c40>] smp_apic_timer_interrupt+0x50/0x70
[904747.615196] [<ffffffff8182e3bc>] apic_timer_interrupt+0xcc/0xe0
[904747.649664] <EOI> [<ffffffff816c2339>] ? cpuidle_enter_state+0xc9/0x280
[904747.690024] [<ffffffff816c232b>] ? cpuidle_enter_state+0xbb/0x280
[904747.728252] [<ffffffff816c2527>] cpuidle_enter+0x17/0x20
[904747.761598] [<ffffffff810c5752>] call_cpuidle+0x32/0x60
[904747.793746] [<ffffffff816c2509>] ? cpuidle_select+0x19/0x20
[904747.827736] [<ffffffff810c5a10>] cpu_startup_entry+0x290/0x350
[904747.863007] [<ffffffff810508cc>] start_secondary+0x16c/0x190
[904747.897172] ---[ end trace b25a46ce8b731c76 ]---

[Tore Anderson (toreanderson)]

And a third crash, this time after upgrading to 4.4.0-146-generic. Uptime just 21h41m. Backtrace looks the same.

I've now reverted back to 4.4.0-116-generic. I'll let you know if we experience similar crashes with this version.

[78056.952451] ------------[ cut here ]------------
[78056.975336] kernel BUG at /build/linux-lts-xenial-Y8MnSS/linux-lts-xenial-4.4.0/net/core/skbuff.c:1207!
[78057.024383] invalid opcode: 0000 [#1] SMP
[78057.046515] Modules linked in: ip6table_raw ip6table_mangle ip6t_REJECT nf_reject_ipv6 iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat xt_mark iptable_mangle xt_comment ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_limit xt_tcpudp xt_iprange ip_set_hash_net hpwdt nfnetlink_log ip6table_filter ip6_tables xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter ip_tables x_tables dm_crypt nf_conntrack_irc nf_conntrack_tftp nf_conntrack_ftp nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 intel_powerclamp nf_conntrack coretemp kvm_intel kvm ipmi_ssif ipmi_devintf irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 hpilo lrw gpio_ich gf128mul ipip glue_helper shpchp tunnel4 ipmi_si ip_tunnel ablk_helper serio_raw ipmi_msghandler cryptd i7core_edac 8250_fintek edac_core 8021q lpc_ich acpi_power_meter garp mrp stp llc mac_hid bonding raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper ixgbe syscopyarea dca sysfillrect sysimgblt vxlan fb_sys_fops ip6_udp_tunnel udp_tunnel ptp hpsa drm pps_core psmouse bnx2 mdio scsi_transport_sas fjes
[78057.630237] CPU: 14 PID: 0 Comm: swapper/14 Not tainted 4.4.0-146-generic #172~14.04.1-Ubuntu
[78057.676457] Hardware name: HP ProLiant DL360 G7, BIOS P68 07/02/2013
[78057.710242] task: ffff88011a90cc80 ti: ffff88011a928000 task.ti: ffff88011a928000
[78057.751071] RIP: 0010:[<ffffffff81711cd7>] [<ffffffff81711cd7>] pskb_expand_head+0x227/0x260
[78057.797573] RSP: 0018:ffff88011bbc3b90 EFLAGS: 00010202
[78057.825843] RAX: 0000000000000002 RBX: ffff8800b8641000 RCX: 0000000002080020
[78057.864650] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff8800b8641000
[78057.902584] RBP: ffff88011bbc3bc8 R08: 0000000000000001 R09: ffff88011a5f6e80
[78057.940809] R10: ffffffffffffffff R11: 0000000000000040 R12: 0000000000000000
[78057.979197] R13: ffff8800b8641000 R14: 0000000000000001 R15: ffff8800b2f5c6ce
[78058.018261] FS: 0000000000000000(0000) GS:ffff88011bbc0000(0000) knlGS:0000000000000000
[78058.061413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[78058.090850] CR2: 00007f6086a9afb8 CR3: 0000000001e0c000 CR4: 0000000000020670
[78058.128810] Stack:
[78058.140078] ffff88011bbc3de0 ffff88011bbc3de0 ffff8800b8641000 ffff88011bbc3ca0
[78058.179734] 0000000000000001 0000000000000001 ffff8800b2f5c6ce ffff88011bbc3c10
[78058.219346] ffffffff81713a70 ffff880218455f28 0000000100000005 ffff8800b8641000
[78058.258709] Call Trace:
[78058.272106] <IRQ>
[78058.282744] [<ffffffff81713a70>] __pskb_pull_tail+0x50/0x350
[78058.314577] [<ffffffff81805456>] _decode_session6+0x2e6/0x490
[78058.346575] [<ffffffff...

[Peter Struhar (struhlo)]

hello,

we have tested in our environment with 16.04 update from 4.4.0-143 to 4.4.0-146 and it seems, based on your original comment, that some fix, implemented probably in kernel versions 144-145, might be contributing to the kernel panic behaviour.

We are experiencing this only on one system from multiple upgraded (all upgraded from versions 142 and 143). Reverting to 4.4.0-143 stabilized the reoccuring behaviour on problematic system0, and it runs for 2 days stable now. Problem happened twice in 5 hours with kernel 146. System ran stable before over month, from last patching with kernel 4.4.0-143.

I can probably replicate this if needed.

s.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tore Anderson (toreanderson)]

Hi Peter, and thanks for confirming the bug.

We first experienced this issue in 4.4.0-144 so if you're saying 4.4.0-143 is stable that would mean the bug was introduced in 4.4.0-144.

Tore

[Peter Struhar (struhlo)]

hello Tore,

yes, you are right, 144. In the meantime, we experienced this also on second system running 5 days before kerne panic, so in case some dumps would be needed, we can probably provide those (if we are pointed to procedure how to generate them). Now it is running 143 but we can switch it also with previous panicking system to 146 for test.

thanks
s.

[David Soltero (juanitod)]

Hi,

We have also experience the bug with 4.4.0-146 and 4.4.0-148 on 2 systems. We confirm 143 and older does not seem to be affected.

Best

David

[Sam (samuel-hameau)]

Same here with 4.4.0-144 and 4.4.0-148.
Kernel 4.4.0-143 is not affected

[Tore Anderson (toreanderson)]

We had another server running 4.4.0-148-generic crash just now. It has a different role then the firewalls that we originally saw the crash with. After an automatic reboot, it got back up with 4.4.0-150-generic (which had been installed at an earlier stage but not rebooted into), and crashed twice in rapid succession. I'm including the traces printed to the serial console below:

Crash #1:

[760425.631142] ------------[ cut here ]------------
[760425.655558] kernel BUG at /build/linux-JhELCR/linux-4.4.0/net/core/skbuff.c:1207!
[760425.698342] invalid opcode: 0000 [#1] SMP
[760425.722139] Modules linked in: cpuid xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables hpwdt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp xt_multiport xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables bonding intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul gpio_ich ghash_clmulni_intel aesni_intel ftdi_sio aes_x86_64 lrw cdc_ether gf128mul usbserial usbnet mii glue_helper input_leds hpilo joydev ablk_helper i7core_edac cryptd edac_core shpchp serio_raw 8250_fintek acpi_power_meter lpc_ich ipmi_ssif mac_hid ipmi_devintf ipmi_si ipmi_msghandler nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack lp parport autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ixgbe dca amdkfd amd_iommu_v2 radeon i2c_algo_bit hid_generic ttm drm_kms_helper syscopyarea sysfillrect vxlan ip6_udp_tunnel sysimgblt fb_sys_fops udp_tunnel usbhid uas ptp hpsa pps_core psmouse drm usb_storage hid bnx2 mdio scsi_transport_sas fjes
[760426.325859] CPU: 9 PID: 0 Comm: swapper/9 Tainted: G I 4.4.0-148-generic #174-Ubuntu
[760426.377154] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018
[760426.413661] task: ffff880c0461d940 ti: ffff880c04634000 task.ti: ffff880c04634000
[760426.456870] RIP: 0010:[<ffffffff81740913>] [<ffffffff81740913>] pskb_expand_head+0x243/0x250
[760426.504553] RSP: 0018:ffff880c1f903b80 EFLAGS: 00010202
[760426.535588] RAX: 0000000000000002 RBX: ffff880c0007ed00 RCX: 0000000002080020
[760426.577271] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff880c0007ed00
[760426.618845] RBP: ffff880c1f903bb8 R08: 0000000000000001 R09: 000000000000000a
[760426.660361] R10: ffff880bffcf6a00 R11: 0000000000000000 R12: ffff880c0007ed00
[760426.702862] R13: 0000000000000001 R14: 0000000000000011 R15: ffff880c0007ed00
[760426.743881] FS: 0000000000000000(0000) GS:ffff880c1f900000(0000) knlGS:0000000000000000
[760426.789899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[760426.822597] CR2: 00007ffccac51ff8 CR3: 0000000001e0a000 CR4: 0000000000020670
[760426.863053] Stack:
[760426.874826] ffffffff81efb140 ffff880c1f903db0 ffff880c0007ed00 ffff880c0007ed00
[760426.917091] 0000000000000001 0000000000000011 0000000000000001 ffff880c1f903c00
[760426.960119] ffffffff81740e10 ffff880c002d29c0 0000000100000001 ffff880c0007ed00
[760427.003239] Call Trace:
[760427.017831] <IRQ>
[76...