Kernel warning after upgrade to 4.15.0.211.194

[Gábor 'Morc' KORMOS]

After rebooting into 4.15.0.211.194 on Ubuntu 18.04.6 the kernel started to produce the below warning. It did not seem to cause any major issue. Revering to 4.15.0.210 stopped the warning. The server is a Droplet at DigitalOcean with their monitoring/management agent installed.

May 21 19:09:59 esolr kernel: [ 18.787737] ------------[ cut here ]------------
May 21 19:09:59 esolr kernel: [ 18.787803] WARNING: CPU: 0 PID: 1027 at /build/linux-RlFMDZ/linux-4.15.0/net/core/stream.c:212 sk_stream_kill_queues+0xed/0x110
May 21 19:09:59 esolr kernel: [ 18.787804] Modules linked in: isofs xt_set nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_conntrack nf_conntrack ip_set_hash_net ip_set nfnetlink nls_iso8859_1 kvm_intel kvm binfmt_misc irqbypass input_leds joydev serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse virtio_blk virtio_scsi virtio_net
May 21 19:09:59 esolr kernel: [ 18.787835] CPU: 0 PID: 1027 Comm: unattended-upgr Not tainted 4.15.0-211-generic #222-Ubuntu
May 21 19:09:59 esolr kernel: [ 18.787836] Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
May 21 19:09:59 esolr kernel: [ 18.787839] RIP: 0010:sk_stream_kill_queues+0xed/0x110
May 21 19:09:59 esolr kernel: [ 18.787840] RSP: 0000:ffff94bf7fc038c0 EFLAGS: 00010282
May 21 19:09:59 esolr kernel: [ 18.787841] RAX: 0000000000000024 RBX: ffff94bf79565500 RCX: 0000000000000006
May 21 19:09:59 esolr kernel: [ 18.787842] RDX: 0000000000000000 RSI: ffff94bf7fc1b4d8 RDI: ffff94bf7fc1b4d0
May 21 19:09:59 esolr kernel: [ 18.787842] RBP: ffff94bf7fc038d0 R08: 0000000000000222 R09: 0000000000000004
May 21 19:09:59 esolr kernel: [ 18.787843] R10: ffff94bf7fc03860 R11: 0000000000000001 R12: ffff94bf795655c8
May 21 19:09:59 esolr kernel: [ 18.787844] R13: 0000000000000006 R14: 000000000000003b R15: 0000000000000000
May 21 19:09:59 esolr kernel: [ 18.787845] FS: 00007f04071d4740(0000) GS:ffff94bf7fc00000(0000) knlGS:0000000000000000
May 21 19:09:59 esolr kernel: [ 18.787846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 21 19:09:59 esolr kernel: [ 18.787846] CR2: 00007f0401995e60 CR3: 0000000067586002 CR4: 00000000001606f0
May 21 19:09:59 esolr kernel: [ 18.787850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 21 19:09:59 esolr kernel: [ 18.787851] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 21 19:09:59 esolr kernel: [ 18.787851] Call Trace:
May 21 19:09:59 esolr kernel: [ 18.787853] <IRQ>
May 21 19:09:59 esolr kernel: [ 18.787866] inet_csk_destroy_sock+0x59/0x150
May 21 19:09:59 esolr kernel: [ 18.787868] tcp_done+0x96/0xa0
May 21 19:09:59 esolr kernel: [ 18.787872] tcp_time_wait+0x1be/0x280
May 21 19:09:59 esolr kernel: [ 18.787874] tcp_fin+0x16b/0x180
May 21 19:09:59 esolr kernel: [ 18.787876] tcp_data_queue+0x594/0xc20
May 21 19:09:59 esolr kernel: [ 18.787881] tcp_rcv_state_process+0x5dd/0xe70
May 21 19:09:59 esolr kernel: [ 18.787887] ? __slab_alloc+0x20/0x40
May 21 19:09:59 esolr kernel: [ 18.787889] ? kmem_cache_alloc+0x161/0x1c0
May 21 19:09:59 esolr kernel: [ 18.787891] ? skb_clone+0x56/0xc0
May 21 19:09:59 esolr kernel: [ 18.787897] tcp_v6_do_rcv+0x1ca/0x440
May 21 19:09:59 esolr kernel: [ 18.787898] ? tcp_v6_do_rcv+0x1ca/0x440
May 21 19:09:59 esolr kernel: [ 18.787899] tcp_v6_rcv+0x9b6/0xa60
May 21 19:09:59 esolr kernel: [ 18.787907] ? update_load_avg+0x5f9/0x780
May 21 19:09:59 esolr kernel: [ 18.787909] ip6_input_finish+0xcc/0x470
May 21 19:09:59 esolr kernel: [ 18.787911] ip6_input+0x3f/0xb0
May 21 19:09:59 esolr kernel: [ 18.787914] ? ip6_dst_check+0xb1/0xf0
May 21 19:09:59 esolr kernel: [ 18.787915] ip6_rcv_finish+0xd8/0x120
May 21 19:09:59 esolr kernel: [ 18.787916] ipv6_rcv+0x347/0x510
May 21 19:09:59 esolr kernel: [ 18.787917] ? __pskb_pull_tail+0x7f/0x4b0
May 21 19:09:59 esolr kernel: [ 18.787918] ? skb_copy_bits+0x141/0x280
May 21 19:09:59 esolr kernel: [ 18.787920] ? __kmalloc_node_track_caller+0x216/0x2b0
May 21 19:09:59 esolr kernel: [ 18.787923] __netif_receive_skb_core+0x432/0xb80
May 21 19:09:59 esolr kernel: [ 18.787929] ? tcp6_gro_receive+0x137/0x1a0
May 21 19:09:59 esolr kernel: [ 18.787931] ? ipv6_gro_receive+0x1a6/0x350
May 21 19:09:59 esolr kernel: [ 18.787932] __netif_receive_skb+0x18/0x60
May 21 19:09:59 esolr kernel: [ 18.787933] ? __netif_receive_skb+0x18/0x60
May 21 19:09:59 esolr kernel: [ 18.787935] netif_receive_skb_internal+0x45/0xe0
May 21 19:09:59 esolr kernel: [ 18.787937] napi_gro_receive+0xc5/0xf0
May 21 19:09:59 esolr kernel: [ 18.787941] receive_buf+0x346/0x17a0 [virtio_net]
May 21 19:09:59 esolr kernel: [ 18.787942] ? check_preempt_curr+0x2d/0x90
May 21 19:09:59 esolr kernel: [ 18.787943] ? ttwu_do_wakeup+0x1e/0x140
May 21 19:09:59 esolr kernel: [ 18.787944] ? ttwu_do_activate+0x77/0x80
May 21 19:09:59 esolr kernel: [ 18.787960] ? vring_unmap_one+0x1b/0x80
May 21 19:09:59 esolr kernel: [ 18.787962] ? detach_buf+0x71/0x120
May 21 19:09:59 esolr kernel: [ 18.787964] virtnet_poll+0x152/0x2f7 [virtio_net]
May 21 19:09:59 esolr kernel: [ 18.787966] net_rx_action+0x140/0x3d0
May 21 19:09:59 esolr kernel: [ 18.787975] __do_softirq+0xe4/0x2d4
May 21 19:09:59 esolr kernel: [ 18.787979] irq_exit+0xc5/0xd0
May 21 19:09:59 esolr kernel: [ 18.787981] do_IRQ+0x86/0xe0
May 21 19:09:59 esolr kernel: [ 18.787982] common_interrupt+0x90/0x90
May 21 19:09:59 esolr kernel: [ 18.787983] </IRQ>
May 21 19:09:59 esolr kernel: [ 18.787984] RIP: 0033:0x57023c
May 21 19:09:59 esolr kernel: [ 18.787985] RSP: 002b:00007fffc87d1d80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffce
May 21 19:09:59 esolr kernel: [ 18.787986] RAX: 0000000000004000 RBX: 00007f0403dc4998 RCX: 0000000000003fff
May 21 19:09:59 esolr kernel: [ 18.787987] RDX: 00007f0403e96010 RSI: 00007f0403e96038 RDI: 00000000009d4540
May 21 19:09:59 esolr kernel: [ 18.787987] RBP: 00007f04071960d8 R08: 00007f0403e96038 R09: 00007f0403e9e038
May 21 19:09:59 esolr kernel: [ 18.787988] R10: 00007f0403dc49c8 R11: 0000000000000065 R12: 0000000000000fe0
May 21 19:09:59 esolr kernel: [ 18.787988] R13: 5c66abb0f49a0fe0 R14: 00007f0403dc4998 R15: 0000000000000857
May 21 19:09:59 esolr kernel: [ 18.787989] Code: 01 00 00 85 d2 74 df 48 c7 c7 f8 62 eb b3 e8 50 49 15 00 0f 0b 8b 83 f8 00 00 00 85 c0 74 d1 48 c7 c7 f8 62 eb b3 e8 38 49 15 00 <0f> 0b 5b 41 5c 5d c3 48 c7 c7 f8 62 eb b3 e8 25 49 15 00 0f 0b
May 21 19:09:59 esolr kernel: [ 18.788006] ---[ end trace 84d6c7d5cb7756cd ]---
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 21 22:41 seq
 crw-rw---- 1 root audio 116, 33 May 21 22:41 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.29
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: DigitalOcean Droplet
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-210-generic root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0
ProcVersionSignature: Ubuntu 4.15.0-210.221-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-210-generic N/A
 linux-backports-modules-4.15.0-210-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-210-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 12/12/2017
dmi.bios.vendor: DigitalOcean
dmi.bios.version: 20171212
dmi.board.asset.tag: 17601301
dmi.board.name: Droplet
dmi.board.vendor: DigitalOcean
dmi.board.version: 20171212
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-6.1
dmi.modalias: dmi:bvnDigitalOcean:bvr20171212:bd12/12/2017:svnDigitalOcean:pnDroplet:pvr20171212:rvnDigitalOcean:rnDroplet:rvr20171212:cvnQEMU:ct1:cvrpc-i440fx-6.1:
dmi.product.family: DigitalOcean_Droplet
dmi.product.name: Droplet
dmi.product.version: 20171212
dmi.sys.vendor: DigitalOcean

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2020279

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Gábor 'Morc' KORMOS (morc)]

apport information

[Jamie Strandboge (jdstrand)]

I started seeing this too after upgrading to this kernel (from https://ubuntu.com/security/notices/USN-6081-1):

ii linux-image-generic 4.15.0.211.194
ii linux-image-4.15.0-211-generic 4.15.0-211.222

$ cat /proc/version_signature
Ubuntu 4.15.0-211.222-generic 4.15.18

From /var/log/syslog:
May 27 15:18:43 <hostname> kernel: [22004.663374] ------------[ cut here ]------------
May 27 15:18:43 <hostname> kernel: [22004.663397] WARNING: CPU: 6 PID: 0 at /build/linux-RlFMDZ/linux-4.15.0/net/core/stream.c:212 sk_stream_kill_queues+0xed/0x110
May 27 15:18:43 <hostname> kernel: [22004.663398] Modules linked in: dm_crypt algif_skcipher af_alg nls_iso8859_1 ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_recent ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat intel_rapl ipmi_ssif x86_pkg_temp_thermal intel_powerclamp binfmt_misc kvm_intel kvm irqbypass dcdbas mgag200 iptable_nat ttm nf_nat_ipv4 crct10dif_pclmul xt_limit crc32_pclmul drm_kms_helper xt_tcpudp ghash_clmulni_intel pcbc drm xt_addrtype i2c_algo_bit aesni_intel aes_x86_64 crypto_simd glue_helper cryptd fb_sys_fops syscopyarea intel_cstate sysfillrect intel_rapl_perf sysimgblt lpc_ich shpchp ie31200_edac ipmi_si ipmi_devintf wmi ipmi_msghandler mac_hid nf_conntrack_ipv4 nf_defrag_ipv4 sch_fq_codel xt_conntrack
May 27 15:18:43 <hostname> kernel: [22004.663428] ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter coretemp lp parport ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 tg3 ahci ptp libahci pps_core
May 27 15:18:43 <hostname> kernel: [22004.663444] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G W 4.15.0-211-generic #222-Ubuntu
May 27 15:18:43 <hostname> kernel: [22004.663446] Hardware name: Dell Inc. PowerEdge T110 II/0PM2CW, BIOS 2.8.0 06/24/2014
May 27 15:18:43 <hostname> kernel: [22004.663449] RIP: 0010:sk_stream_kill_queues+0xed/0x110
May 27 15:18:43 <hostname> kernel: [22004.663450] RSP: 0018:ffff99813fd83920 EFLAGS: 00010282
May 27 15:18:43 <hostname> kernel: [22004.663452] RAX: 0000000000000024 RBX: ffff997fb1479980 RCX: 0000000000000006
May 27 15:18:43 <hostname> kernel: [22004.663453] RDX: 0000000000000000 RSI: ffff99813fd9b4d8 RDI: ffff99813fd9b4d0
May 27 15:18:43 <hostname> kernel: [22004.663454] RBP: ffff99813fd83930 R08: 0000000000007b7b R09: 0000000000000004
May 27 15:18:43 <hostname> kernel: [22004.663456] R10: ffff99813fd83958 R11: 0000000000000001 R12: ffff997fb1479a48
May 27 15:18:43 <hostname> kernel: [22004.663457] R13: 0000000000000006 R14: 000000000000003c R15: 0000000000000000
May 27 15:18:43 <hostname> kernel: [22004.663459] FS: 0000000000000000(0000) GS:ffff99813fd80000(0000) knlGS:0000000000000000
May 27 15:18:43 <hostname> kernel: [22004.663460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 27 15:18:43 <hostname> kernel: [22004.663461] CR2: 0000563185766000 CR3: 000000027d40a001 CR4: 00000000001606e0
May 27 15:18:43 <ho...

[Andrei Gherzan (agherzan)]

I see the original report is on a DigitalOcean Droplet, but I think this should also reproduce in a local VM. Is there anything specific I need to do to trigger the bug, Gabor?

[Gábor 'Morc' KORMOS (morc)]

  Hi Andrei,

  I have no idea what triggers it. The Droplet is a webserver, maybe whenever someone requested a page. The warning did not happen frequently, a few times in an hour, though this server is a test server, probably does not see much traffic. Maybe Jamie has a better idea or could give more info about the environment he saw it on.

[Jamie Strandboge (jdstrand)]

For me this was a bare metal bionic install on a home personal, multi-purpose, intranet server (rsyslog, apache, bind9, etc). The message was triggered 480 times in ~6 hours and 15 minutes before I downgraded. I didn't notice other issues. I looked at the logs and couldn't see a pattern of 'cut here' entries and other log entries.

[Andrei Gherzan (agherzan)]

Thanks for the extra details. I tried to reproduce it locally today, and I couldn't. While checking the code and your logs, I suspect this is IPv6-specific, and my testing env is IPv4-only. I'll come back when I manage to reproduce it. My current suspicion is:

commit e198bdd7ad2834c77567d59158e88363d2f0e05d
Author: Kuniyuki Iwashima <email address hidden>
Date: Thu Feb 9 16:22:01 2023 -0800
    dccp/tcp: Avoid negative sk_forward_alloc by ipv6_pinfo.pktoptions.
    BugLink: https://bugs.launchpad.net/bugs/2015399
    commit ca43ccf41224b023fc290073d5603a755fd12eed upstream.

This would confirm the above IPv6 theory.

PS: The warning is generated by WARN_ON(sk->sk_forward_alloc) in sk_stream_kill_queues.

[Gábor 'Morc' KORMOS (morc)]

My Droplet has an IPv6 address, and traffic on that is even less frequent than IPv4, so the sporadic occurrence we saw could be explained by that.

[Andrei Gherzan (agherzan)]

@Gabor @Jamie We are encountering some difficulties with getting a proper IPv6 instance to work on for this. As an interim solution, I could build a kernel with the patch above dropped and ask you to give it a spin for a while. Is that something you could help with?

[Gábor 'Morc' KORMOS (morc)]

  Hi Andrei, if you give me a link to a DEB package I can test it.

[Andrei Gherzan (agherzan)]

Thanks, Gabor. I appreciate your help.

In the meanwhile, I have found the issue, and I have a fix in place. So I think we should give this a spin directly, as I'm pretty confident it will solve the problem. Note that the debs are unsigned, so you must ensure that secure boot is disabled before attempting to boot these test kernels.

Here you can download the builds. It includes debs for all the supported architectures:
https://people.canonical.com/~agherzan/test_builds/lp2020279/andrei-bionic-a62707b310d6-5bJf.tar.gz

This build includes the following fix https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ddab1698a3ec6d4c5224731379ed062446b58309 and adds a local version modifier as +lp2020279.1 (so you should be able to verify that you are running the right image).

Looking forward to your feedback!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Andrei Gherzan (agherzan)]

Gabor, I could reproduce the bug using the upstream test (https://<email address hidden>/).

On 4.15.0-211-generic #222-Ubuntu I ended up with:

[ 31.286092] WARNING: CPU: 0 PID: 1088 at /build/linux-RlFMDZ/linux-4.15.0/net/core/stream.c:212 sk_stream_kill_queues+0xed/0x110

This was not reproducible with the testing version I provided above (4.15.0-211-generic #222+lp2020279.1).

I've also checked the trees of the other series:
trusty and xenial - not affected
bionic - affected
focal, jammy, kinetic, lunar, and mantic - include the fix already

Given this, I have sent the fix to the mailing list: https://lists.ubuntu.com/archives/kernel-team/2023-June/140472.html

[Gábor 'Morc' KORMOS (morc)]

Andrei, I installed your kernel version about 2 hours ago and did not experience the warnings. It's also great that you could reproduce it. Thanks for the effort!

[Gábor 'Morc' KORMOS (morc)]

  Hi Andrei, Thadeu, will this be release into the public repo (instead of ESM) and in which version of the kernel?

[Thadeu Lima de Souza Cascardo (cascardo)]

Hi, @morc, this should release into the public repo with version 4.15.0-213.224. It should be in -proposed now. Can you verify that it works, please?

[Gábor 'Morc' KORMOS (morc)]

Hi Thadeu, I updated a server a few hours ago and the warning did not appear. I think we can say that the patch works. I see no undesirable side-effect either so far.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-213.224

---------------
linux (4.15.0-213.224) bionic; urgency=medium

  * bionic/linux: 4.15.0-213.224 -proposed tracker (LP: #2024234)

  * Kernel warning after upgrade to 4.15.0.211.194 (LP: #2020279)
    - net: Remove WARN_ON_ONCE(sk->sk_forward_alloc) from sk_stream_kill_queues().

 -- Thadeu Lima de Souza Cascardo <email address hidden> Fri, 16 Jun 2023 14:32:39 -0300