general protection fault on Ubuntu 4.15.0-1103.114~16.04.1-azure 4.15.18

[Eli Corrales]

Hello

The following kernel tainted scenario have been observed in Azure VMs. This happened some weeks after the kernel was upgraded from 4.15.0-1098-azure to 4.15.0-1103-azure. Now, the VMs have been updated to kernel 1106 and the issue has not been observed. but want to confirm if there is a bug for the 1103 kernel which is causing this problem:

NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

# cat /proc/version_signature
Ubuntu 4.15.0-1103.114~16.04.1-azure 4.15.18

BOOT_IMAGE=/boot/vmlinuz-4.15.0-1103-azure
*****************************************************************************************************
[2057238.731281] general protection fault: 0000 [#1] SMP PTI
[2057238.734885] Modules linked in: symap_custom_4_15_0_1103_azure_x86_64(POE) symev_custom_4_15_0_1103_azure_x86_64(OE) dccp_diag dccp unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag iptable_filter ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner iptable_security xt_conntrack nf_conntrack ip_tables x_tables binfmt_misc cmac arc4 md4 nls_utf8 cifs ccm fscache nls_iso8859_1 mlx4_en mlx4_core kvm_intel hv_balloon pci_hyperv kvm irqbypass serio_raw joydev ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel
[2057239.148068] hid_hyperv aes_x86_64 hyperv_fb crypto_simd cfbfillrect glue_helper cfbimgblt cryptd hid cfbcopyarea hv_netvsc pata_acpi hyperv_keyboard hv_utils [last unloaded: symevrm_custom_4_15_0_1103_azure_x86_64]
[2057239.285231] CPU: 7 PID: 2548 Comm: filebeat Tainted: P OE 4.15.0-1103-azure #114~16.04.1-Ubuntu
[2057239.359930] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[2057239.443494] RIP: 0010:kmem_cache_alloc+0x7a/0x1c0
[2057239.514702] RSP: 0018:ffffa2b44bb0f970 EFLAGS: 00010206
[2057239.587510] RAX: 2cbecca03c1aa960 RBX: 0000000000000000 RCX: ffff9336d8a3c5a0
[2057239.670716] RDX: 00000000002dc13a RSI: 0000000001011200 RDI: 00000000000251c0
[2057239.748088] RBP: ffffa2b44bb0f9a0 R08: ffff93371fde51c0 R09: 0000000000000002
[2057239.829346] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000001011200
[2057239.908118] R13: 2cbecca03c1aa960 R14: ffff9336df002a80 R15: ffff9336df002a80
[2057239.988090] FS: 00007ff2d7fff700(0000) GS:ffff93371fdc0000(0000) knlGS:0000000000000000
[2057240.080218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2057240.160992] CR2: 00007fe528002948 CR3: 000000107c902002 CR4: 00000000003706e0
[2057240.244702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2057240.335323] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2057240.416496] Call Trace:
[2057240.490398] ? mempool_alloc_slab+0x15/0x20
[2057240.574478] mempool_alloc_slab+0x15/0x20
[2057240.661848] mempool_alloc+0x73/0x180
[2057240.740075] ? mempool_alloc_slab+0x15/0x20
[2057240.819037] ? mempool_alloc+0x73/0x180
[2057240.904391] bvec_alloc+0x90/0xf0
[2057240.985346] bio_alloc_bioset+0x13b/0x1e0
[2057241.065039] ext4_bio_write_page+0x3e9/0x550
[2057241.148063] mpage_submit_page+0x58/0x70
[2057241.228960] mpage_process_page_bufs+0x10e/0x120
[2057241.317256] mpage_prepare_extent_to_map+0x21f/0x2f0
[2057241.400777] ext4_writepages+0x47d/0xe50
[2057241.484080] do_writepages+0x1f/0x70
[2057241.572104] ? ext4_mark_inode_dirty+0x1d0/0x1d0
[2057241.675375] ? do_writepages+0x1f/0x70
[2057241.756087] __filemap_fdatawrite_range+0xd4/0x110
[2057241.840087] file_write_and_wait_range+0x5a/0xb0
[2057241.924041] ext4_sync_file+0x11d/0x420
[2057242.008030] vfs_fsync_range+0x51/0xb0
[2057242.096269] do_fsync+0x3d/0x70
[2057242.180084] SyS_fdatasync+0x13/0x20
[2057242.262207] do_syscall_64+0x80/0x1e0
[2057242.344070] entry_SYSCALL_64_after_hwframe+0x41/0xa6
[2057242.433441] RIP: 0033:0xc6edf0
[2057242.516094] RSP: 002b:000000c000318d50 EFLAGS: 00000206 ORIG_RAX: 000000000000004b
[2057242.616068] RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 0000000000c6edf0
[2057242.712115] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[2057242.812071] RBP: 000000c000318d90 R08: 0000000000000000 R09: 0000000000000000
[2057242.902700] R10: 0000000000000000 R11: 0000000000000206 R12: 00000000000001a8
[2057242.992107] R13: 00000000000001a7 R14: 0000000000000200 R15: 0000000000000000
[2057243.090260] Code: 50 08 65 4c 03 05 0f b4 fc 5a 49 83 78 10 00 4d 8b 28 0f 84 0b 01 00 00 4d 85 ed 0f 84 02 01 00 00 49 63 47 20 49 8b 3f 4c 01 e8 <48> 8b 18 49 33 9f 40 01 00 00 48 89 c1 48 0f c9 4c 89 e8 48 31
[2057243.268154] RIP: kmem_cache_alloc+0x7a/0x1c0 RSP: ffffa2b44bb0f970
[2057243.358506] ---[ end trace 10983d36521b06bc ]---
[2057245.705285] general protection fault: 0000 [#2] SMP PTI
[2057245.709249] Modules linked in: symap_custom_4_15_0_1103_azure_x86_64(POE) symev_custom_4_15_0_1103_azure_x86_64(OE) dccp_diag dccp unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag iptable_filter ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner iptable_security xt_conntrack nf_conntrack ip_tables x_tables binfmt_misc cmac arc4 md4 nls_utf8 cifs ccm fscache nls_iso8859_1 mlx4_en mlx4_core kvm_intel hv_balloon pci_hyperv kvm irqbypass serio_raw joydev ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel
[2057246.326904] hid_hyperv aes_x86_64 hyperv_fb crypto_simd cfbfillrect glue_helper cfbimgblt cryptd hid cfbcopyarea hv_netvsc pata_acpi hyperv_keyboard hv_utils [last unloaded: symevrm_custom_4_15_0_1103_azure_x86_64]
[2057246.513580] CPU: 7 PID: 2185 Comm: python3 Tainted: P D OE 4.15.0-1103-azure #114~16.04.1-Ubuntu
[2057246.608442] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[2057246.710620] RIP: 0010:kmem_cache_alloc+0x7a/0x1c0
[2057246.800165] RSP: 0018:ffffa2b4486e7978 EFLAGS: 00010206
[2057246.892543] RAX: 2cbecca03c1aa960 RBX: 0000000000000000 RCX: ffff9336d8a3c5a0
[2057246.981963] RDX: 00000000002dc13a RSI: 0000000001011200 RDI: 00000000000251c0
[2057247.070299] RBP: ffffa2b4486e79a8 R08: ffff93371fde51c0 R09: 0000000000000002
[2057247.166705] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000001011200
[2057247.269597] R13: 2cbecca03c1aa960 R14: ffff9336df002a80 R15: ffff9336df002a80
[2057247.361174] FS: 00007fa03ee4b700(0000) GS:ffff93371fdc0000(0000) knlGS:0000000000000000
[2057247.457183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2057247.547234] CR2: 00007fa14ffff860 CR3: 000000107ba7a003 CR4: 00000000003706e0
[2057247.650283] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2057247.744489] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2057247.840117] Call Trace:
[2057247.919761] ? mempool_alloc_slab+0x15/0x20
[2057248.020057] mempool_alloc_slab+0x15/0x20
[2057248.111814] mempool_alloc+0x73/0x180
[2057248.205509] ? mempool_alloc_slab+0x15/0x20
[2057248.297125] ? mempool_alloc+0x73/0x180
[2057248.389777] bvec_alloc+0x90/0xf0
[2057248.478851] bio_alloc_bioset+0x13b/0x1e0
[2057248.575571] ext4_bio_write_page+0x3e9/0x550
[2057248.675719] mpage_submit_page+0x58/0x70
[2057248.772080] mpage_map_and_submit_buffers+0x135/0x240
[2057248.863571] ? jbd2_journal_inode_ranged_wait+0x1b/0x20
[2057248.955521] ? ext4_map_blocks+0x5b9/0x600
[2057249.044219] ? ext4_writepages+0x537/0xe50
[2057249.143147] ext4_writepages+0x877/0xe50
[2057249.251311] do_writepages+0x1f/0x70
[2057249.344244] ? ext4_mark_inode_dirty+0x1d0/0x1d0
[2057249.432440] ? do_writepages+0x1f/0x70
[2057249.521261] __filemap_fdatawrite_range+0xd4/0x110
[2057249.621593] filemap_flush+0x1c/0x20
[2057249.724132] ext4_alloc_da_blocks+0x2c/0x80
[2057249.822409] ext4_release_file+0x79/0xc0
[2057249.915754] __fput+0xea/0x220
[2057250.006629] ____fput+0xe/0x10
[2057250.094252] task_work_run+0x8a/0xb0
[2057250.178480] exit_to_usermode_loop+0xc4/0xd0
[2057250.270206] do_syscall_64+0x1af/0x1e0
[2057250.362057] entry_SYSCALL_64_after_hwframe+0x41/0xa6
[2057250.451345] RIP: 0033:0x7fa03ea2957d
[2057250.526021] RSP: 002b:00007ffec3c849a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[2057250.636063] RAX: 0000000000000000 RBX: 00000000010d94e0 RCX: 00007fa03ea2957d
[2057250.726612] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[2057250.827090] RBP: 0000000000a3f040 R08: 0000000000000000 R09: 0000000000a40b40
[2057250.912853] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000003
[2057250.982463] R13: 00000000004df540 R14: 0000000000a3f040 R15: 00007ffec3c84b90
[2057251.068328] Code: 50 08 65 4c 03 05 0f b4 fc 5a 49 83 78 10 00 4d 8b 28 0f 84 0b 01 00 00 4d 85 ed 0f 84 02 01 00 00 49 63 47 20 49 8b 3f 4c 01 e8 <48> 8b 18 49 33 9f 40 01 00 00 48 89 c1 48 0f c9 4c 89 e8 48 31
[2057251.231345] RIP: kmem_cache_alloc+0x7a/0x1c0 RSP: ffffa2b4486e7978
[2057251.311283] ---[ end trace 10983d36521b06bd ]---
[2057284.805211] general protection fault: 0000 [#3] SMP PTI
[2057284.809128] Modules linked in: symap_custom_4_15_0_1103_azure_x86_64(POE) symev_custom_4_15_0_1103_azure_x86_64(OE) dccp_diag dccp unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag iptable_filter ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner iptable_security xt_conntrack nf_conntrack ip_tables x_tables binfmt_misc cmac arc4 md4 nls_utf8 cifs ccm fscache nls_iso8859_1 mlx4_en mlx4_core kvm_intel hv_balloon pci_hyperv kvm irqbypass serio_raw joydev ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel
[2057285.389838] hid_hyperv aes_x86_64 hyperv_fb crypto_simd cfbfillrect glue_helper cfbimgblt cryptd hid cfbcopyarea hv_netvsc pata_acpi hyperv_keyboard hv_utils [last unloaded: symevrm_custom_4_15_0_1103_azure_x86_64]
[2057285.571163] CPU: 7 PID: 10077 Comm: nscd Tainted: P D OE 4.15.0-1103-azure #114~16.04.1-Ubuntu
[2057285.671480] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[2057285.764659] RIP: 0010:__kmalloc_node_track_caller+0x187/0x2c0
[2057285.856052] RSP: 0018:ffffa2b451743b40 EFLAGS: 00010206
[2057285.940096] RAX: 2cbecca03c1aa960 RBX: 0000000000000000 RCX: ffffffffa55b52d7
[2057286.036082] RDX: 00000000002dc13a RSI: 0000000000000000 RDI: 00000000000251c0
[2057286.128169] RBP: ffffa2b451743b88 R08: ffff93371fde51c0 R09: ffff9336df002a80
[2057286.218157] R10: ffff9336df002a80 R11: 2cbecca03c1aa960 R12: 00000000010112c0
[2057286.318306] R13: 0000000000001140 R14: 00000000ffffffff R15: 00000000ffffffff
[2057286.412074] FS: 00007fdb65626700(0000) GS:ffff93371fdc0000(0000) knlGS:0000000000000000
[2057286.508070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2057286.608044] CR2: 000000000076c318 CR3: 0000000b01f9e002 CR4: 00000000003706e0
[2057286.710722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2057286.807399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2057286.902809] Call Trace:
[2057286.991571] ? __alloc_skb+0x87/0x1c0
[2057287.084058] __kmalloc_reserve.isra.43+0x31/0x90
[2057287.184075] ? __alloc_skb+0x5b/0x1c0
[2057287.280264] __alloc_skb+0x87/0x1c0
[2057287.376057] ? skb_release_data+0x110/0x170
[2057287.469793] netlink_dump+0xd5/0x2d0
[2057287.560091] netlink_recvmsg+0x278/0x420
[2057287.652036] sock_recvmsg+0x43/0x50
[2057287.740053] ___sys_recvmsg+0xf1/0x230
[2057287.829354] ? netlink_sendmsg+0x133/0x3d0
[2057287.916062] ? sock_sendmsg+0x3e/0x50
[2057288.005358] ? SYSC_sendto+0x101/0x190
[2057288.096110] __sys_recvmsg+0x51/0x90
[2057288.185074] ? __sys_recvmsg+0x51/0x90
[2057288.272076] SyS_recvmsg+0x12/0x20
[2057288.362220] do_syscall_64+0x80/0x1e0
[2057288.453107] entry_SYSCALL_64_after_hwframe+0x41/0xa6
[2057288.556460] RIP: 0033:0x7fdb6514c430
[2057288.656147] RSP: 002b:00007fffb21892c8 EFLAGS: 00000246 ORIG_RAX: 000000000000002f
[2057288.749218] RAX: ffffffffffffffda RBX: 00007fffb218a380 RCX: 00007fdb6514c430
[2057288.848061] RDX: 0000000000000000 RSI: 00007fffb218a3a0 RDI: 0000000000000003
[2057288.949552] RBP: 00007fffb218a410 R08: 00007fffb218a360 R09: 000000000000000c
[2057289.042829] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fffb218a360
[2057289.136867] R13: 00007fffb218a3a0 R14: 000000000000275d R15: 0000000000000003
[2057289.231143] Code: 45 89 e1 4c 8b 45 c8 4c 89 e9 48 89 da 48 8b 75 d0 e8 7e a6 9b 00 4d 8b 17 58 4d 85 d2 75 d7 eb 91 49 63 42 20 49 8b 3a 4c 01 d8 <48> 8b 18 49 33 9a 40 01 00 00 48 89 c1 48 0f c9 4c 89 d8 48 31
[2057289.432386] RIP: __kmalloc_node_track_caller+0x187/0x2c0 RSP: ffffa2b451743b40
[2057289.528417] ---[ end trace 10983d36521b06be ]---