arm64 gcov in 5.8 breaks when trying to enable it

Bug #1891288 reported by Colin Ian King
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Linux
Confirmed
Unknown
linux (Ubuntu)
Fix Released
Medium
Colin Ian King

Bug Description

Running:

sudo lcov --zerocounters on a gcov enabled kernel causes a read-only splat:

[ 139.798968] Unable to handle kernel write to read-only memory at virtual address ffffb4511ee6bb10
[ 139.803325] Mem abort info:
[ 139.804882] ESR = 0x9600004e
[ 139.806265] EC = 0x25: DABT (current EL), IL = 32 bits
[ 139.811261] SET = 0, FnV = 0
[ 139.814163] EA = 0, S1PTW = 0
[ 139.816545] Data abort info:
[ 139.817852] ISV = 0, ISS = 0x0000004e
[ 139.820023] CM = 0, WnR = 1
[ 139.821350] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000005086f000
[ 139.825005] [ffffb4511ee6bb10] pgd=00000001b6fff003, p4d=00000001b6fff003, pud=00000001b6ffe003, pmd=004000004e200f91
[ 139.832061] Internal error: Oops: 9600004e [#1] SMP
[ 139.835002] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua efi_pstore qemu_fw_cfg sch_fq_codel drm virtio_rng ip_t
ables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c rai
d1 raid0 multipath linear crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_net net_failover xhci_pci virtio_blk virtio_scsi failover xhci_pci_ren
esas aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
[ 139.865929] CPU: 5 PID: 1235 Comm: lcov Tainted: G W 5.8.0-8-generic #9
[ 139.871293] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 139.876248] pstate: 20400005 (nzCv daif +PAN -UAO BTYPE=--)
[ 139.879993] pc : __memset+0x148/0x188
[ 139.882387] lr : gcov_info_reset+0xa0/0x110
[ 139.885404] sp : ffff8000122f3bf0
[ 139.887730] x29: ffff8000122f3bf0 x28: ffff000167480f40
[ 139.891129] x27: 0000000000000000 x26: ffffb450e46bc060
[ 139.894688] x25: 0000000000000001 x24: ffffb450e46bc040
[ 139.897391] x23: 0000000000000000 x22: ffffb450e46bc0a8
[ 139.900064] x21: ffffb45122ba5db8 x20: ffffb450e46bc068
[ 139.902681] x19: ffffb450e46bc068 x18: 0000000000000000
[ 139.905345] x17: 0000000000000000 x16: 0000000000000000
[ 139.908016] x15: 0000000000000000 x14: 0000000000000000
[ 139.910625] x13: 0000000000000000 x12: 0000000000000000
[ 139.913260] x11: 0000000000000000 x10: 0000000000000000
[ 139.915922] x9 : 0000000000000000 x8 : ffffb4511ee6bb10
[ 139.918531] x7 : 0000000000000000 x6 : 000000000000003f
[ 139.921196] x5 : 0000000000000040 x4 : 0000000000000030
[ 139.923852] x3 : 00000007235d7fd0 x2 : 00000007235d7fd0
[ 139.926460] x1 : 0000000000000000 x0 : ffffb4511ee6bb10
[ 139.929126] Call trace:
[ 139.930349] __memset+0x148/0x188
[ 139.932025] reset_node+0x70/0xc0
[ 139.933662] reset_write+0xc0/0x130
[ 139.935389] full_proxy_write+0x98/0x110
[ 139.937349] vfs_write+0x18c/0x3e0
[ 139.939031] ksys_write+0xc4/0x1f0
[ 139.940766] __arm64_sys_write+0x44/0x60
[ 139.942705] invoke_syscall+0x74/0x150
[ 139.944601] el0_svc_common.constprop.0+0x94/0x264
[ 139.946961] do_el0_svc+0x5c/0x7c
[ 139.948651] el0_svc+0x70/0x114
[ 139.950206] el0_sync_handler+0x140/0x39c
[ 139.952239] el0_sync+0x17c/0x180
[ 139.953894] Code: f101007f fa45a068 54fffc0b aa0303e2 (a9001d07)
[ 139.956946] ---[ end trace 71e2f9cc3eaddf4e ]---

Changed in linux (Ubuntu):
importance: Undecided → Medium
assignee: nobody → Colin Ian King (colin-king)
status: New → In Progress
Revision history for this message
In , colin.king (colin.king-linux-kernel-bugs) wrote :
Download full text (3.3 KiB)

5.8 kernel running on a QEMU KVM Virtual Machine, arm64, gcov enabled:

CONFIG_GCOV_KERNEL=y
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
CONFIG_GCOV_PROFILE_ALL=y

running: sudo lcov --zerocounters

..causes the following splat:

[ 78.634926] Unable to handle kernel write to read-only memory at virtual address ffffd764f255bb10
[ 78.639076] Mem abort info:
[ 78.640313] ESR = 0x9600004e
[ 78.641656] EC = 0x25: DABT (current EL), IL = 32 bits
[ 78.644351] SET = 0, FnV = 0
[ 78.645766] EA = 0, S1PTW = 0
[ 78.649526] Data abort info:
[ 78.654341] ISV = 0, ISS = 0x0000004e
[ 78.656851] CM = 0, WnR = 1
[ 78.659043] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000b775f000
[ 78.665822] [ffffd764f255bb10] pgd=00000001b6fff003, p4d=00000001b6fff003, pud=00000001b6ffe003, pmd=00400000b5000f91
[ 78.677367] Internal error: Oops: 9600004e [#1] SMP
[ 78.681204] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua efi_pstore qemu_fw_cfg sch_fq_codel drm virtio_rng ip_t
ables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c rai
d1 raid0 multipath linear crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_net xhci_pci net_failover virtio_blk virtio_scsi failover xhci_pci_ren
esas aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
[ 78.722642] CPU: 14 PID: 1175 Comm: lcov Tainted: G W 5.8.0-8-generic #9
[ 78.729364] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 78.733467] pstate: 20400005 (nzCv daif +PAN -UAO BTYPE=--)
[ 78.736219] pc : __memset+0x148/0x188
[ 78.737993] lr : gcov_info_reset+0xa0/0x110
[ 78.740037] sp : ffff800010e53bf0
[ 78.741624] x29: ffff800010e53bf0 x28: ffff00015a648000
[ 78.744181] x27: 0000000000000000 x26: ffffd764e3df4060
[ 78.746776] x25: 0000000000000001 x24: ffffd764e3df4040
[ 78.749310] x23: 0000000000000000 x22: ffffd764e3df40a8
[ 78.751976] x21: ffffd764f6295db8 x20: ffffd764e3df4068
[ 78.754642] x19: ffffd764e3df4068 x18: 0000000000000000
[ 78.757265] x17: 0000000000000000 x16: 0000000000000000
[ 78.759922] x15: 0000000000000000 x14: 0000000000000000
[ 78.762585] x13: 0000000000000000 x12: 0000000000000000
[ 78.765209] x11: 0000000000000000 x10: 0000000000000000
[ 78.767861] x9 : 0000000000000000 x8 : ffffd764f255bb10
[ 78.770518] x7 : 0000000000000000 x6 : 000000000000003f
[ 78.773135] x5 : 0000000000000040 x4 : 0000000000000030
[ 78.775793] x3 : 000000071ef97fd0 x2 : 000000071ef97fd0
[ 78.778449] x1 : 0000000000000000 x0 : ffffd764f255bb10
[ 78.781075] Call trace:
[ 78.782329] __memset+0x148/0x188
[ 78.783980] reset_node+0x70/0xc0
[ 78.785628] reset_write+0xc0/0x130
[ 78.787397] full_proxy_write+0x98/0x110
[ 78.789354] vfs_write+0x18c/0x3e0
[ 78.791086] ksys_write+0xc4/0x1f0
[ 78.792778] __arm64_sys_write+0x44/0x60
[ 78.794771] invoke_syscall+0x74/0x150
[ 78.796639] el0_svc_common.constprop.0+0x94/0x264
[ 78.799049] do_el0_svc+0x5c/0x7c
[ 78.800702] el0_svc+0x70/0x114
[ 78.802311] el0_sync_handler+0x140/0x39c
[ 78.804...

Read more...

description: updated
Revision history for this message
Colin Ian King (colin-king) wrote :
Download full text (3.8 KiB)

Ah, earlier on boot I'm seeing:
[ 8.808508] virtio_blk virtio3: [vda] 209715200 512-byte logical blocks (107 GB/100 GiB)
[ 8.808649] ------------[ cut here ]------------
[ 8.815777] vda: detected capacity change from 0 to 107374182400
[ 8.816623] WARNING: CPU: 3 PID: 212 at mm/page_alloc.c:4813 __alloc_pages_nodemask+0x540/0x694
[ 8.816630] Modules linked in: net_failover virtio_blk(+) xhci_pci_renesas virtio_scsi(+) failover fjes(-) aes_neon_bs aes_neon_blk aes_ce_blk crypto_s
imd cryptd aes_ce_cipher
[ 8.837701] CPU: 3 PID: 212 Comm: systemd-udevd Not tainted 5.8.0-8-generic #9
[ 8.842236] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 8.846585] pstate: 20400005 (nzCv daif +PAN -UAO BTYPE=--)
[ 8.850112] pc : __alloc_pages_nodemask+0x540/0x694
[ 8.853106] lr : alloc_pages_current+0xe0/0x17c
[ 8.856024] sp : ffff8000106138c0
[ 8.858175] x29: ffff8000106138c0 x28: ffff000170252dc0
[ 8.861565] x27: 0000000000000000 x26: 0000000000000015
[ 8.865006] x25: 00000001abcfef80 x24: 0000000000000015
[ 8.868466] x23: ffff000170252dc0 x22: 0000000000000000
[ 8.871794] x21: ffffdb2796703050 x20: 0000000000040dc0
[ 8.875143] x19: ffffdb2796703250 x18: 0000000000000000
[ 8.878529] x17: 0000000000000000 x16: 0000000000000000
[ 8.881917] x15: 0000000000000000 x14: 73656a662f74656e
[ 8.885218] x13: 2f73726576697264 x12: 2f636972656e6567
[ 8.888554] x11: 2d646c6975622f64 x10: 6c6975622f6e6169
[ 8.892032] x9 : ffffdb2792b6dfa4 x8 : 63672e7366677562
[ 8.895415] x7 : 65645f73656a662f x6 : ffff0001558ef353
[ 8.898814] x5 : 0000000000000018 x4 : 0000000000000dd5
[ 8.902124] x3 : ffffdb27966ea138 x2 : 0000000000000001
[ 8.905418] x1 : ffffdb27966ea030 x0 : 0000000000000001
[ 8.908698] Call trace:
[ 8.910278] __alloc_pages_nodemask+0x540/0x694
[ 8.913278] alloc_pages_current+0xe0/0x17c
[ 8.915967] kmalloc_order+0x48/0xec
[ 8.918325] kmalloc_order_trace+0x54/0x21c
[ 8.921165] __kmalloc+0x454/0x5c0
[ 8.923368] kmalloc_array.constprop.0+0x30/0x50
[ 8.926301] gcov_info_dup+0x98/0x2c0
[ 8.928557] remove_info+0x1e0/0x244
[ 8.930876] gcov_event+0xfc/0x170
[ 8.933097] gcov_module_notifier+0x164/0x174
[ 8.935848] notifier_call_chain+0xa0/0x110
[ 8.938435] blocking_notifier_call_chain+0x80/0xe0
[ 8.941579] do_init_module+0x360/0x43c
[ 8.944198] load_module+0x8fc/0xa50
[ 8.946591] __do_sys_finit_module+0x190/0x1c0
[ 8.949485] __arm64_sys_finit_module+0x48/0x64
[ 8.952376] invoke_syscall+0x74/0x150
[ 8.954901] el0_svc_common.constprop.0+0x94/0x264
[ 8.956405] sr 0:0:0:0: Power-on or device reset occurred
[ 8.957610] do_el0_svc+0x5c/0x7c
[ 8.960983] sr 0:0:0:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray
[ 8.962169] el0_svc+0x70/0x114
[ 8.965694] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 8.967253] el0_sync_handler+0x140/0x39c
[ 8.971738] el0_sync+0x17c/0x180
[ 8.973450] ---[ end trace 267579295e47a82a ]---
[ 8.976042] gcov: could not save data for '/home/cking/unstable/debian/build/build-generic/drivers/net/fjes/fjes_debugfs.gcda' (out of memory)
[ ...

Read more...

Revision history for this message
In , colin.king (colin.king-linux-kernel-bugs) wrote :
Download full text (3.8 KiB)

Just spotted that earlier in the boot we get:

[ 8.808508] virtio_blk virtio3: [vda] 209715200 512-byte logical blocks (107 GB/100 GiB)
[ 8.808649] ------------[ cut here ]------------
[ 8.815777] vda: detected capacity change from 0 to 107374182400
[ 8.816623] WARNING: CPU: 3 PID: 212 at mm/page_alloc.c:4813 __alloc_pages_nodemask+0x540/0x694
[ 8.816630] Modules linked in: net_failover virtio_blk(+) xhci_pci_renesas virtio_scsi(+) failover fjes(-) aes_neon_bs aes_neon_blk aes_ce_blk crypto_s
imd cryptd aes_ce_cipher
[ 8.837701] CPU: 3 PID: 212 Comm: systemd-udevd Not tainted 5.8.0-8-generic #9
[ 8.842236] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 8.846585] pstate: 20400005 (nzCv daif +PAN -UAO BTYPE=--)
[ 8.850112] pc : __alloc_pages_nodemask+0x540/0x694
[ 8.853106] lr : alloc_pages_current+0xe0/0x17c
[ 8.856024] sp : ffff8000106138c0
[ 8.858175] x29: ffff8000106138c0 x28: ffff000170252dc0
[ 8.861565] x27: 0000000000000000 x26: 0000000000000015
[ 8.865006] x25: 00000001abcfef80 x24: 0000000000000015
[ 8.868466] x23: ffff000170252dc0 x22: 0000000000000000
[ 8.871794] x21: ffffdb2796703050 x20: 0000000000040dc0
[ 8.875143] x19: ffffdb2796703250 x18: 0000000000000000
[ 8.878529] x17: 0000000000000000 x16: 0000000000000000
[ 8.881917] x15: 0000000000000000 x14: 73656a662f74656e
[ 8.885218] x13: 2f73726576697264 x12: 2f636972656e6567
[ 8.888554] x11: 2d646c6975622f64 x10: 6c6975622f6e6169
[ 8.892032] x9 : ffffdb2792b6dfa4 x8 : 63672e7366677562
[ 8.895415] x7 : 65645f73656a662f x6 : ffff0001558ef353
[ 8.898814] x5 : 0000000000000018 x4 : 0000000000000dd5
[ 8.902124] x3 : ffffdb27966ea138 x2 : 0000000000000001
[ 8.905418] x1 : ffffdb27966ea030 x0 : 0000000000000001
[ 8.908698] Call trace:
[ 8.910278] __alloc_pages_nodemask+0x540/0x694
[ 8.913278] alloc_pages_current+0xe0/0x17c
[ 8.915967] kmalloc_order+0x48/0xec
[ 8.918325] kmalloc_order_trace+0x54/0x21c
[ 8.921165] __kmalloc+0x454/0x5c0
[ 8.923368] kmalloc_array.constprop.0+0x30/0x50
[ 8.926301] gcov_info_dup+0x98/0x2c0
[ 8.928557] remove_info+0x1e0/0x244
[ 8.930876] gcov_event+0xfc/0x170
[ 8.933097] gcov_module_notifier+0x164/0x174
[ 8.935848] notifier_call_chain+0xa0/0x110
[ 8.938435] blocking_notifier_call_chain+0x80/0xe0
[ 8.941579] do_init_module+0x360/0x43c
[ 8.944198] load_module+0x8fc/0xa50
[ 8.946591] __do_sys_finit_module+0x190/0x1c0
[ 8.949485] __arm64_sys_finit_module+0x48/0x64
[ 8.952376] invoke_syscall+0x74/0x150
[ 8.954901] el0_svc_common.constprop.0+0x94/0x264
[ 8.956405] sr 0:0:0:0: Power-on or device reset occurred
[ 8.957610] do_el0_svc+0x5c/0x7c
[ 8.960983] sr 0:0:0:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray
[ 8.962169] el0_svc+0x70/0x114
[ 8.965694] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 8.967253] el0_sync_handler+0x140/0x39c
[ 8.971738] el0_sync+0x17c/0x180
[ 8.973450] ---[ end trace 267579295e47a82a ]---
[ 8.976042] gcov: could not save data for '/home/cking/unstable/debian/build/build-generic/drivers/net/fjes/fjes_debugfs.gcda' (out ...

Read more...

Revision history for this message
In , colin.king (colin.king-linux-kernel-bugs) wrote :

I upped the VM memory to 16GB and reduced the CPU count to 1 CPU and still hit the out of memory condition with gcov.

Changed in linux:
status: Unknown → Confirmed
Revision history for this message
Colin Ian King (colin-king) wrote :

This is due to using gcc-10.2. Building with gcc-9 is fine. This compiler support for GCOV will be disabled, see https://lkml.org/lkml/2020/9/2/934

So, marking this as Won't Fix because there is a workaround and 10.2 is not looking favourably for GCOV support at the moment.

Changed in linux (Ubuntu):
status: In Progress → Won't Fix
Revision history for this message
Colin Ian King (colin-king) wrote :

Upstream fix 40249c6962075c040fd071339acae524f18bfac9, this has already been picked up by Sash Levin for 5.8, 5.4, 4.19, 4.14, 4.9 and 4.8 on backport stable AUTOSEL. Lets wait for that fix to trickle into the SRU process.

Changed in linux (Ubuntu):
status: Won't Fix → In Progress
Revision history for this message
Colin Ian King (colin-king) wrote :

Fix applied in risc-v groovy with commit 4fc484ffd959e0d897d073296cfcfafdf473e407

Changed in linux (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.