booting linux-generic-lpae armhf kernel under qemu results in relocation out of range, and thus no modules can be loaded

Bug #1899519 reported by Dimitri John Ledkov on 2020-10-12
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Release Notes for Ubuntu
Undecided
Unassigned
linux (Ubuntu)
Status tracked in Groovy
Groovy
High
Unassigned

Bug Description

[Impact]

Groovy armhf LPAE kernel fails to load kmods (and thus fail to boot).

[Fix]

Enable CONFIG_ARM_MODULE_PLTS for armhf

[Regression potential]

Quoting the arch/arm/Kconfig:

"This [option] allows modules to be allocated in the generic
 vmalloc area after the dedicated module memory area has been
 exhausted. The modules will use slightly more memory, but after
 rounding up to page size, the actual memory footprint is usually
 the same.

 Disabling this is usually safe for small single-platform
 configurations.
"

---

qemu-system-arm -M virt -m 4096 -smp 2,sockets=2,cores=1,threads=1 -drive file=./groovy-server-cloudimg-armhf.img,if=virtio -kernel ./vmlinuz-5.8.0-21-generic-lpae -initrd ./initrd.img-5.8.0-21-generic-lpae -append 'root=LABEL=cloudimg-rootfs break=bottom' -nographic

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.8.0-21-generic-lpae (buildd@bos02-arm64-041) (gcc (Ubuntu 10.2.0-11ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #22-Ubuntu SMP Fri Oct 2 12:30:15 UTC 2020 (Ubuntu 5.8.0-21.22-generic-lpae 5.8.13)
[ 0.000000] CPU: ARMv7 Processor [412fc0f1] revision 1 (ARMv7), cr=30c5387d

Starting version 246.6-1ubuntu1
[ 16.753306] gpio_keys: section 4 reloc 29 sym '_raw_spin_lock_irqsave': relocation 28 out of range (0xbf00f23c -> 0xc10a90ec)
Begin: Loading essential drivers ... [ 22.799409] linear: section 4 reloc 43 sym '__stack_chk_fail': relocation 28 out of range (0xbf0195e8 -> 0xc10a1dd8)
[ 23.000670] multipath: section 4 reloc 31 sym '__stack_chk_fail': relocation 28 out of range (0xbf021268 -> 0xc10a1dd8)
[ 23.227437] raid0: section 4 reloc 53 sym '__stack_chk_fail': relocation 28 out of range (0xbf02a548 -> 0xc10a1dd8)
[ 23.460764] raid1: section 4 reloc 18 sym '_cond_resched': relocation 28 out of range (0xbf035390 -> 0xc10a3b4c)
[ 28.311563] libcrc32c: section 4 reloc 9 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b8 -> 0xc10a1dd8)
[ 31.866657] libcrc32c: section 4 reloc 9 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b8 -> 0xc10a1dd8)
[ 35.632259] libcrc32c: section 4 reloc 9 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b8 -> 0xc10a1dd8)
[ 35.852163] raid10: section 4 reloc 21 sym '__stack_chk_fail': relocation 28 out of range (0xbf00b404 -> 0xc10a1dd8)
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [ 41.265330] libcrc32c: section 4 reloc 9 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b8 -> 0xc10a1dd8)

(initramfs) cat /proc/modules
(initramfs) modprobe virtio_blk
[ 121.814904] virtio_blk: section 4 reloc 3 sym '_raw_spin_lock_irqsave': relocation 28 out of range (0xbf003050 -> 0xc10a90ec)
(initramfs) cat /proc/modules
(initramfs)

Are we miss-building the kernel and/or modules? Or am I miss-launching the VM?

tags: added: armhf groovy
Changed in linux (Ubuntu):
milestone: none → ubuntu-20.10

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1899519

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete

I wonder if this is because -M virt defaults to highmem=on, I will try again with highmem=off

Dimitri John Ledkov (xnox) wrote :

-M virt,highmem=off still shows relocation 28 out of range upon trying to load any kernel modules.

Dimitri John Ledkov (xnox) wrote :

20, 21 kernels were both bad. Testing 22 kernel now.

Dimitri John Ledkov (xnox) wrote :

@ubuntu-kernel-bot given i can't boot past initrd, what do you expect from me?

tags: added: bot-stop-nagging
summary: - booting armhf kernel under qemu results in relocations out of range, and
- thus no modules are loaded
+ booting linux-generic-lpae armhf kernel under qemu results in relocation
+ out of range, and thus no modules can be loaded
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
importance: Undecided → High
Dimitri John Ledkov (xnox) wrote :

and same with 22 kernel.

Dimitri John Ledkov (xnox) wrote :

[ 0.110128] /cpus/cpu@0 missing clock-frequency property
[ 0.110426] /cpus/cpu@1 missing clock-frequency property
[ 0.190376] CPU1: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 11.068045] cryptd: section 4 reloc 15 sym '__stack_chk_fail': relocation 28 out of range (0xbf0031d4 -> 0xc109a5e8)
[ 18.150382] failover: section 4 reloc 5 sym '_raw_spin_lock': relocation 28 out of range (0xbf00f054 -> 0xc10a1754)
[ 18.158236] virtio_blk: section 4 reloc 1 sym '_raw_spin_lock_irqsave': relocation 28 out of range (0xbf01704c -> 0xc10a187c)
[ 18.715695] gpio_keys: section 4 reloc 29 sym '_raw_spin_lock_irqsave': relocation 28 out of range (0xbf02123c -> 0xc10a187c)
[ 24.702666] linear: section 4 reloc 39 sym '__stack_chk_fail': relocation 28 out of range (0xbf02b5e0 -> 0xc109a5e8)
[ 24.895060] multipath: section 4 reloc 27 sym '__stack_chk_fail': relocation 28 out of range (0xbf033260 -> 0xc109a5e8)
[ 25.139196] raid0: section 4 reloc 49 sym '__stack_chk_fail': relocation 28 out of range (0xbf03c540 -> 0xc109a5e8)
[ 25.370769] raid1: section 4 reloc 18 sym '_cond_resched': relocation 28 out of range (0xbf047390 -> 0xc109c350)
[ 28.742440] libcrc32c: section 4 reloc 5 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b0 -> 0xc109a5e8)
[ 31.872236] libcrc32c: section 4 reloc 5 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b0 -> 0xc109a5e8)
[ 34.321203] libcrc32c: section 4 reloc 5 sym '__stack_chk_fail': relocation 28 out of range (0xbf0030b0 -> 0xc109a5e8)
[ 34.528443] raid10: section 4 reloc 17 sym '__stack_chk_fail': relocation 28 out of range (0xbf00b3fc -> 0xc109a5e8)

dann frazier (dannf) wrote :

Perhaps we need to enable CONFIG_ARM_MODULE_PLTS ?

https://www.spinics.net/lists/arm-kernel/msg813310.html

Paolo Pisati (p-pisati) on 2020-10-14
description: updated
Dimitri John Ledkov (xnox) wrote :

CONFIG_ARM64_MODULE_PLTS is enabled; because CONFIG_ARM64_ERRATUM_843419 is enabled in zesty+ (backported to yakkety too) see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647793

groovy/master:debian.master/config/annotations:CONFIG_ARM64_ERRATUM_843419 policy<{'arm64': 'y'}>
groovy/master:debian.master/config/annotations:CONFIG_ARM64_ERRATUM_843419 mark<ENFORCED> note<LP:1647793>

But CONFIG_ARM_MODULE_PLTS was not.

groovy/master:debian.master/config/config.common.ubuntu:CONFIG_ARM64_MODULE_PLTS=y
groovy/master:debian.master/config/config.common.ubuntu:# CONFIG_ARM_MODULE_PLTS is not set

5.8 kernel looks a bit bigger than 5.4 on armhf, thus possibly only now triggering this issue on armhf too now.

Rebuilding with CONFIG_ARM_MODULE_PLTS=y to see if that will help.

Dimitri John Ledkov (xnox) wrote :

I ponder if raspi kernel also needs this on armhf

debian.raspi/config/config.common.ubuntu:# CONFIG_ARM_MODULE_PLTS is not set

Paolo Pisati (p-pisati) wrote :

FWIW, i've already sent the config change to the kernel ml because it actually fixes the kmod loading issue: i reproduce the problem in a local qemu, and ARM_MODULE_PLTS=y fixed it.

Dimitri John Ledkov (xnox) wrote :

@paulo awesome; will there be a kernel respin? if yes, can we also take https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1899582 ?

Paolo Pisati (p-pisati) wrote :

@xnox: yes, we usually respin before release with only hot-fixes on top.

The other patch you mentioned was just applied in Groovy and pushed.

Changed in linux (Ubuntu Groovy):
milestone: ubuntu-20.10 → groovy-updates
Changed in ubuntu-release-notes:
status: New → Won't Fix
Francis Ginther (fginther) wrote :

@xnox, we are preparing a groovy respin which will pick this up.

Changed in linux (Ubuntu Groovy):
milestone: groovy-updates → ubuntu-20.10
Changed in linux (Ubuntu Groovy):
status: Confirmed → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.8.0-25.26

---------------
linux (5.8.0-25.26) groovy; urgency=medium

  * groovy/linux: 5.8.0-25.26 -proposed tracker (LP: #1899940)

  * CVE-2020-12351
    - Bluetooth: L2CAP: Fix calling sk_filter on non-socket based channel

  * CVE-2020-12352
    - Bluetooth: A2MP: Fix not initializing all members

  * CVE-2020-12351 // CVE-2020-12352 // CVE-2020-24490
    - Bluetooth: Disable High Speed by default
    - Bluetooth: MGMT: Fix not checking if BT_HS is enabled
    - [Config] Disable BlueZ highspeed support

  * ec2-hibinit-agent needs to properly initialize swap file (LP: #1892728)
    - ext4: implement swap_activate aops using iomap

 -- Andrea Righi <email address hidden> Thu, 15 Oct 2020 12:09:24 +0200

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers