kernel crash with 0010:ovl_open_realfile+0x4a/0x150 [overlay] in Qemu with focal daily

[Jean-Baptiste Lallement]

Focal daily 20200507 (started earlier)
Kernel is 5.4.0-30.34

Kernel crashes on boot in Qemu with:

[ 4.019922] RIP: 0010:ovl_open_realfile+0x4a/0x150 [overlay]
[ 4.022101] Code: 44 8b 6f 40 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 49 8b 7e 28 41 81 cd 00 00 04 04 e8 9d d4 ff ff 49 89 c7 48 8b 45 c8 <48> 8b 40 68 48 81 78 60 62 6a 65 6a 0f 84 a6 00 00 00 65 48 8b 04
[ 4.027487] RSP: 0018:ffffac44c0707c08 EFLAGS: 00010202
[ 4.029424] RAX: 0f230ca096f12900 RBX: ffff9e44e9c2eaa0 RCX: 0000000000000001
[ 4.031645] RDX: ffff9e44ea9ae540 RSI: ffff9e44e9c2eaa0 RDI: ffff9e44fb281f00
[ 4.033811] RBP: ffffac44c0707c50 R08: ffff9e44e9e3cc00 R09: 0000000000000000
[ 4.036428] R10: ffff9e44ea9ae540 R11: 0000000000000000 R12: ffff9e44f9ec9c00
[ 4.038737] R13: 0000000004048002 R14: ffff9e44eaa51b00 R15: ffff9e44e9e3cc00
[ 4.046087] FS: 00007f91962a6600(0000) GS:ffff9e44fbb00000(0000) knlGS:0000000000000000
[ 4.051277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4.055455] CR2: 000056295fbc4a59 CR3: 0000000169c80000 CR4: 00000000000006e0
[ 4.059614] general protection fault: 0000 [#4] SMP NOPTI
[ 4.061700] CPU: 1 PID: 360 Comm: cp Tainted: G D 5.4.0-30-generic #34-Ubuntu
[ 4.063016] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 4.063016] RIP: 0010:ovl_open_realfile+0x4a/0x150 [overlay]
[ 4.063016] Code: 44 8b 6f 40 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 49 8b 7e 28 41 81 cd 00 00 04 04 e8 9d d4 ff ff 49 89 c7 48 8b 45 c8 <48> 8b 40 68 48 81 78 60 62 6a 65 6a 0f 84 a6 00 00 00 65 48 8b 04
[ 4.063016] RSP: 0018:ffffac44c0373c08 EFLAGS: 00010206
[ 4.063016] RAX: 1579f0d048c0c300 RBX: ffff9e44eaa3f478 RCX: 0000000000000000
[ 4.063016] RDX: 0000000000000000 RSI: ffff9e44eaa3f478 RDI: ffff9e44fb281f00
[ 4.063016] RBP: ffffac44c0373c50 R08: ffff9e44fb3ac6c0 R09: 0000000000000000
[ 4.063016] R10: 0000000000000001 R11: 0000007461642e73 R12: ffff9e44f9b2d700
[ 4.063016] R13: 0000000004048000 R14: ffff9e44fa3d65c0 R15: ffff9e44fb3ac6c0
[ 4.063016] FS: 00007f91962a6600(0000) GS:ffff9e44fbb00000(0000) knlGS:0000000000000000
[ 4.063016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4.063016] CR2: 000056295fbc4a59 CR3: 0000000169c80000 CR4: 00000000000006e0
[ 4.063016] Call Trace:
[ 4.063016] ? ovl_llseek+0x120/0x120 [overlay]
[ 4.063016] ovl_open+0x57/0x90 [overlay]
[ 4.063016] do_dentry_open+0x143/0x3a0
[ 4.063016] vfs_open+0x2d/0x30
[ 4.063016] do_last+0x194/0x900
[ 4.063016] path_openat+0x8d/0x290
[ 4.063016] do_filp_open+0x91/0x100
[ 4.063016] ? __alloc_fd+0x46/0x150
[ 4.063016] do_sys_open+0x17e/0x290
[ 4.063016] __x64_sys_openat+0x20/0x30
[ 4.124120] do_syscall_64+0x57/0x190
[ 4.124120] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 4.124120] RIP: 0033:0x7f91961c3d1b
[ 4.124120] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 4c 24 28 64 48 33 0c 25
[ 4.124120] RSP: 002b:00007ffebc71f1b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 4.124120] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f91961c3d1b
[ 4.124120] RDX: 0000000000000000 RSI: 0000562960241670 RDI: 00000000ffffff9c
[ 4.124120] RBP: 0000562960241670 R08: 0000000000000001 R09: 0000000000000013
[ 4.124120] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 4.124120] R13: 00000000ffffffff R14: 00007ffebc71f298 R15: 0000000000000000
[ 4.124120] Modules linked in: overlay nls_utf8 isofs dm_mirror dm_region_hash dm_log hid_generic usbhid hid qxl ttm drm_kms_helper ahci syscopyarea sysfillrect virtio_net net_failover i2c_i801 libahci psmouse lpc_ich sysimgblt fb_sys_fops failover virtio_blk drm

Reproduced on bionic and focal hosts.
Groovy daily boots fine.
boot message attached.

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: qemu 1:4.2-3ubuntu6
ProcVersionSignature: Ubuntu 5.4.0-18.22-generic 5.4.24
Uname: Linux 5.4.0-18-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu32
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Fri May 8 15:36:38 2020
Dependencies:

InstallationDate: Installed on 2014-07-15 (2124 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Alpha amd64 (20140520)
MachineType: Gigabyte Technology Co., Ltd. GA-890GPA-UD3H
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-18-generic root=UUID=0889d734-ea91-4bdb-9d16-2a0a923ad3d1 ro
SourcePackage: qemu
UpgradeStatus: Upgraded to groovy on 2018-03-24 (775 days ago)
dmi.bios.date: 07/23/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: FD
dmi.board.name: GA-890GPA-UD3H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrFD:bd07/23/2010:svnGigabyteTechnologyCo.,Ltd.:pnGA-890GPA-UD3H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-890GPA-UD3H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-890GPA-UD3H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Paride Legovini (paride)]

This is happening with the Focal live-server images too, starting from the image with serial 20200506. The previous daily (20200505) booted and installed fine.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

This is reported against 5.4.0-30.34 (in the guest I assume).
Per [1] that date window is the change from

2020-05-08 12:58:43 Published Focal proposed main devel 5.4.0-31.35
2020-05-08 13:00:19 Superseded Focal proposed main devel 5.4.0-30.34
2020-05-05 12:18:15 Published Focal security main devel 5.4.0-29.33
2020-05-04 19:13:24 Published Focal updates main devel 5.4.0-29.33

Given the crash it is either the new proposed kernel if it is in there or something in how the overlayfs is created (that is where the crashes are).

I checked myself the images from [2][3][4] and they all worked for me.
This boots for example:
$ sudo qemu-system-x86_64 -m 1024 -cpu host -smp 4 -enable-kvm -hda focal-server-cloudimg-amd64-20200507.img -boot d -curses

@Jobel/Paride - could you add which images and which commandline you used?

[1]: https://launchpad.net/ubuntu/+source/linux/+publishinghistory
[2]: https://cloud-images.ubuntu.com/focal/20200505/focal-server-cloudimg-amd64.img
[3]: https://cloud-images.ubuntu.com/focal/20200506/focal-server-cloudimg-amd64.img
[4]: https://cloud-images.ubuntu.com/focal/20200507/focal-server-cloudimg-amd64.img

[Paride Legovini (paride)]

Speaking for server: the images are working now. I think the images were built with the proposed kernel and this caused trouble.

[Christian Ehrhardt  (paelzer)]

As it seems the issue is neither triggered nor resolvable by qemu - marking the bug task invalid.

[Jean-Baptiste Lallement (jibel)]

The bug automagically fixed itself. Setting to incomplete until it expires or can be reproduced.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]