Kernel BUG in paravirt_enter_lazy_mmu when running as a Xen PV guest

[David Vrabel]

Xen PV guests may crash during boot in paravirt_enter_lazy_mmu() while expanding the grant table (usually when requested by blkfront, when booting). See an example trace below.

This is caused by calling functions that are unsafe in atomic context.

The fix (which has been submitted to 3.16) is available here (also attached):

https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=stable/for-linus-3.16&id=b7dd0e350e0bd4c0fddcc9b8958342700b00b168

The fix is applicable to all kernel since 2.6.39 but only appears to trigger with the 3.13 kernel in 14.04.

[ 2.577876] ------------[ cut here ]------------
[ 2.577896] kernel BUG at /build/buildd/linux-3.13.0/arch/x86/kernel/paravirt.c:239!
[ 2.577910] invalid opcode: 0000 [#1] SMP
[ 2.577922] Modules linked in:
[ 2.577933] CPU: 0 PID: 1 Comm: init Not tainted 3.13.0-24-generic #46-Ubuntu
[ 2.577946] task: ec058000 ti: ec090000 task.ti: ec090000
[ 2.577955] EIP: 0061:[<c1645ebc>] EFLAGS: 00010002 CPU: 0
[ 2.577973] EIP is at enter_lazy.part.1+0x3/0x5
[ 2.577982] EAX: 00000001 EBX: ec0cc000 ECX: 00581980 EDX: 00000000
[ 2.577992] ESI: edc00000 EDI: edc00000 EBP: ec091a50 ESP: ec091a50
[ 2.578001] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[ 2.578014] CR0: 8005003b CR2: bfca2fa4 CR3: 2c392000 CR4: 00002660
[ 2.578027] Stack:
[ 2.578032] ec091a58 c1046564 ec091ab4 c1146fd3 fa83b2da edc00fff edc01000 c1a93018
[ 2.578052] 00000000 edc00fff 00000000 c193ce80 edc01000 00000000 00000000 00000000
[ 2.578076] ed3ef588 ed3ef588 00000000 c1b87b70 ec091ad0 edc01000 c1b65310 00001000
[ 2.578096] Call Trace:
[ 2.578111] [<c1046564>] paravirt_enter_lazy_mmu+0x24/0x30
[ 2.578127] [<c1146fd3>] apply_to_page_range+0x1a3/0x310
[ 2.578141] [<c1008eb8>] arch_gnttab_map_status+0x38/0x60
[ 2.578152] [<c1008d70>] ? map_pte_fn+0x70/0x70
[ 2.578166] [<c13ab020>] gnttab_map_frames_v2+0xb0/0x100
[ 2.578182] [<c13ab205>] gnttab_map+0x95/0x120
[ 2.578198] [<c12c7ff0>] ? blk_update_request+0x190/0x340
[ 2.578209] [<c13ab363>] get_free_entries+0xd3/0x280
[ 2.578221] [<c13ab5d3>] gnttab_alloc_grant_references+0x13/0x30
[ 2.578238] [<c1424be5>] do_blkif_request+0x535/0x6f0
[ 2.578253] [<c16523dc>] ? _raw_spin_unlock_irqrestore+0x1c/0x40
[ 2.578269] [<c12c57ee>] __blk_run_queue+0x2e/0x40
[ 2.578280] [<c12c5825>] blk_start_queue+0x25/0x40
[ 2.578291] [<c1424dbe>] kick_pending_request_queues+0x1e/0x30
[ 2.578304] [<c142546f>] blkif_interrupt+0x69f/0x740
[ 2.578318] [<c100654f>] ? xen_set_pte_at+0xbf/0xf0
[ 2.578335] [<c10a5ba5>] handle_irq_event_percpu+0x35/0x1a0
[ 2.578351] [<c12f136a>] ? radix_tree_lookup+0xa/0x10
[ 2.578364] [<c10a5d41>] handle_irq_event+0x31/0x50
[ 2.578376] [<c10a8036>] handle_edge_irq+0x66/0x110
[ 2.578389] [<c13ac246>] __xen_evtchn_do_upcall+0x1c6/0x2c0
[ 2.578402] [<c13ae100>] xen_evtchn_do_upcall+0x20/0x40
[ 2.578415] [<c165a087>] xen_do_upcall+0x7/0xc
[ 2.578427] [<c1001227>] ? xen_hypercall_xen_version+0x7/0x20
[ 2.578441] [<c10083cf>] ? xen_force_evtchn_callback+0xf/0x20
[ 2.578454] [<c1008c50>] check_events+0x8/0xc
[ 2.578464] [<c1008c47>] ? xen_restore_fl_direct_reloc+0x4/0x4
[ 2.578480] [<c1006373>] ? xen_batched_set_pte+0xb3/0x160
[ 2.578493] [<c10064b8>] xen_set_pte_at+0x28/0xf0
[ 2.578505] [<c10048e6>] ? __raw_callee_save_xen_pte_val+0x6/0x8
[ 2.578521] [<c11447a8>] copy_pte_range+0x258/0x4c0
[ 2.578534] [<c1146d27>] copy_page_range+0x1d7/0x2e0
[ 2.578549] [<c105462e>] dup_mm+0x28e/0x4f0
[ 2.578561] [<c1055866>] copy_process.part.33+0xfa6/0x10d0
[ 2.578574] [<c1055b41>] do_fork+0xc1/0x2c0
[ 2.578591] [<c1067996>] ? SyS_rt_sigprocmask+0x76/0xa0
[ 2.578604] [<c1055e05>] SyS_clone+0x25/0x30
[ 2.578615] [<c1659b4d>] sysenter_do_call+0x12/0x28
[ 2.578626] Code: c4 1c 5b 5e 5f 5d c3 55 89 e5 f3 0f b8 c0 90 5d c3 55 ba a0 2c aa c1 89 e5 b9 25 00 00 00 57 31 c0 89 d7 f3 ab 5f 5d c3 55 89 e5 <0f> 0b 55 89 e5 66 66 66 66 90 0f 0b 8b 15 28 d9 91 c1 55 89 e5
[ 2.578745] EIP: [<c1645ebc>] enter_lazy.part.1+0x3/0x5 SS:ESP 0069:ec091a50
[ 2.578765] ---[ end trace ab5b5344be71ca3d ]---
[ 2.578775] Kernel panic - not syncing: Fatal exception in interrupt

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1350373

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[David Vrabel (david-vrabel)]

appport-collect crashed when attempting to view the report so I've not attached any additional logs as requested by the bot.

I hope this isn't a problem since I have already diagnosed the bug and provided the correct fix.

[Joseph Salisbury (jsalisbury)]

I don't see that the patch landed in mainline as of yet:
x86/xen: safely map and unmap grant frames when in atomic context

As soon as it does, I can build a trusty test kernel with a cherry pick of that commit and we can confirm it fixes this bug.

[David Vrabel (david-vrabel)]

Commit b7dd0e350e0bd4c0fddcc9b8958342700b00b168 (x86/xen: safely map and unmap grant frames when in atomic context)

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b7dd0e350e0bd4c0fddcc9b8958342700b00b168

[Tim Gardner (timg-tpi)]

Test kernel at http://kernel.ubuntu.com/~rtg/3.13.0-9b0a74ddd711e5a6b07b75eb97e985abecdf657d-lp1350373/

[David Vrabel (david-vrabel)]

Thanks for the test kernel. Unfortunately it crashes in gnttab_init() when booting. I think the backport to 3.13 is not correct.

I will try to produced a working backported patch.

[David Vrabel (david-vrabel)]

Here is a working patch against 3.13.10.

[Andrey Kislyuk (weaver)]

We can confirm that we see a large fraction of our 14.04 Trusty Xen PV instances exhibit a stack trace similar to the one referenced above in paravirt_enter_lazy_mmu, followed by a crash.

[Ben Howard (darkmuggle-deactivatedaccount)]

Raising the severity due to the impact on EC2.

[Stefan Bader (smb)]

Thanks for the backport David. I build some test kernels with that and put them to http://people.canonical.com/~smb/lp1350373/ (note that you do *not* need linux-image-extra for ec2 installations). This should get pulled in as stable patch but it would be even better if someone can confirm the test kernels working. Thanks.

[Andrey Kislyuk (weaver)]

Attaching a typical trace on one of our affected machines. This shows the crash happening roughly 33 seconds after boot, which is where we observe most of these. Some also happen much later after boot.

[Kamal Mostafa (kamalmostafa)]

Thanks David! I've queued up your comment #8 backport for v3.13-stable (it will appear in v3.13.11.6).

[David Vrabel (david-vrabel)]

Thanks Stefan, I have tested that test kernel and it seems ok.

[Andrey Kislyuk (weaver)]

Is there an estimate for when this will hit release and cloud images? We are blocked by this, and I don't see a reference to it in https://launchpad.net/ubuntu/+source/linux.

Thanks.

[Stefan Bader (smb)]

As far as I can see, this should be part of 3.13.0-35.62 which is currently in proposed. According to the SRU cycle this should go into updates this week (possibly end of week). Unfortunately the changelog has no reference to this bug report. The cloud images are iirc build daily based on updates. So they should then follow soon.

[Stefan Bader (smb)]

Released in kernel 3.13.0-35.62.

[Stefan Bader (smb)]

While checking Utopic I found

commit 434d59e0718f5fdf91ff9a9bb6c98887e42ba912
Author: David Vrabel <email address hidden>
Date: Tue Aug 5 11:49:19 2014 +0100

    x86/xen: use vmap() to map grant table pages in PVH guests

    commit 7d951f3ccb0308c95bf76d5eef9886dea35a7013 upstream.

    Commit b7dd0e350e0b (x86/xen: safely map and unmap grant frames when
    in atomic context) causes PVH guests to crash in
    arch_gnttab_map_shared() when they attempted to map the pages for the
    grant table.

Not sure PVH is considered to be working with Trusty (12.04) but if yes, this sounds like it might be backport to be requested. David?

[Stefan Bader (smb)]

Was part of upstream 3.16.

[David Vrabel (david-vrabel)]

PVH is experimental and the hypervisor ABI is subject to change so I wouldn't try making it work in a distro kernel.

[Stefan Bader (smb)]

Ok, sounds reasonable. So we are good wrt this bug. Thanks.