xenial 4.4.0-191-generic in -proposed has a regression

[Shay Nagar]

[Impact]

The new xenial kernel in -proposed, 4.4.0-191-generic, and its derivatives, such as 4.4.0-1115-aws suffer a kernel panic on boot on AWS.

I tested t2.micro and t2.medium, it happens every time. Simply install the kernel from -proposed and reboot, we get the following oops:

[ 0.549557] BUG: unable to handle kernel paging request at ffffffffff5f3000
[ 0.552000] IP: [<ffffffff810592ff>] mp_irqdomain_activate+0x5f/0xa0
[ 0.552000] PGD 1e0f067 PUD 1e11067 PMD 1e12067 PTE 0
[ 0.552000] Oops: 0002 [#1] SMP
[ 0.552000] Modules linked in:
[ 0.552000] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.4.0-191-generic #221-Ubuntu
[ 0.552000] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 0.552000] task: ffff88010a0f0000 ti: ffff88010a0f8000 task.ti: ffff88010a0f8000
[ 0.552000] RIP: 0010:[<ffffffff810592ff>] [<ffffffff810592ff>] mp_irqdomain_activate+0x5f/0xa0
[ 0.552000] RSP: 0000:ffff88010a0fbc48 EFLAGS: 00010086
[ 0.552000] RAX: 0000000000000086 RBX: ffff88010a1df480 RCX: 0000000000000000
[ 0.552000] RDX: ffffffffff5f3000 RSI: 0000000000000001 RDI: 000000000020c000
[ 0.552000] RBP: ffff88010a0fbc50 R08: ffffffff81ebdfd0 R09: 00000000ffffffff
[ 0.552000] R10: 0000000000000011 R11: 0000000000000009 R12: ffff88010ad95400
[ 0.552000] R13: 0000000000000001 R14: 0000000000000009 R15: ffff88010a1fc480
[ 0.552000] FS: 0000000000000000(0000) GS:ffff88010b240000(0000) knlGS:0000000000000000
[ 0.552000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.552000] CR2: ffffffffff5f3000 CR3: 0000000001e0a000 CR4: 0000000000160670
[ 0.552000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.552000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.552000] Stack:
[ 0.552000] ffff88010ac0ba58 ffff88010a0fbc70 ffffffff810ea644 ffff88010ac0ba00
[ 0.552000] ffff88010ac0ba58 ffff88010a0fbca0 ffffffff810e6d88 ffffffff810e1009
[ 0.552000] ffff88010ac0ba00 ffff88010ac0baa0 ffff88010a1fc480 ffff88010a0fbd38
[ 0.552000] Call Trace:
[ 0.552000] [<ffffffff810ea644>] irq_domain_activate_irq+0x44/0x50
[ 0.552000] [<ffffffff810e6d88>] irq_startup+0x38/0x90
[ 0.552000] [<ffffffff810e1009>] ? vprintk_default+0x29/0x40
[ 0.552000] [<ffffffff810e55e2>] __setup_irq+0x5a2/0x650
[ 0.552000] [<ffffffff811fc064>] ? kmem_cache_alloc_trace+0x1d4/0x1f0
[ 0.552000] [<ffffffff814a3870>] ? acpi_osi_handler+0xb0/0xb0
[ 0.552000] [<ffffffff810e582b>] request_threaded_irq+0xfb/0x1a0
[ 0.552000] [<ffffffff814a3870>] ? acpi_osi_handler+0xb0/0xb0
[ 0.552000] [<ffffffff814bf624>] ? acpi_ev_sci_dispatch+0x64/0x64
[ 0.552000] [<ffffffff814a3f0a>] acpi_os_install_interrupt_handler+0xaa/0x100
[ 0.552000] [<ffffffff81fb26e1>] ? acpi_sleep_proc_init+0x28/0x28
[ 0.552000] [<ffffffff814bf689>] acpi_ev_install_sci_handler+0x23/0x25
[ 0.552000] [<ffffffff814bcf03>] acpi_ev_install_xrupt_handlers+0x1c/0x6c
[ 0.552000] [<ffffffff81fb3e9d>] acpi_enable_subsystem+0x8f/0x93
[ 0.552000] [<ffffffff81fb276c>] acpi_init+0x8b/0x2c4
[ 0.552000] [<ffffffff8141ee1e>] ? kasprintf+0x4e/0x70
[ 0.552000] [<ffffffff81fb26e1>] ? acpi_sleep_proc_init+0x28/0x28
[ 0.552000] [<ffffffff810021f5>] do_one_initcall+0xb5/0x200
[ 0.552000] [<ffffffff810a6fda>] ? parse_args+0x29a/0x4a0
[ 0.552000] [<ffffffff81f69152>] kernel_init_freeable+0x177/0x218
[ 0.552000] [<ffffffff8185dcf0>] ? rest_init+0x80/0x80
[ 0.552000] [<ffffffff8185dcfe>] kernel_init+0xe/0xe0
[ 0.552000] [<ffffffff8186aea5>] ret_from_fork+0x55/0x80
[ 0.552000] [<ffffffff8185dcf0>] ? rest_init+0x80/0x80
[ 0.552000] Code: 8d 1c d2 8d ba 0b 02 00 00 44 8d 51 11 42 8b 14 dd 74 ec 10 82 c1 e7 0c 48 63 ff 81 e2 ff 0f 00 00 48 81 ea 00 10 80 00 48 29 fa <44> 89 12 89 72 10 42 8b 14 dd 74 ec 10 82 83 c1 10 81 e2 ff 0f
[ 0.552000] RIP [<ffffffff810592ff>] mp_irqdomain_activate+0x5f/0xa0
[ 0.996006] RSP <ffff88010a0fbc48>
[ 0.996006] CR2: ffffffffff5f3000
[ 0.996006] ---[ end trace 1d0c3bd610d641a0 ]---
[ 1.012018] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1896725

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Matthew Ruffell (mruffell)]

Hello @shaynagar, thank you very much for reporting! We will look into this asap.

More details: 4.4.0-190-generic is fine, 4.4.0-191-generic contains the regression.

[Po-Hsu Lin (cypressyew)]

I can see this issue on our AWS test pool as well (instances c3.xlarge, c4.large, m3.large, m4.large, r3.large, t2.small, x1e.xlarge failed with deployment for 4.4.0-1115-aws)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sean Feole (sfeole)]

Testing picked this up a few hours before the bug was open, looks like its affecting quite a few instances as mentioned in comment #4. Marking Confirmed, tagging appropriately.

[Stefan Bader (smb)]

Just some additional data without any final conclusion. Talking to Sean, the affected instance type seem to be all Xen based. The provided stacktrace backs this. In the past that could have been a mix of paravirtualized (PV) and hardware virtualized (HVM). But since spectre/meltdown I expect that to be all HVM guests.

From the stacktrace the bug occurs early during setup of ACPI. At least that part has not changed on the 4.4 kernel side. There is one change in this cycle which changes where internal driver data gets attached to interrupt controller structures. Though that would be a change which was applied to all (4.4, 4.15, 5.4) kernels in this cycle.

[Stefan Bader (smb)]

So I can confirm the behavior with a local Xen Server and a HVM Xenial guest. While the current proposed 4.4 kernel crashes while setting up the acpi interrupt, this does not happen with a 4.15 kernel from proposed on a Bionic HVM guest. Still I was able to get the 4.4 kernel to boot by reverting the following patch. Despite it being applied to 4.4 and 4.15 proposed. So it sounds like something outside the scope of the patch itself that changed between 4.4 and 4,15 is causing the problems.

commit 110e8369be2953ee2d3f7fd00d66034daa4c695c
Author: Thomas Gleixner <email address hidden>
Date: Tue Aug 25 17:22:58 2020 +0200

    XEN uses irqdesc::irq_data_common::handler_data to store a per interrupt XEN data pointer which contains XEN specific information.

    BugLink: https://bugs.launchpad.net/bugs/1895031

    commit c330fb1ddc0a922f044989492b7fcca77ee1db46 upstream.

[Thadeu Lima de Souza Cascardo (cascardo)]

mp_irqdomain_activate from arch/x86/kernel/apic/io_apic.c will use the chip_data that is set by the commit referred by Stefan ("XEN uses irqdesc::irq_data_common::handler_data to store a per interrupt XEN data pointer which contains XEN specific information.").

From drivers/xen/events/events_base.c, xen_allocate_irq_gsi has a test:
        if (gsi < nr_legacy_irqs())
                irq = gsi;
[...]
        xen_irq_init(irq);

And xen_irq_init calls irq_set_chip_data. That is not an irq number that had a irq_desc allocated by Xen, but one of the legacy IRQs, which is controlled by IO-APIC.

That still does not clarify why it works on bionic or upstream. But documenting what I was able to investigate so far.

Cascardo.

[Stefan Bader (smb)]

It looks like being a lucky "side-effect" of

commit 90ad9e2d91067983f3328e21b306323877e5f48a
Author: Thomas Gleixner <email address hidden>
Date: Wed Sep 13 23:29:49 2017 +0200

    x86/io_apic: Reevaluate vector configuration on activate()

This changes factors out the code that accesses the chip data, but at the same time limits this to only do this for the ioapic chip:

static void ioapic_configure_entry(struct irq_data *irqd)
+{
+ struct mp_chip_data *mpd = irqd->chip_data;
+ struct irq_cfg *cfg = irqd_cfg(irqd);
+ struct irq_pin_list *entry;
+
+ /*
+ * Only update when the parent is the vector domain, don't touch it
+ * if the parent is the remapping domain. Check the installed
+ * ioapic chip to verify that.
+ */
+ if (irqd->chip == &ioapic_chip) {
+ mpd->entry.dest = cfg->dest_apicid;
+ mpd->entry.vector = cfg->vector;
+ }
...

[Stefan Bader (smb)]

Or maybe that is not quite the change... I missed that the loop following, still would be using an anchor taken from the chip data...

[Shay Nagar (shaynagar)]

Just to clarify, When reverting one version back to 4.4.0-1114-aws,
no panic was found and instances boot successfully

[Stefan Bader (smb)]

For now we are going with reverting the Xen change that introduced the regression with 4.4 kernels.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ian May (ian-may)]

Hi Shay,

We are working to get verification testing completed, would you be able to confirm this issue is resolved in Xenial 4.4.0-192.222?

Thanks,
Ian

[Matthew Ruffell (mruffell)]

Verifying for xenial:

I enabled -proposed, and installed 4.4.0-1116-aws:

$ uname -rv
4.4.0-1116-aws #129-Ubuntu SMP Tue Sep 29 18:17:22 UTC 2020

I rebooted, and was able to ssh in, and dmesg was clean.

I then installed 4.4.0-192-generic:

$ uname -rv
4.4.0-192-generic #222-Ubuntu SMP Tue Sep 29 18:17:25 UTC 2020

I rebooted, and again was able to ssh in and dmesg was clean.

4.4.0-196-generic and 4.4.0-1116-aws solve the problem. Happy to mark as verified.

[Shay Nagar (shaynagar)]

Hi Ian and Matthew,

I was installed 4.4.0-1116-aws, machine was boot successfully and I was able to ssh.

Thanks again

[Po-Hsu Lin (cypressyew)]

Verified on T-aws 4.4.0-1080.84 as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-193.224

---------------
linux (4.4.0-193.224) xenial; urgency=medium

  * CVE-2020-16119
    - SAUCE: dccp: avoid double free of ccid on child socket

linux (4.4.0-192.222) xenial; urgency=medium

  * xenial/linux: 4.4.0-192.222 -proposed tracker (LP: #1897734)

  * mwifiex stops working after kernel upgrade (LP: #1897299)
    - mwifiex: Increase AES key storage size to 256 bits

  * xenial 4.4.0-191-generic in -proposed has a regression (LP: #1896725)
    - Revert "XEN uses irqdesc::irq_data_common::handler_data to store a per
      interrupt XEN data pointer which contains XEN specific information."

linux (4.4.0-191.221) xenial; urgency=medium

  * xenial/linux: 4.4.0-191.221 -proposed tracker (LP: #1896067)

  * Novalink (mkvterm command failure) (LP: #1892546)
    - tty: hvcs: Don't NULL tty->driver_data until hvcs_cleanup()

  * Xenial update: v4.4.236 upstream stable release (LP: #1895891)
    - HID: core: Correctly handle ReportSize being zero
    - HID: core: Sanitize event code and type when mapping input
    - perf record/stat: Explicitly call out event modifiers in the documentation
    - mm, page_alloc: remove unnecessary variable from free_pcppages_bulk
    - hwmon: (applesmc) check status earlier.
    - ceph: don't allow setlease on cephfs
    - s390: don't trace preemption in percpu macros
    - xen/xenbus: Fix granting of vmalloc'd memory
    - dmaengine: of-dma: Fix of_dma_router_xlate's of_dma_xlate handling
    - batman-adv: Avoid uninitialized chaddr when handling DHCP
    - batman-adv: bla: use netif_rx_ni when not in interrupt context
    - dmaengine: at_hdmac: check return value of of_find_device_by_node() in
      at_dma_xlate()
    - netfilter: nf_tables: incorrect enum nft_list_attributes definition
    - netfilter: nf_tables: fix destination register zeroing
    - dmaengine: pl330: Fix burst length if burst size is smaller than bus width
    - bnxt_en: Check for zero dir entries in NVRAM.
    - fix regression in "epoll: Keep a reference on files added to the check list"
    - tg3: Fix soft lockup when tg3_reset_task() fails.
    - iommu/vt-d: Serialize IOMMU GCMD register modifications
    - thermal: ti-soc-thermal: Fix bogus thermal shutdowns for omap4430
    - include/linux/log2.h: add missing () around n in roundup_pow_of_two()
    - btrfs: drop path before adding new uuid tree entry
    - btrfs: Remove redundant extent_buffer_get in get_old_root
    - btrfs: Remove extraneous extent_buffer_get from tree_mod_log_rewind
    - btrfs: set the lockdep class for log tree extent buffers
    - uaccess: Add non-pagefault user-space read functions
    - uaccess: Add non-pagefault user-space write function
    - btrfs: fix potential deadlock in the search ioctl
    - net: qmi_wwan: MDM9x30 specific power management
    - net: qmi_wwan: support "raw IP" mode
    - net: qmi_wwan: should hold RTNL while changing netdev type
    - net: qmi_wwan: ignore bogus CDC Union descriptors
    - Add Dell Wireless 5809e Gobi 4G HSPA+ Mobile Broadband Card (rev3) to
      qmi_wwan
    - qmi_wwan: Added support for Gemalto's Cinterion PHxx WWAN interface
    - qmi_wwan: add support for Quec...

[Alex Brett (alexbrett)]

For reference, the issue with the now backported patch was identified upstream (https://lkml.org/lkml/2020/9/30/352), and the following patch resolves it:

commit 0891fb39ba67bd7ae023ea0d367297ffff010781
Author: Juergen Gross <email address hidden>
Date: Wed Sep 30 11:16:14 2020 +0200
    xen/events: don't use chip_data for legacy IRQs