4.4.0-7.22 no longer boots on arm64

Bug #1547718 reported by dann frazier
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Critical
Chris J Arges
Xenial
Fix Released
Critical
Chris J Arges

Bug Description

Observed on X-Gene and ThunderX.

[ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
[ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
[ 2.071185] Modules linked in:
[ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
[ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
[ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
[ 2.094840] PC is at apply_to_page_range+0x330/0x378
[ 2.099780] LR is at change_memory_common+0x94/0x148
[ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
[ 2.112073] sp : ffff8001df4a7b30
[ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
[ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
[ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
[ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
[ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
[ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
[ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
[ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
[ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
[ 2.163091] x11: 0000000000000018 x10: 0101010101010101
[ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
[ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
[ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
[ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
[ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
[ 2.194902]
[ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
[ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
[ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
[ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d56000 0008000000000000
[ 2.224862] 7b60: 0000000000000080 ffff7ffffc009500 ffff80000009aa18 ffff7ffffc009550
[ 2.232651] 7b80: ffff8001df4a7df0 0000000000000000 ffff800000f11800 ffff7ffffc00efff
[ 2.240440] 7ba0: ffff7ffffc00f000 ffff7ffffc00f000 ffff7ffffc00f000 ffff800000000000
[ 2.248227] 7bc0: ffff7ffffc00efff 0000000000000000 ffff800000d93af0 ffff800000d93af0
[ 2.256016] 7be0: ffff8001df4a7c40 ffff800000d93af0 ffff8001df4a7c50 ffff80000009aac8
[ 2.263805] 7c00: ffff7ffffc00f000 0000000000000000 ffff7ffffc009300 ffff800000d56000
[ 2.271594] 7c20: ffff800000d80000 0000000000000080 0008000000000000 0000000000000080
[ 2.279383] 7c40: 0008000000000000 cb88537fdc8ba602 ffff8001df4a7c70 ffff800000145400
[ 2.287172] 7c60: ffff7ffffc009550 ffff80000009aa98 ffff8001df4a7c90 ffff800000147350
[ 2.294961] 7c80: ffff7ffffc009300 ffff80000009aa98 ffff8001df4a7cb0 ffff8000001499cc
[ 2.302750] 7ca0: ffff7ffffc009318 ffff8001df4a7e68 ffff8001df4a7e30 ffff800000149ec8
[ 2.310539] 7cc0: ffff800000d56000 0000000000000000 0000000000000006 0000ffff94668108
[ 2.318327] 7ce0: 0000000080000000 0000000000000015 000000000000011d 0000000000000111
[ 2.326116] 7d00: ffff8000008d7000 ffff8001df4a4000 ffff000000adb000 ffff800000e35220
[ 2.333905] 7d20: ffff800000e354a0 ffff7ffffc00f000 ffff000000aef878 ffff8000008e7f68
[ 2.341693] 7d40: 0000000000000072 ffff800100000064 ffff80010000006e 0000003f00000000
[ 2.349482] 7d60: ffff7ffffc00f000 0000000000000018 ffff8001df4a7e30 ffff800000149e8c
[ 2.357271] 7d80: ffff800000d56000 0000000000000000 0000000000000006 0000ffff94668108
[ 2.365060] 7da0: 0000000000000000 0000000000000000 0000000000000000 ffff800000000000
[ 2.372849] 7dc0: 0000000000000000 ffff81a400000000 0000000000000000 0000000000000000
[ 2.380638] 7de0: 0000000000000000 0000000000000000 00006c656e72656b 0000000000000000
[ 2.388427] 7e00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2.396216] 7e20: 0000000000000000 cb88537fdc8ba602 0000ffffc2eeb2c0 ffff800000085c70
[ 2.404005] 7e40: 0000000000000000 0000000000000000 ffffffffffffffff 0000ffff945a8b54
[ 2.411794] 7e60: 000000000000011d ffff000000adb000 00000000000148f8 ffff000000aef078
[ 2.419583] 7e80: ffff000000aeef28 ffff000000ae9860 000000000000a000 000000000000c130
[ 2.427372] 7ea0: ffff7ffffc009220 0000000100000005 0000002100000020 0000001400000019
[ 2.435161] 7ec0: 0000000000000013 cb88537fdc8ba602 0000000000000006 0000ffff94668108
[ 2.442950] 7ee0: 0000000000000000 0000000000000006 0000000000000000 0000ffffc2eeb238
[ 2.450739] 7f00: 644d39656dff7364 644d39656dff7364 0000000000000111 0000000000000002
[ 2.458528] 7f20: 0101010101010101 000000000000000c 0000000000000010 6567782f6d70612f
[ 2.466317] 7f40: 656e6567782f656e 0000ffff944e7694 0000ffff945a8b30 0000ffff9467a1d0
[ 2.474105] 7f60: 0000000000000000 0000aaaad2a8a500 0000000000000000 0000ffff94668108
[ 2.481894] 7f80: 0000aaaad2a99d90 0000000000020000 0000000000000000 0000aaaad2a8a380
[ 2.489682] 7fa0: 0000000000000000 0000000000020000 0000ffffc2eeb450 0000ffffc2eeb2c0
[ 2.497471] 7fc0: 0000ffff94661980 0000ffffc2eeb2c0 0000ffff945a8b54 0000000080000000
[ 2.505260] 7fe0: 0000000000000006 0000000000000111 0000000000000000 0000000000000000
[ 2.513047] Call trace:
[ 2.515481] [<ffff8000001f8f20>] apply_to_page_range+0x330/0x378
[ 2.521455] [<ffff80000009a964>] change_memory_common+0x94/0x148
[ 2.527429] [<ffff80000009aac8>] set_memory_ro+0x30/0x40
[ 2.532714] [<ffff800000145400>] frob_rodata+0x50/0x78
[ 2.537824] [<ffff800000147350>] module_enable_ro+0x58/0x68
[ 2.543366] [<ffff8000001499cc>] load_module+0x12a4/0x14c8
[ 2.548822] [<ffff800000149ec8>] SyS_finit_module+0xf0/0x128
[ 2.554451] [<ffff800000085c70>] el0_svc_naked+0x24/0x28
[ 2.559735] Code: 97fff8a0 35fffbe0 f9400300 17ffffce (d4210000)
[ 2.565807] ---[ end trace a9373e7fe017e4db ]---

Revision history for this message
Tim Gardner (timg-tpi) wrote :

Dann - Paolo and I have seen this same problem with raspi2, which makes me think perhaps it came in with stable updates v4.4.2.

Revision history for this message
dann frazier (dannf) wrote : Re: [Bug 1547718] Re: 4.4.0-7.22 no longer boots on arm64

On Fri, Feb 19, 2016 at 4:56 PM, Tim Gardner <email address hidden> wrote:
> Dann - Paolo and I have seen this same problem with raspi2, which makes
> me think perhaps it came in with stable updates v4.4.2.

I'm guessing it's the livepatch module stuff - I've got a testbuild
going w/ that reverted.

  -dann

Revision history for this message
dann frazier (dannf) wrote :

On Fri, Feb 19, 2016 at 5:21 PM, Dann Frazier
<email address hidden> wrote:
> On Fri, Feb 19, 2016 at 4:56 PM, Tim Gardner <email address hidden> wrote:
>> Dann - Paolo and I have seen this same problem with raspi2, which makes
>> me think perhaps it came in with stable updates v4.4.2.
>
> I'm guessing it's the livepatch module stuff - I've got a testbuild
> going w/ that reverted.

Yep - it is. These reverts get things booting again:

commit e96e20134729121689a0089537c6ed7ebba6e0d5
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:40 2016 -0700

    Revert "module: Use the same logic for setting and unsetting RO/NX"

    This reverts commit 39432f1e05eb5ff177b60a547e44f81f7ec59097.

commit c18184273f53c4f91020d07c5df4cc15f4e2b88f
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:34 2016 -0700

    Revert "gcov: use within_module() helper."

    This reverts commit 581d096d9988adf67ae4b9dd881bfa24aa20a0b6.

commit 42a1b19b7e24d6c2375de5bdf200f5240e556e55
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:24 2016 -0700

    Revert "module: use a structure to encapsulate layout."

    This reverts commit 6404731da759dc7c1551e3d45e0006d3be6beacc.

commit 65dae6242fda8769903c681755358e341b9a11ad
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:14 2016 -0700

    Revert "module: clean up RO/NX handling."

    This reverts commit c8d73ebfe19daac81b7cb5c8d1dd700849bad0b3.

commit 5079a2c03405c0c34af597a2340b773b424ed014
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:01 2016 -0700

    Revert "module: keep percpu symbols in module's symtab"

    This reverts commit 32d96e18fca1f1181c17d9e636efd64ea79bd9fe.

commit 9206884e139260f102d0490c13b0c290d70538fc
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:21:55 2016 -0700

    Revert "livepatch: Cleanup module page permission changes"

    This reverts commit 83f79af46d38abd463ec3eb26b6e0a6e5c59336a.

Tim Gardner (timg-tpi)
Changed in linux (Ubuntu Xenial):
assignee: nobody → Chris J Arges (arges)
Revision history for this message
Ming Lei (tom-leiming) wrote :
Download full text (7.0 KiB)

Looks it is enough to just revert
'e96e20134729121689a0089537c6ed(module: clean up RO/NX handling)'
for fixing the issue.

But the interesting thing is that there isn't the problem in upstream kernel
4.5-rc5, and the commit(module: clean up RO/NX handling) isn't reverted
in upstream yet.

So looks there is one real fix in upstream kernel.

Thanks,

On Mon, Feb 22, 2016 at 12:07 PM, Tim Gardner <email address hidden> wrote:
> ** Changed in: linux (Ubuntu Xenial)
> Assignee: (unassigned) => Chris J Arges (arges)
>
> --
> You received this bug notification because you are subscribed to linux
> in Ubuntu.
> https://bugs.launchpad.net/bugs/1547718
>
> Title:
> 4.4.0-7.22 no longer boots on arm64
>
> Status in linux package in Ubuntu:
> Confirmed
> Status in linux source package in Xenial:
> Confirmed
>
> Bug description:
> Observed on X-Gene and ThunderX.
>
> [ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
> [ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
> [ 2.071185] Modules linked in:
> [ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
> [ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
> [ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
> [ 2.094840] PC is at apply_to_page_range+0x330/0x378
> [ 2.099780] LR is at change_memory_common+0x94/0x148
> [ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
> [ 2.112073] sp : ffff8001df4a7b30
> [ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
> [ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
> [ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
> [ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
> [ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
> [ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
> [ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
> [ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
> [ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
> [ 2.163091] x11: 0000000000000018 x10: 0101010101010101
> [ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
> [ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
> [ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
> [ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
> [ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
> [ 2.194902]
> [ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
> [ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
> [ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
> [ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d56000 0008000000000000
> [ 2.224862] 7b60: 0000000000000080 ffff7ffffc009500 ffff80000009aa18 ffff7ffffc009550
> [ 2.232651] 7b80: ffff8001df4a7df0 0000000000000000 ffff800000f11800 ffff7ffffc00efff
> [ 2.240440] 7ba0: ffff7ffffc00f000 ffff7ffffc00f000 ffff7ffffc00f000 ffff800000000000
> [ 2.248227] 7bc0: ffff7ffffc00efff 0000000000000000...

Read more...

Revision history for this message
Ming Lei (tom-leiming) wrote :
Download full text (7.4 KiB)

On Mon, Feb 22, 2016 at 4:37 PM, Ming Lei <email address hidden> wrote:
> Looks it is enough to just revert
> 'e96e20134729121689a0089537c6ed(module: clean up RO/NX handling)'
> for fixing the issue.
>
> But the interesting thing is that there isn't the problem in upstream kernel
> 4.5-rc5, and the commit(module: clean up RO/NX handling) isn't reverted
> in upstream yet.
>
> So looks there is one real fix in upstream kernel.

The issue was fixed by 57adec866c0440976c96a4b8f5b59f
(arm64: mm: avoid calling apply_to_page_range on empty range), and
this patch is marked as -stalbe.

I have sent this patch outto uk list already.

Thanks,

>
> Thanks,
>
> On Mon, Feb 22, 2016 at 12:07 PM, Tim Gardner <email address hidden> wrote:
>> ** Changed in: linux (Ubuntu Xenial)
>> Assignee: (unassigned) => Chris J Arges (arges)
>>
>> --
>> You received this bug notification because you are subscribed to linux
>> in Ubuntu.
>> https://bugs.launchpad.net/bugs/1547718
>>
>> Title:
>> 4.4.0-7.22 no longer boots on arm64
>>
>> Status in linux package in Ubuntu:
>> Confirmed
>> Status in linux source package in Xenial:
>> Confirmed
>>
>> Bug description:
>> Observed on X-Gene and ThunderX.
>>
>> [ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
>> [ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
>> [ 2.071185] Modules linked in:
>> [ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
>> [ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
>> [ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
>> [ 2.094840] PC is at apply_to_page_range+0x330/0x378
>> [ 2.099780] LR is at change_memory_common+0x94/0x148
>> [ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
>> [ 2.112073] sp : ffff8001df4a7b30
>> [ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
>> [ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
>> [ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
>> [ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
>> [ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
>> [ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
>> [ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
>> [ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
>> [ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
>> [ 2.163091] x11: 0000000000000018 x10: 0101010101010101
>> [ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
>> [ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
>> [ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
>> [ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
>> [ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
>> [ 2.194902]
>> [ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
>> [ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
>> [ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
>> [ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d560...

Read more...

Revision history for this message
Paolo Pisati (p-pisati) wrote :

It affects armhf too, i'm sending a SAUCE PATCH containing the arm side of the fix (identical to the arm64 BTW).

Revision history for this message
Paolo Pisati (p-pisati) wrote :
tags: added: patch
tags: added: kernel-da-key
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (10.1 KiB)

This bug was fixed in the package linux - 4.4.0-8.23

---------------
linux (4.4.0-8.23) xenial; urgency=low

  * cgroup namespace mounts broken in containers (LP: #1549398)
    - SAUCE: kernfs: Always set super block owner to init_user_ns

  * 4.4.0-7.22 no longer boots on arm64 (LP: #1547718)
    - arm64: mm: avoid calling apply_to_page_range on empty range
    - UBUNTU SAUCE: arm: mm: avoid calling apply_to_page_range on empty range

  * kernel install failed /bin/cp: cannot stat ‘/boot/initrd.img-4.3.0-7-generic’: No such file or directory (LP: #1536810)
    - [Config] postinst -- handle recreating symlinks when a real file is present

  * insecure overlayfs xattrs handling in copy_up (LP: #1534961)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs
    - SAUCE: overlayfs: Be more careful about copying up sxid files
    - SAUCE: overlayfs: Propogate nosuid from lower and upper mounts

  * overlayfs over fuse should refuse copy_up of files if uid/gid not mapped (LP: #1535150)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs
    - SAUCE: overlayfs: Be more careful about copying up sxid files
    - SAUCE: overlayfs: Propogate nosuid from lower and upper mounts

  * overlay: mkdir fails if directory exists in lowerdir in a user namespace (LP: #1531747)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs

  * Update Intel ethernet drivers to Fortville SW5 (LP: #1547674)
    - net: bulk free infrastructure for NAPI context, use napi_consume_skb
    - net: Add eth_platform_get_mac_address() helper.
    - i40e: Add mac_filter_element at the end of the list instead of HEAD
    - i40e/i40evf: Fix RSS rx-flow-hash configuration through ethtool
    - i40e: Replace X722 mac check in ethtool get_settings
    - i40evf: allow channel bonding of VFs
    - i40e: define function capabilities in only one place
    - i40evf: null out ring pointers on free
    - i40e: Cleanup the code with respect to restarting autoneg
    - i40e: update features with right offload
    - i40e: bump version to 1.4.10
    - i40e: add new device IDs for X722
    - i40e: Extend ethtool RSS hooks for X722
    - i40e/i40evf: Fix for UDP/TCP RSS for X722
    - i40evf: add new write-back mode
    - i40e/i40evf: Use private workqueue
    - i40e: add new proxy-wol bit for X722
    - i40e: Limit DCB FW version checks to X710/XL710 devices
    - i40e: AQ Add Run PHY Activity struct
    - i40e: AQ Geneve cloud tunnel type
    - i40e: AQ Add external power class to get link status
    - i40e: add 100Mb ethtool reporting
    - ixgbe: bulk free SKBs during TX completion cleanup cycle
    - igb: Remove unnecessary flag setting in igb_set_flag_queue_pairs()
    - igb: Unpair the queues when changing the number of queues...

Changed in linux (Ubuntu Xenial):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.