4.4.0-7.22 no longer boots on arm64

Bug #1547718 reported by dann frazier on 2016-02-19
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Critical
Chris J Arges
Xenial
Critical
Chris J Arges

Bug Description

Observed on X-Gene and ThunderX.

[ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
[ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
[ 2.071185] Modules linked in:
[ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
[ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
[ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
[ 2.094840] PC is at apply_to_page_range+0x330/0x378
[ 2.099780] LR is at change_memory_common+0x94/0x148
[ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
[ 2.112073] sp : ffff8001df4a7b30
[ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
[ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
[ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
[ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
[ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
[ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
[ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
[ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
[ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
[ 2.163091] x11: 0000000000000018 x10: 0101010101010101
[ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
[ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
[ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
[ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
[ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
[ 2.194902]
[ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
[ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
[ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
[ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d56000 0008000000000000
[ 2.224862] 7b60: 0000000000000080 ffff7ffffc009500 ffff80000009aa18 ffff7ffffc009550
[ 2.232651] 7b80: ffff8001df4a7df0 0000000000000000 ffff800000f11800 ffff7ffffc00efff
[ 2.240440] 7ba0: ffff7ffffc00f000 ffff7ffffc00f000 ffff7ffffc00f000 ffff800000000000
[ 2.248227] 7bc0: ffff7ffffc00efff 0000000000000000 ffff800000d93af0 ffff800000d93af0
[ 2.256016] 7be0: ffff8001df4a7c40 ffff800000d93af0 ffff8001df4a7c50 ffff80000009aac8
[ 2.263805] 7c00: ffff7ffffc00f000 0000000000000000 ffff7ffffc009300 ffff800000d56000
[ 2.271594] 7c20: ffff800000d80000 0000000000000080 0008000000000000 0000000000000080
[ 2.279383] 7c40: 0008000000000000 cb88537fdc8ba602 ffff8001df4a7c70 ffff800000145400
[ 2.287172] 7c60: ffff7ffffc009550 ffff80000009aa98 ffff8001df4a7c90 ffff800000147350
[ 2.294961] 7c80: ffff7ffffc009300 ffff80000009aa98 ffff8001df4a7cb0 ffff8000001499cc
[ 2.302750] 7ca0: ffff7ffffc009318 ffff8001df4a7e68 ffff8001df4a7e30 ffff800000149ec8
[ 2.310539] 7cc0: ffff800000d56000 0000000000000000 0000000000000006 0000ffff94668108
[ 2.318327] 7ce0: 0000000080000000 0000000000000015 000000000000011d 0000000000000111
[ 2.326116] 7d00: ffff8000008d7000 ffff8001df4a4000 ffff000000adb000 ffff800000e35220
[ 2.333905] 7d20: ffff800000e354a0 ffff7ffffc00f000 ffff000000aef878 ffff8000008e7f68
[ 2.341693] 7d40: 0000000000000072 ffff800100000064 ffff80010000006e 0000003f00000000
[ 2.349482] 7d60: ffff7ffffc00f000 0000000000000018 ffff8001df4a7e30 ffff800000149e8c
[ 2.357271] 7d80: ffff800000d56000 0000000000000000 0000000000000006 0000ffff94668108
[ 2.365060] 7da0: 0000000000000000 0000000000000000 0000000000000000 ffff800000000000
[ 2.372849] 7dc0: 0000000000000000 ffff81a400000000 0000000000000000 0000000000000000
[ 2.380638] 7de0: 0000000000000000 0000000000000000 00006c656e72656b 0000000000000000
[ 2.388427] 7e00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2.396216] 7e20: 0000000000000000 cb88537fdc8ba602 0000ffffc2eeb2c0 ffff800000085c70
[ 2.404005] 7e40: 0000000000000000 0000000000000000 ffffffffffffffff 0000ffff945a8b54
[ 2.411794] 7e60: 000000000000011d ffff000000adb000 00000000000148f8 ffff000000aef078
[ 2.419583] 7e80: ffff000000aeef28 ffff000000ae9860 000000000000a000 000000000000c130
[ 2.427372] 7ea0: ffff7ffffc009220 0000000100000005 0000002100000020 0000001400000019
[ 2.435161] 7ec0: 0000000000000013 cb88537fdc8ba602 0000000000000006 0000ffff94668108
[ 2.442950] 7ee0: 0000000000000000 0000000000000006 0000000000000000 0000ffffc2eeb238
[ 2.450739] 7f00: 644d39656dff7364 644d39656dff7364 0000000000000111 0000000000000002
[ 2.458528] 7f20: 0101010101010101 000000000000000c 0000000000000010 6567782f6d70612f
[ 2.466317] 7f40: 656e6567782f656e 0000ffff944e7694 0000ffff945a8b30 0000ffff9467a1d0
[ 2.474105] 7f60: 0000000000000000 0000aaaad2a8a500 0000000000000000 0000ffff94668108
[ 2.481894] 7f80: 0000aaaad2a99d90 0000000000020000 0000000000000000 0000aaaad2a8a380
[ 2.489682] 7fa0: 0000000000000000 0000000000020000 0000ffffc2eeb450 0000ffffc2eeb2c0
[ 2.497471] 7fc0: 0000ffff94661980 0000ffffc2eeb2c0 0000ffff945a8b54 0000000080000000
[ 2.505260] 7fe0: 0000000000000006 0000000000000111 0000000000000000 0000000000000000
[ 2.513047] Call trace:
[ 2.515481] [<ffff8000001f8f20>] apply_to_page_range+0x330/0x378
[ 2.521455] [<ffff80000009a964>] change_memory_common+0x94/0x148
[ 2.527429] [<ffff80000009aac8>] set_memory_ro+0x30/0x40
[ 2.532714] [<ffff800000145400>] frob_rodata+0x50/0x78
[ 2.537824] [<ffff800000147350>] module_enable_ro+0x58/0x68
[ 2.543366] [<ffff8000001499cc>] load_module+0x12a4/0x14c8
[ 2.548822] [<ffff800000149ec8>] SyS_finit_module+0xf0/0x128
[ 2.554451] [<ffff800000085c70>] el0_svc_naked+0x24/0x28
[ 2.559735] Code: 97fff8a0 35fffbe0 f9400300 17ffffce (d4210000)
[ 2.565807] ---[ end trace a9373e7fe017e4db ]---

Tim Gardner (timg-tpi) wrote :

Dann - Paolo and I have seen this same problem with raspi2, which makes me think perhaps it came in with stable updates v4.4.2.

On Fri, Feb 19, 2016 at 4:56 PM, Tim Gardner <email address hidden> wrote:
> Dann - Paolo and I have seen this same problem with raspi2, which makes
> me think perhaps it came in with stable updates v4.4.2.

I'm guessing it's the livepatch module stuff - I've got a testbuild
going w/ that reverted.

  -dann

dann frazier (dannf) wrote :

On Fri, Feb 19, 2016 at 5:21 PM, Dann Frazier
<email address hidden> wrote:
> On Fri, Feb 19, 2016 at 4:56 PM, Tim Gardner <email address hidden> wrote:
>> Dann - Paolo and I have seen this same problem with raspi2, which makes
>> me think perhaps it came in with stable updates v4.4.2.
>
> I'm guessing it's the livepatch module stuff - I've got a testbuild
> going w/ that reverted.

Yep - it is. These reverts get things booting again:

commit e96e20134729121689a0089537c6ed7ebba6e0d5
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:40 2016 -0700

    Revert "module: Use the same logic for setting and unsetting RO/NX"

    This reverts commit 39432f1e05eb5ff177b60a547e44f81f7ec59097.

commit c18184273f53c4f91020d07c5df4cc15f4e2b88f
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:34 2016 -0700

    Revert "gcov: use within_module() helper."

    This reverts commit 581d096d9988adf67ae4b9dd881bfa24aa20a0b6.

commit 42a1b19b7e24d6c2375de5bdf200f5240e556e55
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:24 2016 -0700

    Revert "module: use a structure to encapsulate layout."

    This reverts commit 6404731da759dc7c1551e3d45e0006d3be6beacc.

commit 65dae6242fda8769903c681755358e341b9a11ad
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:14 2016 -0700

    Revert "module: clean up RO/NX handling."

    This reverts commit c8d73ebfe19daac81b7cb5c8d1dd700849bad0b3.

commit 5079a2c03405c0c34af597a2340b773b424ed014
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:22:01 2016 -0700

    Revert "module: keep percpu symbols in module's symtab"

    This reverts commit 32d96e18fca1f1181c17d9e636efd64ea79bd9fe.

commit 9206884e139260f102d0490c13b0c290d70538fc
Author: dann frazier <email address hidden>
Date: Fri Feb 19 16:21:55 2016 -0700

    Revert "livepatch: Cleanup module page permission changes"

    This reverts commit 83f79af46d38abd463ec3eb26b6e0a6e5c59336a.

Tim Gardner (timg-tpi) on 2016-02-22
Changed in linux (Ubuntu Xenial):
assignee: nobody → Chris J Arges (arges)
Ming Lei (tom-leiming) wrote :
Download full text (7.0 KiB)

Looks it is enough to just revert
'e96e20134729121689a0089537c6ed(module: clean up RO/NX handling)'
for fixing the issue.

But the interesting thing is that there isn't the problem in upstream kernel
4.5-rc5, and the commit(module: clean up RO/NX handling) isn't reverted
in upstream yet.

So looks there is one real fix in upstream kernel.

Thanks,

On Mon, Feb 22, 2016 at 12:07 PM, Tim Gardner <email address hidden> wrote:
> ** Changed in: linux (Ubuntu Xenial)
> Assignee: (unassigned) => Chris J Arges (arges)
>
> --
> You received this bug notification because you are subscribed to linux
> in Ubuntu.
> https://bugs.launchpad.net/bugs/1547718
>
> Title:
> 4.4.0-7.22 no longer boots on arm64
>
> Status in linux package in Ubuntu:
> Confirmed
> Status in linux source package in Xenial:
> Confirmed
>
> Bug description:
> Observed on X-Gene and ThunderX.
>
> [ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
> [ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
> [ 2.071185] Modules linked in:
> [ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
> [ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
> [ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
> [ 2.094840] PC is at apply_to_page_range+0x330/0x378
> [ 2.099780] LR is at change_memory_common+0x94/0x148
> [ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
> [ 2.112073] sp : ffff8001df4a7b30
> [ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
> [ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
> [ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
> [ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
> [ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
> [ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
> [ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
> [ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
> [ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
> [ 2.163091] x11: 0000000000000018 x10: 0101010101010101
> [ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
> [ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
> [ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
> [ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
> [ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
> [ 2.194902]
> [ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
> [ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
> [ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
> [ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d56000 0008000000000000
> [ 2.224862] 7b60: 0000000000000080 ffff7ffffc009500 ffff80000009aa18 ffff7ffffc009550
> [ 2.232651] 7b80: ffff8001df4a7df0 0000000000000000 ffff800000f11800 ffff7ffffc00efff
> [ 2.240440] 7ba0: ffff7ffffc00f000 ffff7ffffc00f000 ffff7ffffc00f000 ffff800000000000
> [ 2.248227] 7bc0: ffff7ffffc00efff 0000000000000000...

Read more...

Ming Lei (tom-leiming) wrote :
Download full text (7.4 KiB)

On Mon, Feb 22, 2016 at 4:37 PM, Ming Lei <email address hidden> wrote:
> Looks it is enough to just revert
> 'e96e20134729121689a0089537c6ed(module: clean up RO/NX handling)'
> for fixing the issue.
>
> But the interesting thing is that there isn't the problem in upstream kernel
> 4.5-rc5, and the commit(module: clean up RO/NX handling) isn't reverted
> in upstream yet.
>
> So looks there is one real fix in upstream kernel.

The issue was fixed by 57adec866c0440976c96a4b8f5b59f
(arm64: mm: avoid calling apply_to_page_range on empty range), and
this patch is marked as -stalbe.

I have sent this patch outto uk list already.

Thanks,

>
> Thanks,
>
> On Mon, Feb 22, 2016 at 12:07 PM, Tim Gardner <email address hidden> wrote:
>> ** Changed in: linux (Ubuntu Xenial)
>> Assignee: (unassigned) => Chris J Arges (arges)
>>
>> --
>> You received this bug notification because you are subscribed to linux
>> in Ubuntu.
>> https://bugs.launchpad.net/bugs/1547718
>>
>> Title:
>> 4.4.0-7.22 no longer boots on arm64
>>
>> Status in linux package in Ubuntu:
>> Confirmed
>> Status in linux source package in Xenial:
>> Confirmed
>>
>> Bug description:
>> Observed on X-Gene and ThunderX.
>>
>> [ 2.059489] kernel BUG at /build/linux-RKt9qy/linux-4.4.0/mm/memory.c:1887!
>> [ 2.066418] Internal error: Oops - BUG: 0 [#1] SMP
>> [ 2.071185] Modules linked in:
>> [ 2.074234] CPU: 1 PID: 156 Comm: systemd-udevd Not tainted 4.4.0-7-generic #22-Ubuntu
>> [ 2.082109] Hardware name: APM X-Gene Mustang board (DT)
>> [ 2.087393] task: ffff8001f679d780 ti: ffff8001df4a4000 task.ti: ffff8001df4a4000
>> [ 2.094840] PC is at apply_to_page_range+0x330/0x378
>> [ 2.099780] LR is at change_memory_common+0x94/0x148
>> [ 2.104717] pc : [<ffff8000001f8f20>] lr : [<ffff80000009a964>] pstate: 60000145
>> [ 2.112073] sp : ffff8001df4a7b30
>> [ 2.115369] x29: ffff8001df4a7b30 x28: 0000000000000000
>> [ 2.120671] x27: ffff7ffffc00f000 x26: ffff7ffffc009550
>> [ 2.125974] x25: ffff80000009aa18 x24: ffff7ffffc009500
>> [ 2.131276] x23: ffff8001df4a7c38 x22: ffff80000009a878
>> [ 2.136579] x21: ffff800000d56000 x20: ffff7ffffc00f000
>> [ 2.141881] x19: ffff7ffffc00f000 x18: ffff80000090f928
>> [ 2.147183] x17: ffff8001df4a79e8 x16: b4cc9cbc209fc557
>> [ 2.152485] x15: ffffffffffffffff x14: ffff000000000000
>> [ 2.157788] x13: ffffffffffffffff x12: 0000000000000010
>> [ 2.163091] x11: 0000000000000018 x10: 0101010101010101
>> [ 2.168393] x9 : ffffffffffffffff x8 : 7f7f7f7f7f7f7f7f
>> [ 2.173694] x7 : 636e712cff646b61 x6 : 000000802e726f64
>> [ 2.178996] x5 : 646f722e00000000 x4 : ffff8001df4a7c38
>> [ 2.184298] x3 : ffff80000009a878 x2 : 0000000000000000
>> [ 2.189601] x1 : ffff7ffffc00f000 x0 : ffff80000009a964
>> [ 2.194902]
>> [ 2.196384] Process systemd-udevd (pid: 156, stack limit = 0xffff8001df4a4020)
>> [ 2.203568] Stack: (0xffff8001df4a7b30 to 0xffff8001df4a8000)
>> [ 2.209283] 7b20: ffff8001df4a7bf0 ffff80000009a964
>> [ 2.217073] 7b40: ffff7ffffc00f000 ffff7ffffc00f000 ffff800000d560...

Read more...

Paolo Pisati (p-pisati) wrote :

It affects armhf too, i'm sending a SAUCE PATCH containing the arm side of the fix (identical to the arm64 BTW).

tags: added: patch
tags: added: kernel-da-key
Launchpad Janitor (janitor) wrote :
Download full text (10.1 KiB)

This bug was fixed in the package linux - 4.4.0-8.23

---------------
linux (4.4.0-8.23) xenial; urgency=low

  * cgroup namespace mounts broken in containers (LP: #1549398)
    - SAUCE: kernfs: Always set super block owner to init_user_ns

  * 4.4.0-7.22 no longer boots on arm64 (LP: #1547718)
    - arm64: mm: avoid calling apply_to_page_range on empty range
    - UBUNTU SAUCE: arm: mm: avoid calling apply_to_page_range on empty range

  * kernel install failed /bin/cp: cannot stat ‘/boot/initrd.img-4.3.0-7-generic’: No such file or directory (LP: #1536810)
    - [Config] postinst -- handle recreating symlinks when a real file is present

  * insecure overlayfs xattrs handling in copy_up (LP: #1534961)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs
    - SAUCE: overlayfs: Be more careful about copying up sxid files
    - SAUCE: overlayfs: Propogate nosuid from lower and upper mounts

  * overlayfs over fuse should refuse copy_up of files if uid/gid not mapped (LP: #1535150)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs
    - SAUCE: overlayfs: Be more careful about copying up sxid files
    - SAUCE: overlayfs: Propogate nosuid from lower and upper mounts

  * overlay: mkdir fails if directory exists in lowerdir in a user namespace (LP: #1531747)
    - SAUCE: cred: Add clone_cred() interface
    - SAUCE: overlayfs: Use mounter's credentials instead of selectively raising caps
    - SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.* xattrs

  * Update Intel ethernet drivers to Fortville SW5 (LP: #1547674)
    - net: bulk free infrastructure for NAPI context, use napi_consume_skb
    - net: Add eth_platform_get_mac_address() helper.
    - i40e: Add mac_filter_element at the end of the list instead of HEAD
    - i40e/i40evf: Fix RSS rx-flow-hash configuration through ethtool
    - i40e: Replace X722 mac check in ethtool get_settings
    - i40evf: allow channel bonding of VFs
    - i40e: define function capabilities in only one place
    - i40evf: null out ring pointers on free
    - i40e: Cleanup the code with respect to restarting autoneg
    - i40e: update features with right offload
    - i40e: bump version to 1.4.10
    - i40e: add new device IDs for X722
    - i40e: Extend ethtool RSS hooks for X722
    - i40e/i40evf: Fix for UDP/TCP RSS for X722
    - i40evf: add new write-back mode
    - i40e/i40evf: Use private workqueue
    - i40e: add new proxy-wol bit for X722
    - i40e: Limit DCB FW version checks to X710/XL710 devices
    - i40e: AQ Add Run PHY Activity struct
    - i40e: AQ Geneve cloud tunnel type
    - i40e: AQ Add external power class to get link status
    - i40e: add 100Mb ethtool reporting
    - ixgbe: bulk free SKBs during TX completion cleanup cycle
    - igb: Remove unnecessary flag setting in igb_set_flag_queue_pairs()
    - igb: Unpair the queues when changing the number of queues...

Changed in linux (Ubuntu Xenial):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers