artful 4.13 i386 kernels crash after memory hotplug remove

Bug #1747069 reported by Andy Whitcroft on 2018-02-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Colin Ian King

Bug Description

== SRU Request, Artful ==

Hotplug removal causes i386 crashes when exercised with the kernel
selftest mem-on-off-test script.

== Fix ==

Revert commit f1dd2cd13c4b (""mm, memory_hotplug: do not associate hotadded memory to zones until online")

Note: A fix occurs in 4.15 however this requires a large set of changes that are way too large to be SRU'able and the least risky way forward is to revert the offending commit.

== Testcase ==

Running the kernel selftest script mem-on-off-test.sh, followed by a sync, followed by re-installing kernel packages will always trigger this issue. Simply running the mem-on-off-test.sh script sometimes won't trigger the problem. I believe this is why we've not seen this happen too frequently with our ADT tests. I can reproduce this in a VM with 4 CPUs and 2GB of memory.

== Regression Potential ==

Reverting this commit does remove some functionality, however this does not regress the kernel compared to previous releases and having a working reliable memory hotplug is the preferred option. This fix does touch some memory hotplug, so there is a risk that this may break this functionality that is not covered by the kernel regression testing.

Andy Whitcroft (apw) on 2018-02-02
Changed in linux (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
importance: Undecided → High
status: New → Confirmed
Changed in linux (Ubuntu):
status: Confirmed → In Progress
Colin Ian King (colin-king) wrote :

Bisected, bad commit: b3c6858fb172512f63838523ae7817ae8adec564 - this is a merge and contains a lot of misc changes across the tree that may have broken this.

Colin Ian King (colin-king) wrote :

Fixed in 4.14 with:

f7f99100d8d95dbcf09e0216a143211e79418b9f is the first bad commit
commit f7f99100d8d95dbcf09e0216a143211e79418b9f
Author: Pavel Tatashin <email address hidden>
Date: Wed Nov 15 17:36:44 2017 -0800

    mm: stop zeroing memory during allocation in vmemmap

Colin Ian King (colin-king) wrote :

Requires backports of commits:

a86d69d58aad561b6bbb44e60f74c41cd4b5f3ab
ed067d4a859ff696373324c5061392e013a7561a
f7f99100d8d95dbcf09e0216a143211e79418b9f
a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b
ea1f5f3712afe895dfa4176ec87376b4a9ac23be

Colin Ian King (colin-king) wrote :
Download full text (7.3 KiB)

Ignore my above comments. I bisected this again using a more reliable reproducer and found the first bad commit to be:
f1dd2cd13c4bbbc9a7c4617b3b034fa643de98fe is the first bad commit
commit f1dd2cd13c4bbbc9a7c4617b3b034fa643de98fe
Author: Michal Hocko <email address hidden>
Date: Thu Jul 6 15:38:11 2017 -0700

    mm, memory_hotplug: do not associate hotadded memory to zones until online

    The current memory hotplug implementation relies on having all the
    struct pages associate with a zone/node during the physical hotplug
    phase (arch_add_memory->__add_pages->__add_section->__add_zone). In the
    vast majority of cases this means that they are added to ZONE_NORMAL.
    This has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory
    hotadd without sparsemem") and it wasn't a big deal back then because
    movable onlining didn't exist yet.

    Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
    onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
    memory and portion memory") and then things got more complicated.
    Rather than reconsidering the zone association which was no longer
    needed (because the memory hotplug already depended on SPARSEMEM) a
    convoluted semantic of zone shifting has been developed. Only the
    currently last memblock or the one adjacent to the zone_movable can be
    onlined movable. This essentially means that the online type changes as
    the new memblocks are added.

    Let's simulate memory hot online manually
      $ echo 0x100000000 > /sys/devices/system/memory/probe
      $ grep . /sys/devices/system/memory/memory32/valid_zones
      Normal Movable

      $ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
      $ grep . /sys/devices/system/memory/memory3?/valid_zones
      /sys/devices/system/memory/memory32/valid_zones:Normal
      /sys/devices/system/memory/memory33/valid_zones:Normal Movable

      $ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
      $ grep . /sys/devices/system/memory/memory3?/valid_zones
      /sys/devices/system/memory/memory32/valid_zones:Normal
      /sys/devices/system/memory/memory33/valid_zones:Normal
      /sys/devices/system/memory/memory34/valid_zones:Normal Movable

      $ echo online_movable > /sys/devices/system/memory/memory34/state
      $ grep . /sys/devices/system/memory/memory3?/valid_zones
      /sys/devices/system/memory/memory32/valid_zones:Normal
      /sys/devices/system/memory/memory33/valid_zones:Normal Movable
      /sys/devices/system/memory/memory34/valid_zones:Movable Normal

    This is an awkward semantic because an udev event is sent as soon as the
    block is onlined and an udev handler might want to online it based on
    some policy (e.g. association with a node) but it will inherently race
    with new blocks showing up.

    This patch changes the physical online phase to not associate pages with
    any zone at all. All the pages are just marked reserved and wait for
    the onlining phase to be associated with the zone as per the online
    request. There are only two requirements

            - exist...

Read more...

description: updated

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-artful
Colin Ian King (colin-king) wrote :

Ran the tests against the i386 -proposed kernel, cannot reproduce the issue with the fixed kernel. Also ran the ADT tests and could not reproduce the issue (and these run the memory hotplug tests too).

Tested, and verified.

tags: added: verification-done-artful
removed: verification-needed-artful
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers