[INTEL UBUNTU 18.04] RAID reshape hangs after OS reboot

[Michał Żyłowski]

Steps to reproduce:

1. Create IMSM RAID container:
mdadm --create /dev/md/imsm0 --metadata=imsm --raid-devices=3 /dev/sde /dev/sdc /dev/sdd --run
2. Create IMSM RAID volume:
mdadm --create /dev/md/r0d3s128h1_A --level=0 --chunk 128 --raid-devices=3 /dev/sde /dev/sdc /dev/sdd --run --assume-clean
3. Add disk to container:
mdadm --add /dev/md127 /dev/sdb
4. Invoke RAID grow:
export MDADM_EXPERIMENTAL=1;mdadm --grow /dev/md127 --raid-devices=4
5. By cat /proc/mdstat we will see status of the reshape.
Also systemd unit file has been created:

cat /<email address hidden>
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.

[Unit]
Description=Manage MD Reshape on /dev/%I
DefaultDependencies=no

[Service]
ExecStart=/sbin/mdadm --grow --continue /dev/%I
#StandardInput=null
#StandardOutput=null
#StandardError=null
KillMode=none

And status of this service is:

● <email address hidden> - Manage MD Reshape on /dev/md127
   Loaded: loaded (/<email address hidden>; static; vendor preset: enabled)
   Active: active (running) since Fri 2018-03-23 09:20:54 CET; 23s ago
 Main PID: 3364 (mdadm)
    Tasks: 1 (limit: 37710)
   CGroup: /system.slice/system-mdadm\x2dgrow\<email address hidden>
           └─3364 /sbin/mdadm --grow --continue /dev/md127

6. Reboot the platform.

7. During OS booting I saw print:
[ 16.552076] md: reshape of RAID array md126

8. After successfully boot we can see hanging reshape process (by cat /proc/mdstat).

And status of the service:
systemctl status <email address hidden>
● <email address hidden> - Manage MD Reshape on /dev/md127
   Loaded: loaded (/<email address hidden>; static; vendor preset: enabled)
   Active: inactive (dead)

Possible workarounds:
a) systemctl restart <email address hidden>
or
b) mdadm --grow --continue /dev/md127

So the clue of the problem is in OS booting process and with correctly starting mdadm-grow-continue service.

[Dimitri John Ledkov (xnox)]

This seems to me to be an upstream bug. The grow command starts the systemd unit, but doesn't enable it to be persistent. The md device names can change after reboot (if an additional array is connected whilst the system is off), therefore I think upstream udev rules should be improved to detect arrays that need mdadm-grow-continue@.service, and trigger that unit via SYSTEMD_WANTS stanza in the udev rules.

Could you provide a dump of /sys/devices/virtual/block/md126/md after reboot, when you expect the mdadm-grow-continue to resume? (assuming md126 is the one undergoing reshaping)

I hope that, e.g. /sys/devices/virtual/block/md126/md/sync_action is "reshape" and /sys/devices/virtual/block/md126/md/reshape_position is not "none".

Then it should be trivial to adjust the udev rules, to kick off mdadm-grow-continue@.service, based on those sysfs attributes on boot.

[Pawel Baldysiak (pbaldysi)]

Mdadm is running it's additional instance for reshape purposes.
It might do it in two ways:
running systemd service file (mdadm-grow-continue service), or by forking.
The second one is used as fallback method, executed if first one fails.

Mdadm will run this within "mdadm --incremental" run from udev rules what assembles the volume under reshape.

In ubuntu MD raids are assemble in initramfs and there is no unit file for grow-continue there.
So systemd/udev will run mdadm --incremental that will fork into manage reshape.
Also, there is a mechanism in systemd that kills udev events after some sort of timeout (it treats them as hung tasks).
As a result, the helper for managing reshape is killed, and reshape progress is stuck.

As a patch I propose to add some sort of flag to initramfs, so it will skip assembling the arrays in init before switchroot. It should solve the issue (at least for array w/o OS installed on them - they have to be assembled in init).

Thanks
Pawel

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mariusz Tkaczyk (mtkaczyk)]

Hi,
We have two issues here:
1. mdraid stuff is always included in initrd even system isn't on raid( ubuntu specific).
2. grow continue service is not restarted after switch root( to be fixed in upstream)

First one should be detected and managed in ubuntu installer. This is how it is done in other distributions. It will hide second problem when system is not on raid.
Initial ram disk should be minimal as possible, in this case md stuff is unnecessary.

As Pawel wrote reshape is managed by mdadm-grow-continue@.service or fork which is started by mdadm (if needed) as a last step of raid assembly. If it is started in initrd then it is not restarted after switch root because raid assemblation is already done. No fix for today- work in progress.

Please look into first one. I think that it is not a mdadm defect now, new launchpad is needed?
I will notify this lp when fix for second defect appears.

Thanks
Mariusz

[Mariusz Tkaczyk (mtkaczyk)]

Hello,
fix for second problem has been accepted upstream:
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=a64f1263a56bd8653267c1a9800daa0bc993a743

Mariusz

[Jeff Lane  (bladernr)]

Is this resolved with 4.2-rc2 or earlier?

[Launchpad Janitor (janitor)]

[Expired for mdadm (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for mdadm (Ubuntu Bionic) because there has been no activity for 60 days.]