Mdadm crash on raid5 reshape
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
mdadm (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Ubuntu 12.04 LTS
mdadm 3.2.3-2ubuntu1 0
I try to grow mdadm raid5 array with changing device number and chunk size on same time:
mdadm --grow /dev/md2 --raid-devices=4 --chunk=512 --backup-
After this /proc/mdstat don't respond and I can't gen any info about raid status
May 17 23:27:17 protoss kernel: [40835.240145] RAID conf printout:
May 17 23:27:17 protoss kernel: [40835.240153] --- level:5 rd:4 wd:4
May 17 23:27:17 protoss kernel: [40835.240160] disk 0, o:1, dev:sdc1
May 17 23:27:17 protoss kernel: [40835.240166] disk 1, o:1, dev:sdd1
May 17 23:27:17 protoss kernel: [40835.240171] disk 2, o:1, dev:sdf1
May 17 23:27:17 protoss kernel: [40835.240176] disk 3, o:1, dev:sdg1
May 17 23:27:17 protoss kernel: [40835.240332] md: reshape of RAID array md2
May 17 23:27:17 protoss kernel: [40835.240342] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
May 17 23:27:17 protoss kernel: [40835.240348] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
May 17 23:27:17 protoss kernel: [40835.240365] md: using 128k window, over a total of 1953513472k.
May 17 23:27:17 protoss kernel: [40835.507761] md: md_do_sync() got signal ... exiting
May 17 23:30:42 protoss kernel: [41040.652048] INFO: task md2_raid5:259 blocked for more than 120 seconds.
May 17 23:30:42 protoss kernel: [41040.652055] "echo 0 > /proc/sys/
May 17 23:30:42 protoss kernel: [41040.652062] md2_raid5 D ffffffff81806240 0 259 2 0x00000000
May 17 23:30:42 protoss kernel: [41040.652074] ffff8801336a1c40 0000000000000046 ffffea0001d7f180 ffffffff7fffffff
May 17 23:30:42 protoss kernel: [41040.652086] ffff8801336a1fd8 ffff8801336a1fd8 ffff8801336a1fd8 0000000000013780
May 17 23:30:42 protoss kernel: [41040.652097] ffff880138b796f0 ffff880133ba2de0 ffff8801336a1c30 ffff880133ba2de0
May 17 23:30:42 protoss kernel: [41040.652108] Call Trace:
May 17 23:30:42 protoss kernel: [41040.652123] [<ffffffff8165a
May 17 23:30:42 protoss kernel: [41040.652157] [<ffffffffa00a3
May 17 23:30:42 protoss kernel: [41040.652167] [<ffffffff81056
May 17 23:30:42 protoss kernel: [41040.652176] [<ffffffff8105f
May 17 23:30:42 protoss kernel: [41040.652192] [<ffffffffa00a3
May 17 23:38:42 protoss kernel: [41520.652310] [<ffffffff8108a
May 17 23:38:42 protoss kernel: [41520.652319] [<ffffffff81666
May 17 23:38:42 protoss kernel: [41520.652357] INFO: task lvm:25938 blocked for more than 120 seconds.
May 17 23:38:42 protoss kernel: [41520.652362] "echo 0 > /proc/sys/
May 17 23:38:42 protoss kernel: [41520.652367] lvm D ffffffff81806240 0 25938 25935 0x00000000
May 17 23:38:42 protoss kernel: [41520.652377] ffff88006c6b1ae8 0000000000000086 0000000000000000 ffff88006c6b1aa8
May 17 23:38:42 protoss kernel: [41520.652388] ffff88006c6b1fd8 ffff88006c6b1fd8 ffff88006c6b1fd8 0000000000013780
May 17 23:38:42 protoss kernel: [41520.652398] ffff880138b616f0 ffff88010c02dbc0 ffff88006c6b1ab8 ffff88013fc94040
May 17 23:38:42 protoss kernel: [41520.652409] Call Trace:
May 17 23:38:42 protoss kernel: [41520.652417] [<ffffffff8165a
May 17 23:38:42 protoss kernel: [41520.652424] [<ffffffff8165a
May 17 23:38:42 protoss kernel: [41520.652434] [<ffffffff811b0
May 17 23:38:42 protoss kernel: [41520.652442] [<ffffffff811b3
May 17 23:38:42 protoss kernel: [41520.652451] [<ffffffff811af
May 17 23:38:42 protoss kernel: [41520.652461] [<ffffffff811af
May 17 23:38:42 protoss kernel: [41520.652468] [<ffffffff811af
May 17 23:38:42 protoss kernel: [41520.652478] [<ffffffff81119
May 17 23:38:42 protoss kernel: [41520.652488] [<ffffffff81187
May 17 23:38:42 protoss kernel: [41520.652496] [<ffffffff81177
May 17 23:38:42 protoss kernel: [41520.652506] [<ffffffff8129c
May 17 23:38:42 protoss kernel: [41520.652514] [<ffffffff81177
May 17 23:38:42 protoss kernel: [41520.652521] [<ffffffff81177
May 17 23:38:42 protoss kernel: [41520.652529] [<ffffffff81177
May 17 23:38:42 protoss kernel: [41520.652537] [<ffffffff81664
I've seen this bug too - I was also doing a reshape of a RAID5 array (but without changing chunk size - just adding a new drive).
I'm also seeing this intermittently only. One thing that might be relevant is that I'm reshaping several arrays in succession through a script (doing them in series, but with no waiting between reshapes). I've attached my dmesg output - similar error and stack to the above.
Let me know if there are any other diags which would help here, I can hopefully repro this without too much trouble.