Performance: excessive paging during mdadm resync

Bug #2015396 reported by Simon Lambourn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mdadm (Ubuntu)
New
Undecided
Unassigned

Bug Description

Periodically, RAID arrays are re-synced even when no faults have been reported.
This naturally causes a large amount of I/O but this I/O seems to use large amounts of buffer space and comes to dominate system memory. This causes excessive swapping and unresponsiveness in other applications and in some cases has caused applications to be terminated, presumably due to severe memory shortage. This can have a serious user impact making the system unusable for many hours while the re-sync completes, especially on large RAID arrays.

I have tried using ionice and renice to reduce the priority of the resync tasks: the problem is somewhat reduced but sometimes the system is still unusable and applications are terminated without warning.

This seems unreasonable since the re-sync is doing a serial read of the whole disk, and being predictable should not need large amounts of bufferspace.

Desired behaviour: bufferspace for resync should be restricted to a reasonable limit, permitting resync to happen in a timely manner without stealing the majority of system memory.

ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: mdadm 4.2-3ubuntu1
ProcVersionSignature: Ubuntu 5.19.0-38.39-generic 5.19.17
Uname: Linux 5.19.0-38-generic x86_64
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Wed Apr 5 20:12:14 2023
Grub.cfg: Error: [Errno 13] Permission denied: '/boot/grub/grub.cfg'
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sda1: Error: command ['/sbin/mdadm', '-E', '/dev/sda1'] failed with exit code 1: mdadm: cannot open /dev/sda1: Permission denied
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: cannot open /dev/sda2: Permission denied
MDadmExamine.dev.sda3: Error: command ['/sbin/mdadm', '-E', '/dev/sda3'] failed with exit code 1: mdadm: cannot open /dev/sda3: Permission denied
MDadmExamine.dev.sda4: Error: command ['/sbin/mdadm', '-E', '/dev/sda4'] failed with exit code 1: mdadm: cannot open /dev/sda4: Permission denied
MDadmExamine.dev.sda5: Error: command ['/sbin/mdadm', '-E', '/dev/sda5'] failed with exit code 1: mdadm: cannot open /dev/sda5: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdb1: Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: cannot open /dev/sdb1: Permission denied
MDadmExamine.dev.sdb2: Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: cannot open /dev/sdb2: Permission denied
MDadmExamine.dev.sdb3: Error: command ['/sbin/mdadm', '-E', '/dev/sdb3'] failed with exit code 1: mdadm: cannot open /dev/sdb3: Permission denied
MDadmExamine.dev.sdb4: Error: command ['/sbin/mdadm', '-E', '/dev/sdb4'] failed with exit code 1: mdadm: cannot open /dev/sdb4: Permission denied
MDadmExamine.dev.sdb5: Error: command ['/sbin/mdadm', '-E', '/dev/sdb5'] failed with exit code 1: mdadm: cannot open /dev/sdb5: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MDadmExamine.dev.sdc1: Error: command ['/sbin/mdadm', '-E', '/dev/sdc1'] failed with exit code 1: mdadm: cannot open /dev/sdc1: Permission denied
MDadmExamine.dev.sdc2: Error: command ['/sbin/mdadm', '-E', '/dev/sdc2'] failed with exit code 1: mdadm: cannot open /dev/sdc2: Permission denied
MDadmExamine.dev.sdc3: Error: command ['/sbin/mdadm', '-E', '/dev/sdc3'] failed with exit code 1: mdadm: cannot open /dev/sdc3: Permission denied
MDadmExamine.dev.sdc4: Error: command ['/sbin/mdadm', '-E', '/dev/sdc4'] failed with exit code 1: mdadm: cannot open /dev/sdc4: Permission denied
MDadmExamine.dev.sdc5: Error: command ['/sbin/mdadm', '-E', '/dev/sdc5'] failed with exit code 1: mdadm: cannot open /dev/sdc5: Permission denied
MDadmExamine.dev.sdd: Error: command ['/sbin/mdadm', '-E', '/dev/sdd'] failed with exit code 1: mdadm: cannot open /dev/sdd: Permission denied
MDadmExamine.dev.sdd1: Error: command ['/sbin/mdadm', '-E', '/dev/sdd1'] failed with exit code 1: mdadm: cannot open /dev/sdd1: Permission denied
MDadmExamine.dev.sdd2: Error: command ['/sbin/mdadm', '-E', '/dev/sdd2'] failed with exit code 1: mdadm: cannot open /dev/sdd2: Permission denied
MDadmExamine.dev.sdd3: Error: command ['/sbin/mdadm', '-E', '/dev/sdd3'] failed with exit code 1: mdadm: cannot open /dev/sdd3: Permission denied
MDadmExamine.dev.sdd4: Error: command ['/sbin/mdadm', '-E', '/dev/sdd4'] failed with exit code 1: mdadm: cannot open /dev/sdd4: Permission denied
MDadmExamine.dev.sdd5: Error: command ['/sbin/mdadm', '-E', '/dev/sdd5'] failed with exit code 1: mdadm: cannot open /dev/sdd5: Permission denied
MachineType: System manufacturer System Product Name
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=/dev/mapper/vg1-rootfs ro raid=noautodetect
ProcMounts: Error: [Errno 40] Too many levels of symbolic links: '/proc/mounts'
SourcePackage: mdadm
UpgradeStatus: Upgraded to kinetic on 2022-11-03 (153 days ago)
dmi.bios.date: 09/10/2010
dmi.bios.release: 8.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1801
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: M3N78-VM
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1801:bd09/10/2010:br8.14:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnM3N78-VM:rvrRevX.0x:cvnChassisManufacture:ct3:cvrChassisVersion:skuToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: System Product Name
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'

Revision history for this message
Simon Lambourn (simon-lambourn-o) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.