xfs freeze every week on multiple machines
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
We run multiple machines that have the purpose of acting as a data repository for Veeam backup. Each machine has 2x 256G XFS volumes that use heavy reflinking. This works as intended, for one issue: our xfs freeze once a week, a few minutes after midnight on Monday nights. We can only do a reboot to get the servers working again. Then it works for a week again.
We have been interacting with support from Veeam, but this looks like some XFS kernel issue, or a race condition between the Veeam process and XFS.
[562599.834734] INFO: task kworker/6:3:3534660 blocked for more than 120 seconds.
[562599.834794] Not tainted 5.4.0-124-generic #140-Ubuntu
[562599.834832] "echo 0 > /proc/sys/
[562599.834891] kworker/6:3 D 0 3534660 2 0x80004000
[562599.834962] Workqueue: xfs-conv/dm-3 xfs_end_io [xfs]
[562599.834964] Call Trace:
[562599.834975] __schedule+
[562599.835026] ? xfs_log_
[562599.835031] ? kmem_cache_
[562599.835035] schedule+0x42/0xb0
[562599.835041] rwsem_down_
[562599.835045] ? __switch_
[562599.835088] ? __xfs_setfilesi
[562599.835091] down_write+
[562599.835137] xfs_ilock+
[562599.835178] __xfs_setfilesi
[562599.835181] ? __switch_
[562599.835220] xfs_setfilesize
[562599.835257] xfs_end_
[562599.835260] ? __switch_
[562599.835298] xfs_end_
[562599.835304] process_
[562599.835309] worker_
[562599.835312] kthread+0x104/0x140
[562599.835316] ? process_
[562599.835319] ? kthread_
[562599.835322] ret_from_
[562599.836171] INFO: task veeamagent:3674754 blocked for more than 120 seconds.
[562599.836219] Not tainted 5.4.0-124-generic #140-Ubuntu
[562599.836261] "echo 0 > /proc/sys/
[562599.836318] veeamagent D 0 3674754 3674651 0x00004000
[562599.836321] Call Trace:
[562599.836326] __schedule+
[562599.836330] schedule+0x42/0xb0
[562599.836333] schedule_
[562599.836335] ? schedule_
[562599.836337] __down+0x82/0xd0
[562599.836341] ? wake_up_q+0x70/0x70
[562599.836383] ? xfs_buf_
[562599.836391] down+0x47/0x60
[562599.836434] xfs_buf_
[562599.836476] xfs_buf_
[562599.836518] xfs_buf_
[562599.836557] xfs_buf_
[562599.836610] xfs_trans_
[562599.836643] xfs_read_
[562599.836664] ? update_
[562599.836700] xfs_alloc_
[562599.836753] ? xfs_alloc_
[562599.836783] xfs_alloc_
[562599.836786] ? check_preempt_
[562599.836788] ? ttwu_do_
[562599.836793] ? radix_tree_
[562599.836836] ? xfs_perag_
[562599.836839] ? radix_tree_
[562599.836877] ? xfs_perag_
[562599.836906] xfs_alloc_
[562599.836938] xfs_bmap_
[562599.836973] xfs_bmap_
[562599.837004] xfs_bmapi_
[562599.837043] xfs_bmapi_
[562599.837084] xfs_map_
[562599.837123] xfs_do_
[562599.837130] write_cache_
[562599.837171] ? xfs_vm_
[562599.837209] xfs_vm_
[562599.837215] do_writepages+
[562599.837221] __filemap_
[562599.837226] file_write_
[562599.837268] xfs_file_
[562599.837274] ? __do_sys_
[562599.837281] vfs_fsync_
[562599.837284] do_fsync+0x3d/0x70
[562599.837288] __x64_sys_
[562599.837295] do_syscall_
[562599.837298] entry_SYSCALL_
[562599.837301] RIP: 0033:0x7f4092abd93b
[562599.837308] Code: Bad RIP value.
[562599.837310] RSP: 002b:00007f4066
[562599.837313] RAX: ffffffffffffffda RBX: 00007f4066ffd650 RCX: 00007f4092abd93b
[562599.837315] RDX: 00007f4066ffc800 RSI: 00007f4066ffc800 RDI: 0000000000000b06
[562599.837316] RBP: 00007f40440329e0 R08: 0000000000000000 R09: 00007f40580008d0
[562599.837317] R10: 00007f40580008d0 R11: 0000000000000293 R12: 00007f4044029940
[562599.837319] R13: 0000000000000b06 R14: 0000000000000000 R15: 00007f4044017b70
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-
ProcVersionSign
Uname: Linux 5.4.0-124-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 11:51 seq
crw-rw---- 1 root audio 116, 33 Aug 22 11:51 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckR
Date: Mon Aug 29 00:34:28 2022
InstallationDate: Installed on 2021-02-10 (564 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Dell Inc. PowerEdge M630
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.187.33
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/05/2022
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.15.0
dmi.board.name: 0R10KJ
dmi.board.vendor: Dell Inc.
dmi.board.version: A02
dmi.chassis.type: 25
dmi.chassis.vendor: Dell Inc.
dmi.chassis.
dmi.modalias: dmi:bvnDellInc.
dmi.product.name: PowerEdge M630
dmi.product.sku: SKU=NotProvided
dmi.sys.vendor: Dell Inc.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 11:51 seq
crw-rw---- 1 root audio 116, 33 Aug 22 11:51 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckR
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2021-09-16 (346 days ago)
InstallationMedia: Ubuntu-Server 20.04.3 LTS "Focal Fossa" - Release amd64 (20210824)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Dell Inc. PowerEdge M630
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.187.33
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.4.0-124-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 07/05/2022
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.15.0
dmi.board.name: 0R10KJ
dmi.board.vendor: Dell Inc.
dmi.board.version: A05
dmi.chassis.type: 25
dmi.chassis.vendor: Dell Inc.
dmi.chassis.
dmi.modalias: dmi:bvnDellInc.
dmi.product.name: PowerEdge M630
dmi.product.sku: SKU=NotProvided
dmi.sys.vendor: Dell Inc.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 22 11:51 seq
crw-rw---- 1 root audio 116, 33 Aug 22 11:51 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckR
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2021-09-16 (346 days ago)
InstallationMedia: Ubuntu-Server 20.04.3 LTS "Focal Fossa" - Release amd64 (20210824)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Dell Inc. PowerEdge M630
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.187.33
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.4.0-124-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 07/05/2022
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.15.0
dmi.board.name: 0R10KJ
dmi.board.vendor: Dell Inc.
dmi.board.version: A05
dmi.chassis.type: 25
dmi.chassis.vendor: Dell Inc.
dmi.chassis.
dmi.modalias: dmi:bvnDellInc.
dmi.product.name: PowerEdge M630
dmi.product.sku: SKU=NotProvided
dmi.sys.vendor: Dell Inc.
summary: |
- xfs freeze + xfs freeze every week on multiple machines |
apport information