btrfs_direct_write hung during copy using dd

[sushma Gane]

Hi,

I have btrfs filesystem with 2 subvolumes in it. I have copied 48G data to first subvolume and then attached 99G disk to it. Then I started copy 48G to second volume using below dd command[1]. After 3G data copied to second volume dd got stuck with below stack trace[2].

1. dd if=./test1 of=/lucidity_tmp1/test1 bs=32M oflag=direct iflag=direct conv=sync

2. mount point informationubuntu@ip-172-31-8-216:~$ sudo mount | grep btrfs
ubuntu@ip-172-31-8-216:~$ sudo mount | grep btrfs
/dev/nvme10n1 on /lucidity_diskpool type btrfs (rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache=v2,subvolid=5,subvol=/)
/dev/nvme10n1 on /home/test_mnt type btrfs (rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache=v2,subvolid=256,subvol=/lucidity_subvol_15520)
/dev/nvme10n1 on /lucidity_tmp1 type btrfs (rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache=v2,subvolid=257,subvol=/lucidity_subvol_15521)

3. Stack trace[ 573.977394] Spurious interrupt (vector 0xef) on CPU#0. Acked
[ 573.978949] nvme nvme13: 2/0/0 default/read/poll queues
[ 595.087826] BTRFS info (device nvme10n1): disk added /dev/nvme13n1
[ 968.457121] INFO: task dd:10718 blocked for more than 120 seconds.
[ 968.459372] Not tainted 5.19.0-1025-aws #26~22.04.1-Ubuntu
[ 968.461557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 968.464327] task:dd state:D stack: 0 pid:10718 ppid: 10717 flags:0x00004002
[ 968.464332] Call Trace:
[ 968.464334] <TASK>
[ 968.464337] __schedule+0x254/0x5a0
[ 968.464343] schedule+0x5d/0x100
[ 968.464346] io_schedule+0x46/0x80
[ 968.464348] blk_mq_get_tag+0x117/0x300
[ 968.464353] ? destroy_sched_domains_rcu+0x40/0x40
[ 968.464359] __blk_mq_alloc_requests+0xc4/0x1e0
[ 968.464362] blk_mq_get_new_requests+0xcc/0x190
[ 968.464366] blk_mq_submit_bio+0x1eb/0x450
[ 968.464368] __submit_bio+0xf6/0x190
[ 968.464370] submit_bio_noacct_nocheck+0xc2/0x120
[ 968.464372] submit_bio_noacct+0x209/0x560
[ 968.464374] submit_bio+0x40/0xf0
[ 968.464378] btrfs_map_bio+0x17b/0x480 [btrfs]
[ 968.464432] ? __bio_clone+0x5d/0xd0
[ 968.464436] btrfs_submit_direct+0x186/0x620 [btrfs]
[ 968.464467] ? iov_iter_get_pages+0xd4/0x2e0
[ 968.464470] iomap_dio_submit_bio+0x65/0xa0
[ 968.464473] iomap_dio_bio_iter+0x384/0x490
[ 968.464475] __iomap_dio_rw+0x3b1/0x6f0
[ 968.464479] btrfs_dio_write+0x3a/0x70 [btrfs]
[ 968.464511] btrfs_direct_write+0x120/0x350 [btrfs]
[ 968.464545] btrfs_do_write_iter+0xf1/0x230 [btrfs]
[ 968.464576] ? rseq_ip_fixup+0x72/0x180
[ 968.464580] btrfs_file_write_iter+0x10/0x20 [btrfs]
[ 968.464612] new_sync_write+0xfe/0x190
[ 968.464615] vfs_write+0x1eb/0x270
[ 968.464617] ksys_write+0x67/0xf0
[ 968.464619] __x64_sys_write+0x19/0x30
[ 968.464621] do_syscall_64+0x5c/0x90
[ 968.464624] ? do_syscall_64+0x69/0x90
[ 968.464625] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 968.464629] RIP: 0033:0x7fbeb5f14a37
[ 968.464631] RSP: 002b:00007ffedc627878 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 968.464633] RAX: ffffffffffffffda RBX: 0000000002000000 RCX: 00007fbeb5f14a37
[ 968.464635] RDX: 0000000002000000 RSI: 00007fbeb39fe000 RDI: 0000000000000001
[ 968.464636] RBP: 00007fbeb39fe000 R08: 0000000000000400 R09: 00007fbeb39fe000
[ 968.464637] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000000000
[ 968.464639] R13: 0000000000000400 R14: ffffffffffffffff R15: 00007fbeb39fe000
[ 968.464641] </TASK>

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu

ubuntu@ip-172-31-8-216:~$ lsb_release -rd
Description: Ubuntu 22.04.2 LTS
Release: 22.04

2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software
 Center
N: Unable to locate package pkgname

3) What you expected to happen

dd command should copy data to subvolume

4) What happened instead

It hung

Thank You
Sushma

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.19.0-1025-aws 5.19.0-1025.26~22.04.1
ProcVersionSignature: User Name 5.19.0-1025.26~22.04.1-aws 5.19.17
Uname: Linux 5.19.0-1025-aws x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Fri Jul 14 10:35:02 2023
Ec2AMI: ami-0395e642c96ea66a1
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-south-1b
Ec2InstanceType: t3.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: linux-signed-aws-5.19
UpgradeStatus: No upgrade log present (probably fresh install)