zfs recv unknown error 1041

[Simon Déziel]

[Description]

In my setup, 'xeon' pushes and pulls multiple ZFS to/from another system called 'jupiter'. On xeon, I did a full zpool transfer from a single disk to a newer one so that all the data and snapshots are on the newer disk. Since the disk change on xeon, it can still pull from jupiter all the ZFS that were replicated in that direction but the other way around (xeon pushing to jupiter) errors out on jupiter trying to receive them:

root@xeon:~# zfs snapshot data/aude/photos@new
root@xeon:~# zfs send -Lce -v -i data/aude/photos@manual-sync data/aude/photos@new > /tmp/foo
send from @manual-sync to data/aude/photos@new estimated size is 624B
total estimated size is 624B
TIME SENT SNAPSHOT data/aude/photos@new

# use scp/rsync/whatever to transfer /tmp/foo from xeon->jupiter
# use md5sum /tmp/foo on xeon and jupiter to ensure good transfer

root@jupiter:~# zfs recv -uv data/aude/photos < /tmp/foo
receiving incremental stream of data/aude/photos@new into data/aude/photos@new
internal error: Unknown error 1041
Aborted

[What was tried]

1) reversing the sync direction (xeon pulling from jupiter) => works (but isn't what I want)
2) moving from 5.13 kernel (HWE) to 5.15 (HWE edge) on both sides => no change
3) on xeon, touch a file to have some actual data to move around => no change
4) try various combination of `zfs send` params (even just `-vi`) => no change
5) `zpool scrub` on both sides to ensure all data is clean => no error found

[Additional information]

root@xeon:~# uname -a
Linux xeon 5.15.0-41-generic #44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@xeon:~# apt-cache policy zfsutils-linux
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.14
  Candidate: 0.8.3-1ubuntu12.14
  Version table:
 *** 0.8.3-1ubuntu12.14 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12.9 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     0.8.3-1ubuntu12 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

root@jupiter:~# uname -a
Linux jupiter 5.15.0-41-generic #44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@jupiter:~# apt-cache policy zfsutils-linux
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.14
  Candidate: 0.8.3-1ubuntu12.14
  Version table:
 *** 0.8.3-1ubuntu12.14 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12.9 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     0.8.3-1ubuntu12 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Simon Déziel (sdeziel)]

Sure enough, putting the old disk back in xeon makes the problem go away.