zsysd failing to start: panic: runtime error: index out of range [40] with length 37

[Michał Sawicz]

This might've been caused by me clearing up a bunch of snapshots manually, when having upgraded to Plucky my system went haywire (bug#2106492).

Still, zsysd should start up, shouldn't it…

ProblemType: Bug
DistroRelease: Ubuntu 25.04
Package: zsys 0.5.11.3build4
ProcVersionSignature: Ubuntu 6.14.0-13.13-generic 6.14.0
Uname: Linux 6.14.0-13-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.32.0-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Tue Apr 8 18:54:26 2025
InstallationDate: Installed on 2024-02-07 (426 days ago)
InstallationMedia: Ubuntu 22.04.3 LTS "Jammy Jellyfish" - Release amd64 (20230807.2)
Mounts: Error: [Errno 40] Za duże zagnieżdżenie dowiązań symbolicznych: '/proc/mounts'
ProcEnviron:
 LANG=pl_PL.UTF-8
 PATH=(custom, no user)
 SHELL=/usr/bin/zsh
 TERM=screen-256color
 XDG_RUNTIME_DIR=<set>
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_itgc0k@/vmlinuz-6.14.0-13-generic root=ZFS=rpool/ROOT/ubuntu_itgc0k ro recovery nomodeset dis_ucode_ldr
RelatedPackageVersions:
 zfs-initramfs 2.3.1-1ubuntu1
 zfsutils-linux 2.3.1-1ubuntu1
SourcePackage: zsys
UpgradeStatus: Upgraded to plucky on 2025-04-08 (0 days ago)
ZFSImportedPools:
 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
 bpool 1.88G 901M 1019M - - 16% 46% 1.00x ONLINE -
 rpool 920G 606G 314G - - 49% 65% 1.00x ONLINE -
ZFSListcache-bpool:
 bpool /boot off on on off on off on off - none - - - - - - - -
 bpool/BOOT none off on on off on off on off - none - - - - - - - -
 bpool/BOOT/ubuntu_itgc0k /boot on on on off on off on off - none - - - - - - - -
ZFSModules:
 zfs 6770688 27 - Live 0x0000000000000000 (PO)
 spl 184320 1 zfs, Live 0x0000000000000000 (O)
ZSYSDump: Error: command ['zsysctl', 'service', 'dump'] failed with exit code 1: level=error msg="couldn't connect to zsys daemon: connection error: desc = \"transport: Error while dialing: dial unix /run/zsysd.sock: connect: connection refused\""

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[StoatWblr (stoatwblr)]

me too

# /sbin/zsysd -v
INFO Building new machines list
panic: runtime error: index out of range [40] with length 37

goroutine 1 [running]:
github.com/ubuntu/zsys/vendor/github.com/bicomsystems/go-libzfs.(*Pool).ReloadProperties(0xc0001daea0)
        github.com/ubuntu/zsys/vendor/github.com/bicomsystems/go-libzfs/zpool.go:518 +0x592
github.com/ubuntu/zsys/vendor/github.com/bicomsystems/go-libzfs.(*Dataset).Pool(0xc000002380?)
        github.com/ubuntu/zsys/vendor/github.com/bicomsystems/go-libzfs/zfs.go:306 +0x85
github.com/ubuntu/zsys/internal/zfs.(*Dataset).refreshProperties(0xc000199300, {0xc2c9c0, 0x1122b80})
        github.com/ubuntu/zsys/internal/zfs/helpers.go:41 +0x406
github.com/ubuntu/zsys/internal/zfs.newDatasetTree({0xc2c9c0, 0x1122b80}, {0xc345f0, 0xc000210a80}, 0xc0001db520)
        github.com/ubuntu/zsys/internal/zfs/helpers.go:214 +0x239
github.com/ubuntu/zsys/internal/zfs.(*Zfs).Refresh(0xc000035920, {0xc2c9c0, 0x1122b80})
        github.com/ubuntu/zsys/internal/zfs/zfs.go:118 +0x1f6
github.com/ubuntu/zsys/internal/zfs.New({0xc2c9c0, 0x1122b80}, {0xc0001db7c0, 0x1, 0xc0001db7b0?})
        github.com/ubuntu/zsys/internal/zfs/zfs.go:93 +0xf4
github.com/ubuntu/zsys/internal/machines.New({0xc2c9c0, _}, {_, _}, {_, _, _})
        github.com/ubuntu/zsys/internal/machines/machines.go:128 +0x2dd
github.com/ubuntu/zsys/internal/daemon.New({0xb3cf90, 0xf}, {0x0, 0x0, 0x0?})
        github.com/ubuntu/zsys/internal/daemon/daemon.go:115 +0x52d
github.com/ubuntu/zsys/cmd/zsysd/daemon.init.func2(0xc000199100?, {0xc0000474e0?, 0x4?, 0xb34909?})
        github.com/ubuntu/zsys/cmd/zsysd/daemon/zsysd.go:30 +0x4e
github.com/ubuntu/zsys/vendor/github.com/spf13/cobra.(*Command).execute(0x109e7c0, {0xc0000340d0, 0x1, 0x1})
        github.com/ubuntu/zsys/vendor/github.com/spf13/cobra/command.go:987 +0xa91
github.com/ubuntu/zsys/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x109e7c0)
        github.com/ubuntu/zsys/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff
github.com/ubuntu/zsys/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        github.com/ubuntu/zsys/vendor/github.com/spf13/cobra/command.go:1039
main.main()
        github.com/ubuntu/zsys/cmd/zsysd/main.go:36 +0xa7

[StoatWblr (stoatwblr)]

AFter that error, subsequent start attempts show:

# /sbin/zsysd -v
ERROR Couldn't register grpc server: failed to listen on "/run/zsysd.sock": listen unix /run/zsysd.sock: bind: address already in use

# ps aux | grep zsys
root 541253 0.0 0.0 6844 2168 pts/5 S+ 18:21 0:00 grep --color=auto zsys

# lsof | grep run/zsys

[empty]

I haven't been removing snapshots

[Virnik (virnik)]

I can confirm, the same symptoms with the same zsys version. Initially, upgrade from 24.04 to 25.04 failed due to inability of grub to process zfs snapshots. Later, I tracked the failure down to zsys, which couldn't start. My system is so far recovered, but zsys is failing exactly as it does in Op's report.

[Pascal Schwarz (isoch)]

I updated using https://github.com/openzfs/zfs/issues/17337#issuecomment-2893283104 (after running into the described issue and rolling back snapshots manually) and am now also affected by zsysd not being able to start with error `panic: runtime error: index out of range [40] with length 37`.

I did not encounter the issue in 24.10, even after rolling back snapshots - it only started after/during the 25.04 update.

[Marcus Klein (kleini76)]

I am affected by zsysd not being able to start due to error "index out of range". Everything worked fine with 24.10 and started to break with the upgrade to 25.04.

[Randall Leeds (randall-leeds)]

I was able to fix this by adding the missing props from libzfs/sys/fs/zfs.h to the common.go in the vendored go-libzfs. The zsys package is using a [fork](https://github.com/ubuntu/go-libzfs) but [upstream](https://github.com/bicomsystems/go-libzfs) has a [commit](https://github.com/bicomsystems/go-libzfs/commit/cecf993df3de9f190f80389b94710d29b6c57066) that avoids loading props not known to go-libzfs.

I could cherry-pick that commit and make a PR and/or add the unknown props, but I don't know if anyone is maintaining the canonical fork of go-libzfs. I'll give it a shot, though.

[Randall Leeds (randall-leeds)]

I'm actually not sure that upstream commit fixes the issue, since it's counting the number of props returned and my testing shows that the props are not necessarily returned in order. I'm evaluating a fix and struggling through packaging and testing and will report back.

[Randall Leeds (randall-leeds)]

It _looks_ like I could assign this to me, but I don't know if that's the right action here. Unless I'm just forgetting (possible, after these years!) I am a first-time committer. I'm not sure whether there's a launchpad code repository I should push to or just use GitHub or how to go about getting changes merged and released. Any guidance would be appreciated. I've opened an issue on the zsys GitHub repository and linked to my branch where I've attempted a fix: https://github.com/ubuntu/zsys/issues/262

[Michał Sawicz (saviq)]

Hi Randall,

Thanks for looking at this!

Wherever the fix, it's best to propose it upstream (and yes the above is zsys's upstream), we can then bring that as a distro-patch to Ubuntu.

[Randall Leeds (randall-leeds)]

Opened a PR here: https://github.com/ubuntu/go-libzfs/pull/2