Ubuntu 16.04.1 Install zfsutils-linux Panic Error, endless loop

Bug #1628553 reported by Matthew Hillman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Native ZFS for Linux
Fix Released
Unknown
zfs-linux (Ubuntu)
Won't Fix
Medium
Colin Ian King

Bug Description

Fresh install of Ubuntu 16.04.1, installed all updates / upgrades for system before installing zfsutils-linux. The install processes all the way to Reached target Swap then has a PANIC error and loops endlessly.

There is a zpool there already (was) with 3TB of data on it. This only happened after the system upgrade, so I reinstalled the boot drive / OS to see if it was just a issue with that, clearly it's an issue with the zfsutils-linux package for 16.04.1

Package version installed zfsutils-linux 0.6.5.6-0ubuntu12

Screenshot attached as I can't actually do anything once the boot gets to this stage:

Revision history for this message
Matthew Hillman (matthewahillman) wrote :
Revision history for this message
Richard Laager (rlaager) wrote :

I think I got the numbers right, but in any case, this is the gist of the error:
PANIC: blkptr at ffff88040b993640 DVA 1 has invalid OFFSET 36830022909049856

What was the last version of ZFS-on-Linux (that you were using) that works?

Was that the only version of ZFS you had ever used?

Can you attach the output of:
sudo zpool status
sudo zpool upgrade
sudo zdb

I assume you have swap on ZFS and this is breaking when it tries to access the swap dataset? If so, can you provide this output (substitute for rpool/swap as appropriate), which you should probably compress:
sudo zdb -dvvvv rpool/swap

I'm not sure that'll be enough information (as I'm not expert enough to debug at this level), but it should be a start for me to take this to someone else.

If it is happening with swap, try moving the swap dataset out of the way, and create a new one:
sudo zfs move rpool/swap rpool/swap-broken
sudo zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \
      -o logbias=throughput -o sync=always \
      -o primarycache=metadata -o secondarycache=none \
      -o com.sun:auto-snapshot=false rpool/swap
sudo mkswap -f /dev/zvol/rpool/swap

This will hopefully get you working again on the latest ZFS. If you keep the old dataset around, then we may still be able to debug how this happened.

Revision history for this message
Matthew Hillman (matthewahillman) wrote :

The latest version was Ubuntu 16.04 whatever the latest version on that was, however I tried a wipe of my boot drive and a reinstall of 16.04 last night with the same errors, panic, and endless loop thing it does at the bottom of the picture.

I even tried 16.10 beta 2 server but that didn't even boot.

Unfortunately I can't do any zpool commands as the screenshot is from when I was trying to install zfsutils-linux, it doesn't even finish the install enough for me to use any commands.

When it gets to the Panic line my only options are to reinstall the whole boot drive and try again.

Revision history for this message
Richard Laager (rlaager) wrote :

If you have a separate boot drive, you should be able to install zfsutils-linux. Worst case, disconnect the ZFS drives.

If necessary, disable the zfs-import-scan and zfs-import-cache services. You should reach a point where you can boot, and zfsutils-linux is installed, and the ZFS drives are connected, but the pool is not imported.

Then, import the pool without mounting anything:
sudo zpool import -N POOLNAME

Revision history for this message
Matthew Hillman (matthewahillman) wrote :

Ok I will try removing the drives install zfsutils-linux and see if it works when I put the drives back in, didn't think of that tbh. Will try that tonight and update later with the results.

Thanks

Revision history for this message
Richard Laager (rlaager) wrote :

For the record, this is also posted upstream:
https://github.com/zfsonlinux/zfs/issues/5173

Revision history for this message
Matthew Hillman (matthewahillman) wrote :

I've managed to get zfsutils-linux installed, but I am unable to import it, I just get the timeout loop in my picture.

Revision history for this message
Richard Laager (rlaager) wrote :

Try loading the ZFS module with zfs_recover=1, and then try a read-only import.

rmmod zfs
modprobe zfs zfs_recover=1

zpool import -o readonly -f -R /mnt POOL

If it imports, at least then you can see how your data looks.

Revision history for this message
Richard Laager (rlaager) wrote :

That should be "-o readonly=on".

Revision history for this message
Colin Ian King (colin-king) wrote :

@Matthew, did Richard's advice help?

Changed in zfs-linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Colin Ian King (colin-king) wrote :

Hi, if this is still and issue please let me know. I'm going to close the bug if I don't get any response in 1 week.

Changed in zfs-linux (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
importance: Undecided → Medium
Changed in zfs-linux (Ubuntu):
status: Incomplete → Won't Fix
Changed in zfs:
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.