zfs frequently hangs (up to 30 secs) during sequential read

Bug #1725859 reported by Colin Leavett-Brown
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
Medium
Colin Ian King

Bug Description

Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as:

root@hetty:/home/crlb# zpool status
  pool: storage
 state: ONLINE
  scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017
config:

 NAME STATE READ WRITE CKSUM
 storage ONLINE 0 0 0
   mirror-0 ONLINE 0 0 0
     sdb ONLINE 0 0 0
     sdc ONLINE 0 0 0

errors: No known data errors
root@hetty:/home/crlb#

Found that I could re-create the problem with:

 rsync -av --progress <zfs_file> <internal_ext4_path>

Also found that:

  dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=10000000

Where "X" is either "b" or "c" does not hang.

Installed:

root@hetty:/home/crlb# apt list --installed | grep -i zfs
libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic]
zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic]
zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed]
root@hetty:/home/crlb#

Help please.

Revision history for this message
Colin Leavett-Brown (crlb) wrote :

Just updated to:
root@hetty:~# apt list --installed | awk '/zpool|zfs/'
libzfs2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic]
libzpool2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed]
zfs-zed/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic]
zfsutils-linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed]
root@hetty:~#

A little better, but the problem still happens.

Revision history for this message
Colin Ian King (colin-king) wrote :

1. what is your ZFS configuration (number of drives?)
2. size of your system (total memory, free memory).
3. Do your drive(s) suffer from large I/O latencies?
   - what is the output from iostat -mx

You may possibly get some benefit from setting spl_kmem_cache_slab_limit to zero as a module parameter for the spl ZoL kernel module. This avoids SLUB slab merging entirely and maybe helpful from anecdotal evidence I've seen.

Changed in zfs-linux (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Colin Ian King (colin-king)
status: Triaged → Incomplete
Revision history for this message
Colin Leavett-Brown (crlb) wrote :

Hello Colin,
  Thank you for responding. To Answer your questions:
  1. I have two external drives, sdb and sdc, in USB 3 Orico enclosure. The drives are mirrored.
  2. Atom D525 with 4GiB RAM. The attached screenshot shows "vmstat 5 5" output while VLC is playing a movie. It shows 1.5GiB of free memory, no swap used.
  3. Screenshot also shows output from "iostat -mx", and that I manually set the spl_kmem_cache_slab_limit to zero as suggested.

  I played a movie which paused repeatedly for 30 seconds and one time for several minutes. When the pause occurs, VLC is completely hung until the requested I/O completes. The end of pause is heralded by the I/O activity LED on the drive enclosure.

  I want to emphasize/re-iterate a couple of points from my original post:
  1. This is not a VLC problem. I can recreate the problem with any app (cp, rsyn, etc) that reads sequentially from the ZFS file system.
  2. I do not believe this is a drive/latency problem: I can dd the content of sdb and sdc (to /dev/null) without any hang.

  Regards, Colin.

Revision history for this message
Colin Ian King (colin-king) wrote :

The iostat -mx shows that the /dev/sdb /dev/sdc devices have a very high read I/O latency of ~80-110 milliseconds which may be the underlying issue.

I suggest seeing how long it takes to read data from these raw devices to sanity check their maximum speed. One can do this by doing:

sync
echo 3 | sudo tee /proc/sys/vm/drop_caches
dd if=/dev/sdb of=/dev/null bs=512 count=1048576

..and wait for 512MB of data to be read and see what the read speed is of the raw device.

Revision history for this message
Colin Leavett-Brown (crlb) wrote :

Hello Colin, I have a couple more screen shots showing the raw performance of both drives; the first using 512MiB as you suggested and the second with a larger sample and using the dd commands status=progress option to monitor for pauses. Also in the second ss, I performed the test on an iso file from the zfs file system. I did not see any pauses on the raw devices, but then I only saw momentary pauses (one second or less) on the zfs file. Regards, Colin.

Revision history for this message
Colin Leavett-Brown (crlb) wrote :
Revision history for this message
Colin Ian King (colin-king) wrote :

Just one more check, what is the block size of the raw devices? One can check this using:

sudo blockdev --getpbsz /dev/sdb
sudo blockdev --getpbsz /dev/sdc

Revision history for this message
Colin Leavett-Brown (crlb) wrote :

root@hetty:/home/crlb# blockdev --getpbsz /dev/sdb
512
root@hetty:/home/crlb# blockdev --getpbsz /dev/sdc
512
root@hetty:/home/crlb#

Revision history for this message
Colin Ian King (colin-king) wrote :

I've tried to simulate this with very slow media and can't yet reproduce this issue. Can you play the DVD image and also run the following command in another terminal:

iostat 1 300 > iostat.log

this will capture 5 minutes of activity. please then attach the log to the bug report.

Revision history for this message
Colin Leavett-Brown (crlb) wrote :

Hello Colin,
  I set a couple of ZFS options a day or two ago, and have bee playing videos to see if we still have the problem. The options I set were:

  o zfs set xattr=sa storage
  o zfs set xattr=sa storage/vault
  o zfs set ralatime=on storage
  o zfs set ralatime=on storage/vault

  We've played about four hours of video, and we have not seen the hang.

  Do you still want to pursue this? I could set the options back and do the trace or leave the options and do the trace, or should we just consider this fixed? My only concern about the latter option is that the installation was a default install with Ubuntu 17.10 and it seems these options should be the default on any linux distro.

  Let me know what you want to do?

Best regards, Colin.

Revision history for this message
Colin Ian King (colin-king) wrote :

Ah, that is a good solution. By default, ZFS on Linux will store xattrs in a hidden folder, so there are multiple seeks per xattr and it's poorly cached, so it will impact on performance especially on the slower block devices. The zfs set xattr=sa will store this in the inodes and will really help.

Let's mark this as a suitable workaround and hence a fix.

Changed in zfs-linux (Ubuntu):
status: Incomplete → Fix Released
Revision history for this message
Colin Ian King (colin-king) wrote :

And if anyone is wondering why this is not the default, the answer is here: https://github.com/zfsonlinux/zfs/issues/443

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.