inexplicably large file reported by zfs filesystem

[Matt LaPlante]

I have a zfs filesystem containing a single qemu kvm disk image. The vm has been working normally. The image file is only allocated 10G, however today I became aware that the file, when examined from the ZFS host (hypervisor) is reporting an inexplicable, massive file size of around 12T. 12T is larger than the pool itself. Snapshots or other filesystem features should not be involved. I'm suspicious that the file(system?) has been corrupted.

root@fusion:~# ls -l /data/store/vms/plexee/
total 6164615
-rw-r--r-- 1 root root 12201321037824 Apr 23 11:49 plexee-root
                        ^^ !!!!!!

root@fusion:~# qemu-img info /data/store/vms/plexee/plexee-root
image: /data/store/vms/plexee//plexee-root
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 5.9G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

root@fusion:~# zfs list rpool/DATA/fusion/store/vms/plexee
NAME USED AVAIL REFER MOUNTPOINT
rpool/DATA/fusion/store/vms/plexee 5.88G 484G 5.88G /data/store/vms/plexee

root@fusion:~# zfs get all rpool/DATA/fusion/store/vms/plexee
NAME PROPERTY VALUE SOURCE
rpool/DATA/fusion/store/vms/plexee type filesystem -
rpool/DATA/fusion/store/vms/plexee creation Mon Mar 26 9:50 2018 -
rpool/DATA/fusion/store/vms/plexee used 5.88G -
rpool/DATA/fusion/store/vms/plexee available 484G -
rpool/DATA/fusion/store/vms/plexee referenced 5.88G -
rpool/DATA/fusion/store/vms/plexee compressratio 1.37x -
rpool/DATA/fusion/store/vms/plexee mounted yes -
rpool/DATA/fusion/store/vms/plexee quota none default
rpool/DATA/fusion/store/vms/plexee reservation none default
rpool/DATA/fusion/store/vms/plexee recordsize 128K default
rpool/DATA/fusion/store/vms/plexee mountpoint /data/store/vms/plexee inherited from rpool/DATA/fusion
rpool/DATA/fusion/store/vms/plexee sharenfs off default
rpool/DATA/fusion/store/vms/plexee checksum on default
rpool/DATA/fusion/store/vms/plexee compression lz4 inherited from rpool
rpool/DATA/fusion/store/vms/plexee atime off inherited from rpool
rpool/DATA/fusion/store/vms/plexee devices off inherited from rpool
rpool/DATA/fusion/store/vms/plexee exec on default
rpool/DATA/fusion/store/vms/plexee setuid on default
rpool/DATA/fusion/store/vms/plexee readonly off default
rpool/DATA/fusion/store/vms/plexee zoned off default
rpool/DATA/fusion/store/vms/plexee snapdir hidden default
rpool/DATA/fusion/store/vms/plexee aclinherit restricted default
rpool/DATA/fusion/store/vms/plexee canmount on default
rpool/DATA/fusion/store/vms/plexee xattr on default
rpool/DATA/fusion/store/vms/plexee copies 1 default
rpool/DATA/fusion/store/vms/plexee version 5 -
rpool/DATA/fusion/store/vms/plexee utf8only off -
rpool/DATA/fusion/store/vms/plexee normalization none -
rpool/DATA/fusion/store/vms/plexee casesensitivity sensitive -
rpool/DATA/fusion/store/vms/plexee vscan off default
rpool/DATA/fusion/store/vms/plexee nbmand off default
rpool/DATA/fusion/store/vms/plexee sharesmb off default
rpool/DATA/fusion/store/vms/plexee refquota none default
rpool/DATA/fusion/store/vms/plexee refreservation none default
rpool/DATA/fusion/store/vms/plexee primarycache all default
rpool/DATA/fusion/store/vms/plexee secondarycache all default
rpool/DATA/fusion/store/vms/plexee usedbysnapshots 0 -
rpool/DATA/fusion/store/vms/plexee usedbydataset 5.88G -
rpool/DATA/fusion/store/vms/plexee usedbychildren 0 -
rpool/DATA/fusion/store/vms/plexee usedbyrefreservation 0 -
rpool/DATA/fusion/store/vms/plexee logbias latency default
rpool/DATA/fusion/store/vms/plexee dedup off default
rpool/DATA/fusion/store/vms/plexee mlslabel none default
rpool/DATA/fusion/store/vms/plexee sync standard default
rpool/DATA/fusion/store/vms/plexee refcompressratio 1.37x -
rpool/DATA/fusion/store/vms/plexee written 5.88G -
rpool/DATA/fusion/store/vms/plexee logicalused 7.94G -
rpool/DATA/fusion/store/vms/plexee logicalreferenced 7.94G -
rpool/DATA/fusion/store/vms/plexee filesystem_limit none default
rpool/DATA/fusion/store/vms/plexee snapshot_limit none default
rpool/DATA/fusion/store/vms/plexee filesystem_count none default
rpool/DATA/fusion/store/vms/plexee snapshot_count none default
rpool/DATA/fusion/store/vms/plexee snapdev hidden default
rpool/DATA/fusion/store/vms/plexee acltype off default
rpool/DATA/fusion/store/vms/plexee context none default
rpool/DATA/fusion/store/vms/plexee fscontext none default
rpool/DATA/fusion/store/vms/plexee defcontext none default
rpool/DATA/fusion/store/vms/plexee rootcontext none default
rpool/DATA/fusion/store/vms/plexee relatime off default
rpool/DATA/fusion/store/vms/plexee redundant_metadata all default
rpool/DATA/fusion/store/vms/plexee overlay off default

root@fusion:~# dpkg -l | grep zfs
ii libzfs2linux 0.6.5.6-0ubuntu20 amd64 Native OpenZFS filesystem library for Linux
ii zfs-dkms 0.6.5.6-0ubuntu20 amd64 Native OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.6.5.6-0ubuntu20 all Native OpenZFS filesystem documentation and examples.
ii zfs-initramfs 0.6.5.6-0ubuntu20 all Native OpenZFS root filesystem capabilities for Linux
ii zfs-zed 0.6.5.6-0ubuntu20 amd64 OpenZFS Event Daemon (zed)
ii zfsutils-linux 0.6.5.6-0ubuntu20 amd64 Native OpenZFS management utilities for Linux

root@fusion:~# uname -a
Linux fusion 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[Matt LaPlante (cybrmatt)]

A scrub of the pool detected no issues, no errors.

[Colin Ian King (colin-king)]

There could be two explanations for this huge file that seems to be larger than the file system. From the path name it seems that this file is probably a virtual machine image, so it may contain either a lot of "empty" data (zero'd) or may have lots of file "holes". So...

1. Is it compressed?

For example, a VM image file containing lots of zeros can be very efficiently compressed, hence it's compressed size is very small but can appear to be much larger than the real size used in the zfs file system.

2. Does the image have "holes"?

VM image files may just use space that is non-zero and the zero'd data blocks are not physically stored on the zfs file system.

You can see the real physical size of the file using 'du -h' on the file, e.g.

/data/store/vms/plexee/plexee-root

..that should tell you the size of the file as really used on the file system rather than the size of the file as it appears when you read it.

[Colin Ian King (colin-king)]

@Matt, did the above help?

[Matt LaPlante (cybrmatt)]

1. Yes, compression is on for the parent and inherited. The ratio only shows 1.37x, which seems rather low if a 12T file was all zeros except for >10G?

root@fusion:~# zfs get all rpool/DATA/fusion/store/vms/plexee | grep comp
rpool/DATA/fusion/store/vms/plexee compressratio 1.37x -
rpool/DATA/fusion/store/vms/plexee compression lz4 inherited from rpool
rpool/DATA/fusion/store/vms/plexee refcompressratio 1.37x

2.
root@fusion:~# du -h /data/store/vms/plexee/plexee-root
5.6G /data/store/vms/plexee/plexee-root

This is what I would expect the VM to be using based on how the file was provisioned and its internal disk state.

[Hajo Möller (dasjoe)]

Zero-filled data is not compressed by the set compression algorithm but
gets filtered by zero-length encoding, so the zeros never hit lz4 and
compressratio does not include them.

[Colin Ian King (colin-king)]

You can get an idea of how "full" a file is using zdb, e.g:

sudo zdb -O pool-ssd/virt ubuntu17.10-amd64-desktop.qcow2

    Object lvl iblk dblk dsize dnsize lsize %full type
    114737 3 128K 128K 14.0G 512 20.0G 84.49 ZFS plain file

in the above example, pool-ssd/virt was the pool + zfs file system and ubuntu17.10-amd64-desktop.qcow2 was the name of the VM image.

[Matt LaPlante (cybrmatt)]

Great, thank you. zdb confirms what we already sort of suspected: the file is ~12T with only 5.5G of real content. I was aware that ZFS reported on disk size differently, but the fact that this isn't a feature of compression threw me off in the output.

 Object lvl iblk dblk dsize dnsize lsize %full type
 7 5 16K 128K 5.51G 512 11.1T 0.07 ZFS plain file

At any rate, it's still not clear to me how we arrived at this giant (but sparse) file, since the VM and image tools (qemu-img) only consider it to be 10G logical. I'm guessing that the implication is that the bug is with qemu rather than ZFS. My initial concern was that it had been a storage layer corruption, but I guess it's equally plausible that somehow the emulation layer ran wild and just started massively growing the file out of bounds on disk.

[Colin Ian King (colin-king)]

Thanks Matt for the update. Shall we just this bug against ZFS then?

[Colin Ian King (colin-king)]

I meant to say "just close this bug against ZFS"...

[Matt LaPlante (cybrmatt)]

Yes, please close... thanks again.