zpool create -f lxd /dev/vdb fails on cosmic (18.10) -- func27

[Alex Kavanagh]

Test: tests/gate-basic-cosmic-rocky

As part of the config, the lxd charm creates a pool device depending on the config. The test config is:

        lxd_config = {
            'block-devices': '/dev/vdb',
            'ephemeral-unmount': '/mnt',
            'storage-type': 'zfs',
            'overwrite': True
        }

The config drive is normally mounted on /mnt, and the lxd charm umounts it as part of the start up. The /etc/fstab on the unit is:

# cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
/dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
/dev/vdc none swap sw,comment=cloudconfig 0 0

However, even after umount-ing the /mnt off of /dev/vdb, the zpool create command still fails:

# zpool create -f lxd /dev/vdb
/dev/vdb is in use and contains a unknown filesystem.

If the /etc/fstab is edited so that /dev/vdb is *never* mounted and then rebooted, then the zpool create command succeeds:

# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
lxd 14.9G 106K 14.9G - 0% 0% 1.00x ONLINE -

# zpool status lxd
  pool: lxd
 state: ONLINE
  scan: none requested
config:

        NAME STATE READ WRITE CKSUM
        lxd ONLINE 0 0 0
          vdb ONLINE 0 0 0

errors: No known data errors

Something odd is going on with cosmic (18.10) and the combination of lxd, zfs and the kernel

lxd version: 3.6
zfsutils-linux/cosmic,now 0.7.9-3ubuntu6
Linux: 4.18.0-10-generic

[Alex Kavanagh (ajkavanagh)]

An option, for the test, is to change the lxd storage to a "dir" and *avoid* zfs. However, the underlying cause should be resolved.

[Ryan Beisner (1chb1n)]

Please provide a vanilla (non-charm) repro with a straightforward nova boot approach and confirm with those findings and logs here.

[Alex Kavanagh (ajkavanagh)]

Also checking on a non-juju serverstack cosmic instance:

# cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
/dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

# umount /mnt
# grep vdb /etc/mtab
# grep vdb /proc/mounts

# zpool create -f lxd /dev/vdb
/dev/vdb is in use and contains a ext4 filesystem.

Then edit out /dev/vdb from /etc/fstab and reboot:

# vim /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
#/dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

# reboot

# zpool create -f lxd /dev/vdb
(no error)

Definitely *something* is caching that it was mounted? Kernel, ZFS, something else?

[Alex Kavanagh (ajkavanagh)]

Further details of the vanilla nova instance:

# uname -a
Linux tinwood-cosmic 4.18.0-10-generic #11-Ubuntu SMP Thu Oct 11 15:13:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# apt show zfsutils-linux
Package: zfsutils-linux
Version: 0.7.9-3ubuntu6

# /snap/bin/lxc version
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:18.04

Client version: 3.6
Server version: 3.6

[Alex Kavanagh (ajkavanagh)]

Additionally, after the umount, the jbd kernel thread is still running for the block device, which is why it can't be claimed.

$ sudo ps ax | grep jbd
  333 ? S 0:00 [jbd2/vda1-8]
  824 ? S 0:00 [jbd2/vdb-8]
                                 ^^^^^
                                 This shouldn't be running.

However, nothing, anywhere, *seems* to be holding it. Kernel bug?

sudo grep -h vdb /proc/*/task/*/mountinfo

gives no output.

[Colin Ian King (colin-king)]

Can you supply me the commands required to reproduce this issue, I'm not familiar with the test environment you are using.

Thanks

[Colin Ian King (colin-king)]

Some notes:

1. /dev/vdb did appear to have an ext4 file system on it before we created the file system.
2. [jbd2/vdb-8] running implies the file system was mounted at some point on /dev/vdb
3. Nothing seemed to have mounted /dev/vdb

The file system was indeed mounted during boot:

[ 94.945578] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null)

After a reboot the [jbd2/vdb-8] thread is no longer running and zfs can create the file systems w/o any issues.

[Colin Ian King (colin-king)]

This is not a zfs related bug per se, it's more of a kernel related, so I'm re-assigned the bug

[Colin Ian King (colin-king)]

I added debug into the kernel and observed /dev/vdb is being mounted by systemd:

[ 9.859895] mount: 1624 mount, parent 1 systemd
[ 9.866398] mount: 1624 mount /dev/vdb

and also snap-confine is mounting mount too:

[ 9.966827] mount: 1663 snap-confine /mnt

Curiously, I've not seen any associated umounts.

I wonder if the namespace/confinement is causing the issue of not being able to see the mount. I need to debug this further.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/609024
Committed: https://git.openstack.org/cgit/openstack/charm-lxd/commit/?id=ff0ee580b808102be534c3f6035bcc157799c325
Submitter: Zuul
Branch: master

commit ff0ee580b808102be534c3f6035bcc157799c325
Author: Ryan Beisner <email address hidden>
Date: Tue Oct 9 09:46:54 2018 -0500

    Update func tests and sync charm-helpers

    dev-basic-cosmic-rocky stays due to:
    https://bugs.launchpad.net/charm-lxd/+bug/1801349<Paste>

    Change-Id: I4d00957e8092b1fefc71e337237aefb38b292914
    Related-Bug: #1801349

[Colin Ian King (colin-king)]

Is the above commit a fix to this issue?

[Alex Kavanagh (ajkavanagh)]

Colin, sorry, no; it's the bug that 'found' the issue; it's basically saying that we aren't enabling a particular gate test (i.e. it's staying dev-) due to this bug.

[Colin Ian King (colin-king)]

Hi Alex,

Is it possible to get the necessary commands to allow me to create the instances so I can do some experimental changes? I'm wrestling with getting a sane debugging environment with the instance you provided.

Colin

[Ryan Beisner (1chb1n)]

Hi Colin - We believe we've provided that in comment #3 above. It is a fresh Cosmic instance, followed by reproducer commands. Please let us know if this does not allow you to reproduce. Thank you.

[Alex Kavanagh (ajkavanagh)]

Ryan, my bad -- I should have updated the bug; I've provided Colin with a serverstack bastion and some scripts to do testing, and Colin has been doing that. Apologies that the bug didn't reflect this new information.

[Colin Ian King (colin-king)]

After adding a lot of debug into the kernel I can see what is causing the issue:

1. /dev/vdb is being mounted onto /mnt by systemd:

Nov 13 11:26:06 devstack kernel: [ 259.373361] mount: 1658 mount, parent 1 systemd
Nov 13 11:26:06 devstack kernel: [ 259.394961] mount: 1658 mount /dev/vdb
Nov 13 11:26:06 devstack kernel: [ 259.412916] do_mount: 1658 mount dev: /dev/vdb opts: '(null)' dir: '/mnt'

2. snap-confine is mounting /tmp/snap.rootfs_2SnJZF//mnt onto /mnt:

Nov 13 11:26:07 devstack kernel: [ 261.546874] mount: 1706 snap-confine /mnt
/inaccessible/dir opts: '(null)' dir: '/run/systemd/unit-root/root'
Nov 13 11:26:07 devstack kernel: [ 261.554827] do_mount: 1706 snap-confine dev: /mnt opts: '(null)' dir: '/tmp/snap.rootfs_2SnJZF//mnt'

So unmounting /mnt still leaves /dev/vdb marked as in use because of the snap-confine mount. There is no kernel bug here, the kernel still sees /dev/vdb as being, hence it's doing the correct thing in not allowing zfs to mount on the in-use /tmp/snap.rootfs_2SnJZF//mnt mount.

Colin

[Alex Kavanagh (ajkavanagh)]

Thanks Colin for tracking this done; very pleased it's not a kernel bug. I'll now take this up with the snappy team. Thank again for doing the detective work; much appreciated!

[Stéphane Graber (stgraber)]

Closing as we are dealing with a completely different charm now.