systemd var-lib-machines.mount wrong fs type

[Satish Patel]

Openstack Release: Wallaby
OS Distro: Ubuntu 20.04

After rebooting infra node encounter following error:

root@ostack-ams-infra-1-3:~# journalctl -u var-lib-machines.mount -f
May 16 18:23:13 ostack-ams-infra-1-3.example.net mount[1026]: mount: /var/lib/machines: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
May 16 18:23:13 ostack-ams-infra-1-3.example.net systemd[1]: Mounting Virtual Machine and Container Storage (Compatibility)...
May 16 18:23:13 ostack-ams-infra-1-3.example.net systemd[1]: var-lib-machines.mount: Mount process exited, code=exited, status=32/n/a
May 16 18:23:13 ostack-ams-infra-1-3.example.net systemd[1]: var-lib-machines.mount: Failed with result 'exit-code'.
May 16 18:23:13 ostack-ams-infra-1-3.example.net systemd[1]: Failed to mount Virtual Machine and Container Storage (Compatibility).

# filesystem type is ext4
root@ostack-ams-infra-1-3:~# file /var/lib/machines.raw
/var/lib/machines.raw: Linux rev 1.0 ext4 filesystem data, UUID=b2c7accf-c3bb-4ab0-9301-371a099de544 (extents) (64bit) (large files) (huge files)

# In var-lib-machines.mount systemd has wrong filesystem Type (Type: btrfs).

root@ostack-ams-infra-1-3:~# cat /lib/systemd/system/var-lib-machines.mount
# SPDX-License-Identifier: LGPL-2.1+
#
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

# This unit is required for pre-240 versions of systemd that automatically set
# up /var/lib/machines.raw as loopback-mounted btrfs file system. Later
# versions don't do that anymore, but let's keep minimal compatibility by
# mounting the image still, if it exists.

[Unit]
Description=Virtual Machine and Container Storage (Compatibility)
ConditionPathExists=/var/lib/machines.raw

[Mount]
What=/var/lib/machines.raw
Where=/var/lib/machines
Tyep=btrfs
Options=loop

# Solution
Workaround is to change Type=ext4

[Dmitriy Rabotyagov (noonedeadpunk)]

I think this patch is related: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/834183

However, it doesn't take into account, that the mount in topic can be already mounted by system package. I'm not sure if they have same naming in the end - if yes, it should solve issue as that would be a simple override then.

If not - some care should be taken of that.

Also something backportable should be placed to the stable branches.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/843169

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/843392

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/843392
Committed: https://opendev.org/openstack/openstack-ansible-lxc_hosts/commit/7d71769b721cca23ced26453d303112515b0a508
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7d71769b721cca23ced26453d303112515b0a508
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue May 24 18:12:18 2022 +0200

    Do not mess up with distro-provided machines mount

    var-lib-machines.mount is provided by systemd-container package
    on ubuntu/debian [1]

    With package update our file will be overriden which end up on
    mount failure.

    To avoid that we place it as override to /etc/

    [1] https://packages.ubuntu.com/focal/amd64/systemd-container/filelist

    Change-Id: I226ab48b1ff7ec3c7da963e00e76b24b2b79608c
    Closes-Bug: #1973762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/843169
Committed: https://opendev.org/openstack/openstack-ansible-lxc_hosts/commit/fc0810cc9761335a4d2920d05a926daff5c93139
Submitter: "Zuul (22348)"
Branch: stable/xena

commit fc0810cc9761335a4d2920d05a926daff5c93139
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue May 24 18:12:18 2022 +0200

    Do not mess up with distro-provided machines mount

    var-lib-machines.mount is provided by systemd-container package
    on ubuntu/debian [1]

    With package update our file will be overriden which end up on
    mount failure.

    To avoid that we place it as override to /etc/

    [1] https://packages.ubuntu.com/focal/amd64/systemd-container/filelist

    Change-Id: I226ab48b1ff7ec3c7da963e00e76b24b2b79608c
    Closes-Bug: #1973762

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts xena-eom release.