LVM driver can't create snapshots when run with Ubuntu/Bionic Beaver

[Goutham Pacha Ravi]

Steps to Reproduce:

1) Setup manila with the LVM driver on Ubuntu Bionic Beaver (18.04 LTS)
2) Create a share type that supports snapshots and creating shares from snapshots
3) Create a share with share type from 1
4) Create a snapshot of share from 3
5) Monitor status of snapshot

Expected:
Snapshot creation should succeed

Actual:
Snapshot creation fails, this is the error in manila-share manager log:

ERROR oslo_messaging.rpc.server [None req-074c807e-3216-4c4a-ad32-a4567381e46d None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Command: sudo manila-rootwrap /etc/manila/rootwrap.conf tune2fs -U random /dev/mapper/lvm--shares-share--snapshot--65ddd2f4--6628--4c0e--ba78--613c3a7f91ea
Exit code: 1
Stdout: u'tune2fs 1.44.1 (24-Mar-2018)\n\nThis operation requires a freshly checked filesystem.\n\nPlease run e2fsck -f on the filesystem.\n\n'
Stderr: u''
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 187, in wrapped
ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 589, in wrapper
ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2908, in create_snapshot
ERROR oslo_messaging.rpc.server exception=excep)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2894, in create_snapshot
ERROR oslo_messaging.rpc.server context, snapshot_instance, share_server=share_server) or {}
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/lvm.py", line 425, in create_snapshot
ERROR oslo_messaging.rpc.server self._create_snapshot(context, snapshot)
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/lvm.py", line 136, in _create_snapshot
ERROR oslo_messaging.rpc.server 'tune2fs', '-U', 'random', snapshot_device_name, run_as_root=True,
ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 97, in execute
ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.

RCA:
- tune2fs comes from e2fsprogs and that package has been updated in Ubuntu 18.04 https://packages.ubuntu.com/search?keywords=e2fsprogs
- When executing tune2fs -U random <lvm-share> manually, the error becomes clearer:

$ sudo tune2fs -U random /dev/mapper/lvm--shares1-share--snapshot--ea4571ca--7ed4--4f68--8fbb--5ddb6b7a12d0
tune2fs 1.44.1 (24-Mar-2018)

This operation requires a freshly checked filesystem.

Please run e2fsck -f on the filesystem.

- Looking at the upstream release notes for e2fsprogs, there seems to be a change in the tune2fs package and the docs were enhanced with debian issue https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=857336

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611199

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611199
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=817cce347a567863173e3065084bfabc35845907
Submitter: Zuul
Branch: master

commit 817cce347a567863173e3065084bfabc35845907
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Oct 16 16:48:58 2018 -0700

    [LVM] Run filesystem check before assigning UUID

    We assign a random UUID with tune2fs to snapshots
    and shares created from snapshots so that they
    don't conflict with the parent shares/snapshots
    when both of them are being mounted.

    tune2fs requires that a filesystem check be
    performed "recently" before UUID assignments.
    So, perform a filesystem check right away to
    allow tune2fs to assign a random UUID.

    Change-Id: I858a318f7a83e033cc3f2699859e38b6b74c8d24
    Related-Bug: #1645751
    Closes-Bug: #1798219

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/637378

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637378
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=55507cc603f8ab00b2b6dbeb2d4a2fa4e862f317
Submitter: Zuul
Branch: stable/rocky

commit 55507cc603f8ab00b2b6dbeb2d4a2fa4e862f317
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Oct 16 16:48:58 2018 -0700

    [LVM] Run filesystem check before assigning UUID

    We assign a random UUID with tune2fs to snapshots
    and shares created from snapshots so that they
    don't conflict with the parent shares/snapshots
    when both of them are being mounted.

    tune2fs requires that a filesystem check be
    performed "recently" before UUID assignments.
    So, perform a filesystem check right away to
    allow tune2fs to assign a random UUID.

    Depends-on: https://review.openstack.org/#/c/637287/
    Change-Id: I858a318f7a83e033cc3f2699859e38b6b74c8d24
    Related-Bug: #1645751
    Closes-Bug: #1798219
    (cherry picked from commit 817cce347a567863173e3065084bfabc35845907)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637566

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637566
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e9e4a3a6e3917216c284018b4f7db854a309fe4b
Submitter: Zuul
Branch: stable/queens

commit e9e4a3a6e3917216c284018b4f7db854a309fe4b
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Oct 16 16:48:58 2018 -0700

    [LVM] Run filesystem check before assigning UUID

    We assign a random UUID with tune2fs to snapshots
    and shares created from snapshots so that they
    don't conflict with the parent shares/snapshots
    when both of them are being mounted.

    tune2fs requires that a filesystem check be
    performed "recently" before UUID assignments.
    So, perform a filesystem check right away to
    allow tune2fs to assign a random UUID.

    Depends-on: https://review.openstack.org/#/c/637565
    Change-Id: I858a318f7a83e033cc3f2699859e38b6b74c8d24
    Related-Bug: #1645751
    Closes-Bug: #1798219
    (cherry picked from commit 817cce347a567863173e3065084bfabc35845907)
    (cherry picked from commit 55507cc603f8ab00b2b6dbeb2d4a2fa4e862f317)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 8.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 7.2.0 release.