Superfluous re-mount attempts with the Quobyte Nova driver and multi-registry volume URLs

[Silvan Kaiser]

When using a multi-registry volume URL in the Cinder Quobyte driver the Nova Quobyte driver does not detect existing mounts correctly. Upon trying to mount the given volume the driver fails because the mount already exists:

[..]
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/block_device.py", line 389, in attach
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server device_type=self['device_type'], encryption=encryption)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1248, in attach_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server self._connect_volume(connection_info, disk_info, instance)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1181, in _connect_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server vol_driver.connect_volume(connection_info, disk_info, instance)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/quobyte.py", line 147, in connect_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server CONF.libvirt.quobyte_client_cfg)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/quobyte.py", line 61, in mount_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server utils.execute(*command)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/utils.py", line 229, in execute
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 419, in execute
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Command: mount.quobyte --disable-xattrs 78.46.57.153:7861,78.46.57.153:7861,78.46.57.153:7861/82000e41-c6ac-4be2-b31a-0543db93767c /mnt/quobyte-volume/531b7439e360bdea0a79870354906cab
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Exit code: 4
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Stdout: u'mount.quobyte failed: Unable to initialize mount point\n'
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Stderr: u"Logging to file /opt/stack/logs/quobyte_client.log.\nfuse: mountpoint is not empty\nfuse: if you are sure this is safe, use the 'nonempty' mount option\n"
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server

[Silvan Kaiser (2-silvan)]

I added this to Cinder, too, as the volume url used in here originates from there and we want to update the doc on that config option.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/526671

[Silvan Kaiser (2-silvan)]

Nova fix is available for review in https://review.openstack.org/#/c/522245/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/522245
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7
Submitter: Zuul
Branch: master

commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Added mount fstype based validation of Quobyte mounts

    The validation of Quobyte mounts is extended to validate based
    on a mounts file system type being set to "fuse.quobyte".
    This includes adding a new StaleVolumeMount exception type to
    the Nova exceptions.
    This also closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Finally this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    Closes-Bug: #1730933
    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/660706

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660706
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=656aa1cd40570154df606484d31616989b5296aa
Submitter: Zuul
Branch: stable/stein

commit 656aa1cd40570154df606484d31616989b5296aa
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Fixes multi-registry config in Quobyte driver

    This closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Besides, this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    NOTE: The original commit also added a new feature (fs type
    based validation) which is omitted in this backport.

    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59
    (cherry picked from commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/662695

[Silvan Kaiser (2-silvan)]

So far i haven't been able to find out how to add a release (rocky in this case) as a an also affected project, sorry.
Hints welcome.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662695
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c958ad8a68ea9a8ea465d9e3dd889248d9f42481
Submitter: Zuul
Branch: stable/rocky

commit c958ad8a68ea9a8ea465d9e3dd889248d9f42481
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Fixes multi-registry config in Quobyte driver

    This closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Besides, this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    NOTE: The original commit also added a new feature (fs type
    based validation) which is omitted in this backport.

    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59
    (cherry picked from commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7)
    (cherry picked from commit 656aa1cd40570154df606484d31616989b5296aa)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/687058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687058
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=37e45ad44c22a5b9346bece8279cde3691562e52
Submitter: Zuul
Branch: stable/queens

commit 37e45ad44c22a5b9346bece8279cde3691562e52
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Fixes multi-registry config in Quobyte driver

    This closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Besides, this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    NOTE: The original commit also added a new feature (fs type
    based validation) which is omitted in this backport.

    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59
    (cherry picked from commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7)
    (cherry picked from commit 656aa1cd40570154df606484d31616989b5296aa)
    (cherry picked from commit c958ad8a68ea9a8ea465d9e3dd889248d9f42481)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.13 release.