Superfluous re-mount attempts with the Quobyte Nova driver and multi-registry volume URLs

Bug #1737131 reported by Silvan Kaiser on 2017-12-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Silvan Kaiser
Queens
Low
Silvan Kaiser
Rocky
Low
Silvan Kaiser
Stein
Low
Silvan Kaiser

Bug Description

When using a multi-registry volume URL in the Cinder Quobyte driver the Nova Quobyte driver does not detect existing mounts correctly. Upon trying to mount the given volume the driver fails because the mount already exists:

[..]
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/block_device.py", line 389, in attach
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server device_type=self['device_type'], encryption=encryption)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1248, in attach_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server self._connect_volume(connection_info, disk_info, instance)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1181, in _connect_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server vol_driver.connect_volume(connection_info, disk_info, instance)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/quobyte.py", line 147, in connect_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server CONF.libvirt.quobyte_client_cfg)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/quobyte.py", line 61, in mount_volume
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server utils.execute(*command)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/utils.py", line 229, in execute
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 419, in execute
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Command: mount.quobyte --disable-xattrs 78.46.57.153:7861,78.46.57.153:7861,78.46.57.153:7861/82000e41-c6ac-4be2-b31a-0543db93767c /mnt/quobyte-volume/531b7439e360bdea0a79870354906cab
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Exit code: 4
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Stdout: u'mount.quobyte failed: Unable to initialize mount point\n'
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server Stderr: u"Logging to file /opt/stack/logs/quobyte_client.log.\nfuse: mountpoint is not empty\nfuse: if you are sure this is safe, use the 'nonempty' mount option\n"
2017-12-08 08:32:29.277 25660 ERROR oslo_messaging.rpc.server

Silvan Kaiser (2-silvan) on 2017-12-08
Changed in nova:
assignee: nobody → Silvan Kaiser (2-silvan)
Silvan Kaiser (2-silvan) on 2017-12-08
Changed in cinder:
assignee: nobody → Silvan Kaiser (2-silvan)
Silvan Kaiser (2-silvan) wrote :

I added this to Cinder, too, as the volume url used in here originates from there and we want to update the doc on that config option.

Fix proposed to branch: master
Review: https://review.openstack.org/526671

Changed in cinder:
status: New → In Progress
Silvan Kaiser (2-silvan) on 2017-12-11
no longer affects: cinder
Changed in nova:
status: New → In Progress
Silvan Kaiser (2-silvan) wrote :

Nova fix is available for review in https://review.openstack.org/#/c/522245/

Reviewed: https://review.openstack.org/522245
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7
Submitter: Zuul
Branch: master

commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Added mount fstype based validation of Quobyte mounts

    The validation of Quobyte mounts is extended to validate based
    on a mounts file system type being set to "fuse.quobyte".
    This includes adding a new StaleVolumeMount exception type to
    the Nova exceptions.
    This also closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Finally this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    Closes-Bug: #1730933
    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/660706
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=656aa1cd40570154df606484d31616989b5296aa
Submitter: Zuul
Branch: stable/stein

commit 656aa1cd40570154df606484d31616989b5296aa
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Fixes multi-registry config in Quobyte driver

    This closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Besides, this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    NOTE: The original commit also added a new feature (fs type
    based validation) which is omitted in this backport.

    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59
    (cherry picked from commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7)

Silvan Kaiser (2-silvan) wrote :

So far i haven't been able to find out how to add a release (rocky in this case) as a an also affected project, sorry.
Hints welcome.

This issue was fixed in the openstack/nova 19.0.1 release.

Matt Riedemann (mriedem) on 2019-06-11
Changed in nova:
importance: Undecided → Low

Reviewed: https://review.opendev.org/662695
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c958ad8a68ea9a8ea465d9e3dd889248d9f42481
Submitter: Zuul
Branch: stable/rocky

commit c958ad8a68ea9a8ea465d9e3dd889248d9f42481
Author: Silvan Kaiser <email address hidden>
Date: Wed Nov 22 14:35:11 2017 +0100

    Fixes multi-registry config in Quobyte driver

    This closes a bug concerning multi-registry configurations
    for Quobyte volumes due to no longer using the is_mounted()
    method that failed in that case.
    Besides, this adds exception handling for the unmount call that
    is issued on trying to mount an already mounted volume.

    NOTE: The original commit also added a new feature (fs type
    based validation) which is omitted in this backport.

    Closes-Bug: #1737131

    Change-Id: Ia5a23ce1123a68608ee2ec6f2ac5dca02da67c59
    (cherry picked from commit 05a73c0f3a9f8edf9024f9870279bc6fb7bba2e7)
    (cherry picked from commit 656aa1cd40570154df606484d31616989b5296aa)

This issue was fixed in the openstack/nova 18.2.1 release.

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers