OpenStack Shared File Systems Service (Manila)

Glusterfs create share from snapshot failed

Bug #1922075 reported by Lin PeiWen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Lin PeiWen

Bug Description

I created a snapshot of the available state and failed to create an instance with this snapshot

Environment
===========
I'm using the Manila Master/Victoria version

Glusetrfs NFS drive

gluster version: glusterfs 8.2

Configs:

glusterfs_share_layout = layout_volume.GlusterfsVolumeMappedLayout
glusterfs_nfs_server_type = Gluster

Logs

2021-03-31 01:59:15.503 117291 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_8.0)
2021-03-31 01:59:15.576 117291 INFO paramiko.transport [-] Authentication (password) successful!
2021-03-31 01:59:21.764 117291 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_8.0)
2021-03-31 01:59:21.832 117291 INFO paramiko.transport [-] Authentication (password) successful!
2021-03-31 01:59:22.054 117291 ERROR manila.share.manager [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Share instance cd9eafc1-83a1-413f-8673-86b09f1dda96 failed on creation.: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.066 117291 WARNING manila.share.manager [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.130 117291 INFO manila.message.api [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Creating message record for request_id = req-30f36f55-f773-4a83-be74-27a0b0c49c41
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Exception during message handling: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
Unexpected error while running command.
Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
Exit code: 1
Stdout: ''
Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 198, in _gluster_call
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return gluster_execf(*(('gluster',) + args), **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 79, in __call__
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 586, in ssh_execute
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Exit code: 1
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stdout: ''
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 193, in wrapped
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 1954, in create_share_instance
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server exception=e)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server raise value
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 1896, in create_share_instance
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server share_server=share_server, parent_share=parent_share_dict)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/layout.py", line 171, in create_share_from_snapshot
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return self.layout.create_share_from_snapshot(context, share, *a, **kw)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/layout_volume.py", line 508, in create_share_from_snapshot
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server {'share': snapshot['share_instance'], 'manager': old_gmgr})
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/__init__.py", line 110, in _setup_via_manager
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server gluster_manager.set_vol_option(*setting)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 57, in wrapper
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 327, in set_vol_option
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server 'volume', args[0], self.volume, *args[1], error_policy=policy)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 214, in _gluster_call
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server 'details': exc})
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Unexpected error while running command.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Exit code: 1
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stdout: ''
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:44.271 117291 INFO manila.share.manager [req-2c022031-572f-44f4-b102-162e50a4acb4 - - - - -] Updating share status

See original description
Lin PeiWen (linpeiwen)
Changed in manila:
assignee: nobody → Lin PeiWen (linpeiwen)
Lin PeiWen (linpeiwen)
description: updated
Revision history for this message
Tom Barron (tpb) wrote :

https://review.opendev.org/c/openstack/manila/+/784189 proposed as fix

OpenStack Infra (hudson-openstack)
Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/784189
Committed: https://opendev.org/openstack/manila/commit/63e255248b7072c92c084a0cea269a27bd47f813
Submitter: "Zuul (22348)"
Branch: master

commit 63e255248b7072c92c084a0cea269a27bd47f813
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/788608

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/manila/+/788608
Committed: https://opendev.org/openstack/manila/commit/3635ade670cce08525c946eb9029940d86fcafa8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3635ade670cce08525c946eb9029940d86fcafa8
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/manila/+/792167

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/manila/+/792167
Committed: https://opendev.org/openstack/manila/commit/a04e13196c125b7d748ffdacc226b97104498cdd
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a04e13196c125b7d748ffdacc226b97104498cdd
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)
    (cherry picked from commit 3635ade670cce08525c946eb9029940d86fcafa8)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/manila/+/792277

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/manila/+/792277
Committed: https://opendev.org/openstack/manila/commit/f40a428bdbf0b820ed44a3784a7ac4b2f71aeb98
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f40a428bdbf0b820ed44a3784a7ac4b2f71aeb98
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)
    (cherry picked from commit 3635ade670cce08525c946eb9029940d86fcafa8)
    (cherry picked from commit a04e13196c125b7d748ffdacc226b97104498cdd)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 10.1.0

This issue was fixed in the openstack/manila 10.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 13.0.0.0rc1

This issue was fixed in the openstack/manila 13.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 12.1.0

This issue was fixed in the openstack/manila 12.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 11.1.0

This issue was fixed in the openstack/manila 11.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.