Glusterfs create share from snapshot failed

Bug #1922075 reported by Lin PeiWen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Undecided
Lin PeiWen

Bug Description

I created a snapshot of the available state and failed to create an instance with this snapshot

Environment
===========
I'm using the Manila Master/Victoria version

Glusetrfs NFS drive

gluster version: glusterfs 8.2

Configs:

glusterfs_share_layout = layout_volume.GlusterfsVolumeMappedLayout
glusterfs_nfs_server_type = Gluster

Logs

2021-03-31 01:59:15.503 117291 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_8.0)
2021-03-31 01:59:15.576 117291 INFO paramiko.transport [-] Authentication (password) successful!
2021-03-31 01:59:21.764 117291 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_8.0)
2021-03-31 01:59:21.832 117291 INFO paramiko.transport [-] Authentication (password) successful!
2021-03-31 01:59:22.054 117291 ERROR manila.share.manager [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Share instance cd9eafc1-83a1-413f-8673-86b09f1dda96 failed on creation.: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.066 117291 WARNING manila.share.manager [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.130 117291 INFO manila.message.api [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Creating message record for request_id = req-30f36f55-f773-4a83-be74-27a0b0c49c41
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server [req-30f36f55-f773-4a83-be74-27a0b0c49c41 63a560d401fb41adac0333ab6e5aecdd 964f97eb3c044495874b08c5de2207e8 - - -] Exception during message handling: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
Unexpected error while running command.
Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
Exit code: 1
Stdout: ''
Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 198, in _gluster_call
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return gluster_execf(*(('gluster',) + args), **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/ganesha/utils.py", line 79, in __call__
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 586, in ssh_execute
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Exit code: 1
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stdout: ''
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 193, in wrapped
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 1954, in create_share_instance
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server exception=e)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server raise value
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/manager.py", line 1896, in create_share_instance
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server share_server=share_server, parent_share=parent_share_dict)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/layout.py", line 171, in create_share_from_snapshot
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return self.layout.create_share_from_snapshot(context, share, *a, **kw)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/layout_volume.py", line 508, in create_share_from_snapshot
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server {'share': snapshot['share_instance'], 'manager': old_gmgr})
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/__init__.py", line 110, in _setup_via_manager
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server gluster_manager.set_vol_option(*setting)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 57, in wrapper
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 327, in set_vol_option
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server 'volume', args[0], self.volume, *args[1], error_policy=policy)
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/manila/share/drivers/glusterfs/common.py", line 214, in _gluster_call
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server 'details': exc})
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject *' failed with details as follows:
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Unexpected error while running command.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Command: gluster volume set manila-cd9eafc1-83a1-413f-8673-86b09f1dda96 nfs.rpc-auth-reject '*'
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Exit code: 1
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stdout: ''
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Commit failed on localhost. Please check the log file for more details.\n'.
2021-03-31 01:59:22.220 117291 ERROR oslo_messaging.rpc.server
2021-03-31 01:59:44.271 117291 INFO manila.share.manager [req-2c022031-572f-44f4-b102-162e50a4acb4 - - - - -] Updating share status

Lin PeiWen (linpeiwen)
Changed in manila:
assignee: nobody → Lin PeiWen (linpeiwen)
Lin PeiWen (linpeiwen)
description: updated
Revision history for this message
Tom Barron (tpb) wrote :
Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/784189
Committed: https://opendev.org/openstack/manila/commit/63e255248b7072c92c084a0cea269a27bd47f813
Submitter: "Zuul (22348)"
Branch: master

commit 63e255248b7072c92c084a0cea269a27bd47f813
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/788608

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/manila/+/788608
Committed: https://opendev.org/openstack/manila/commit/3635ade670cce08525c946eb9029940d86fcafa8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3635ade670cce08525c946eb9029940d86fcafa8
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/manila/+/792167

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/manila/+/792167
Committed: https://opendev.org/openstack/manila/commit/a04e13196c125b7d748ffdacc226b97104498cdd
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a04e13196c125b7d748ffdacc226b97104498cdd
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)
    (cherry picked from commit 3635ade670cce08525c946eb9029940d86fcafa8)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/manila/+/792277

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/manila/+/792277
Committed: https://opendev.org/openstack/manila/commit/f40a428bdbf0b820ed44a3784a7ac4b2f71aeb98
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f40a428bdbf0b820ed44a3784a7ac4b2f71aeb98
Author: LinPeiWen <email address hidden>
Date: Thu Apr 1 02:56:36 2021 +0000

    [Glusterfs] Fix create share from snapshot failed

    1、After performing a snapshot clone of the glusterfs vol,
      the status of the vol is'Created', and the parameter
      "gluster volume set nfs.rpc-auth-reject'*'" is required for
      the vol in the'Started' state.
    2、The cloned volume needs to activate the snapshot,
      if the snapshot is already activated, you need to skip the activation step

    Closes-Bug: #1922075
    Change-Id: I304bf59b3f8c0d5b847078a5752bac8ac4f21690
    (cherry picked from commit 63e255248b7072c92c084a0cea269a27bd47f813)
    (cherry picked from commit 3635ade670cce08525c946eb9029940d86fcafa8)
    (cherry picked from commit a04e13196c125b7d748ffdacc226b97104498cdd)

tags: added: in-stable-ussuri
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers