2023-03-22 02:54:17.348 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 02:55:17.359 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 02:56:17.382 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 03:01:59.126 8701 ERROR oslo.messaging._drivers.impl_rabbit [-] [1108eda3-a6a6-4202-ad66-dad1ca6a90c8] AMQP server on 10.105.121.36:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
2023-03-22 03:01:59.127 8701 ERROR manila.share.manager [None req-3f669d7b-f1ac-4039-9311-b3ba7a8976cc a6c7ceede9af466089380c141428d439 ec101c020cd94de3bdc7ef514021e0e2 - - - -] Share instance f25f4db5-c4ac-424f-93cf-0be5712d4dda failed on creation.: manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume create, argdict={'vol_name': 'ceph-fs-sc1', 'sub_name': 'f25f4db5-c4ac-424f-93cf-0be5712d4dda', 'size': 1073741824, 'namespace_isolated': True, 'mode': '755', 'format': 'json'} - exception message: [errno -110] error calling ceph_mount.
2023-03-22 03:01:59.129 8701 WARNING manila.share.manager [None req-3f669d7b-f1ac-4039-9311-b3ba7a8976cc a6c7ceede9af466089380c141428d439 ec101c020cd94de3bdc7ef514021e0e2 - - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume create, argdict={'vol_name': 'ceph-fs-sc1', 'sub_name': 'f25f4db5-c4ac-424f-93cf-0be5712d4dda', 'size': 1073741824, 'namespace_isolated': True, 'mode': '755', 'format': 'json'} - exception message: [errno -110] error calling ceph_mount.
2023-03-22 03:01:59.134 8701 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2023-03-22 03:01:59.157 8701 INFO manila.message.api [None req-3f669d7b-f1ac-4039-9311-b3ba7a8976cc a6c7ceede9af466089380c141428d439 ec101c020cd94de3bdc7ef514021e0e2 - - - -] Creating message record for request_id = req-3f669d7b-f1ac-4039-9311-b3ba7a8976cc
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server [None req-3f669d7b-f1ac-4039-9311-b3ba7a8976cc a6c7ceede9af466089380c141428d439 ec101c020cd94de3bdc7ef514021e0e2 - - - -] Exception during message handling: manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume create, argdict={'vol_name': 'ceph-fs-sc1', 'sub_name': 'f25f4db5-c4ac-424f-93cf-0be5712d4dda', 'size': 1073741824, 'namespace_isolated': True, 'mode': '755', 'format': 'json'} - exception message: [errno -110] error calling ceph_mount.
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 203, in rados_command
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     raise rados.Error(outs, ret)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server rados.Error: [errno -110] error calling ceph_mount
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server 
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server 
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 220, in wrapped
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/utils.py", line 579, in wrapper
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 2123, in create_share_instance
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     raise self.value
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 2111, in create_share_instance
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     export_locations = self.driver.create_share(
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 473, in create_share
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     rados_command(self.rados_client, "fs subvolume create", argdict)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/cephfs/driver.py", line 215, in rados_command
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server     raise exception.ShareBackendException(msg)
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server manila.exception.ShareBackendException: json_command failed - prefix=fs subvolume create, argdict={'vol_name': 'ceph-fs-sc1', 'sub_name': 'f25f4db5-c4ac-424f-93cf-0be5712d4dda', 'size': 1073741824, 'namespace_isolated': True, 'mode': '755', 'format': 'json'} - exception message: [errno -110] error calling ceph_mount.
2023-03-22 03:01:59.183 8701 ERROR oslo_messaging.rpc.server 
2023-03-22 03:01:59.220 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 03:02:00.149 8701 INFO oslo.messaging._drivers.impl_rabbit [-] [1108eda3-a6a6-4202-ad66-dad1ca6a90c8] Reconnected to AMQP server on 10.105.121.36:5672 via [amqp] client with port 52176.
2023-03-22 03:02:59.189 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 03:03:59.186 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status
2023-03-22 03:04:59.192 8701 INFO manila.share.manager [None req-557732ac-3561-468f-8914-a5a81411bf20 - - - - - -] Updating share status