2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments [req-4fe0c71b-fda7-4ff5-8873-051f29d441d1 1e6e904b469749f883c06fefaaf133c8 840055620f8446d2b298ea01fc6d8997 - default default] Unable to update the attachment.: tooz.coordination.ToozConnectionError: [Errno 32] Br
oken pipe (with errno EPIPE [32])
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments Traceback (most recent call last):
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 43, in _failure_translator
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments yield
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 70, in wrapper
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return func(*args, **kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 99, in _acquire
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments if self.coord.client.add(
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/pymemcache/client/base.py", line 1168, in add
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return client.add(key, value, expire=expire, noreply=noreply,
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/pymemcache/client/base.py", line 392, in add
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return self._store_cmd(b'add', {key: value}, expire, noreply,
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/pymemcache/client/base.py", line 935, in _store_cmd
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments self.sock.sendall(b''.join(cmds))
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/eventlet/greenio/base.py", line 402, in sendall
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments tail = self.send(data, flags)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/eventlet/greenio/base.py", line 396, in send
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return self._send_loop(self.fd.send, data, flags)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/eventlet/greenio/base.py", line 383, in _send_loop
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return send_method(data, *args)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments BrokenPipeError: [Errno 32] Broken pipe
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments The above exception was the direct cause of the following exception:
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments Traceback (most recent call last):
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/v3/attachments.py", line 238, in update
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments self.volume_api.attachment_update(context,
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "<decorator-gen-748>", line 2, in attachment_update
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/cinder/coordination.py", line 144, in _synchronized
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments with lock(blocking):
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/locking.py", line 29, in __enter__
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return self.lock.__enter__(*self.args, **self.kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/locking.py", line 49, in __enter__
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments acquired = self.acquire(*args, **kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 110, in acquire
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return _acquire()
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return self.call(f, *args, **kw)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments do = self.iter(retry_state=retry_state)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return fut.result()
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return self.__get_result()
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments raise self._exception
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments result = fn(*args, **kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 70, in wrapper
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments return func(*args, **kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments self.gen.throw(type, value, traceback)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/drivers/memcached.py", line 57, in _failure_translator
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments utils.raise_with_cause(coordination.ToozConnectionError,
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/tooz/utils.py", line 224, in raise_with_cause
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments File "<string>", line 3, in raise_from
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments tooz.coordination.ToozConnectionError: [Errno 32] Broken pipe (with errno EPIPE [32])
2021-02-22 01:35:13.345 13 ERROR cinder.api.v3.attachments
Looks like tooz does not retry if underlying socket is disconnected.
