When creating a cloned volume, cinder-vlume is disconnected by RabbitMQ.
2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
...
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit
2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
We observe "missed heartbeats from client, timeout: 60s" in RabbitMQ log and it seems that this disconnection is caused by missing heatbreats to client when flatten is caused.
This issue was initially reported in downstream bug and reproduced with Queens. https://bugzilla.redhat.com/show_bug.cgi?id=1885734
However the call of flatten in create_cloned_volume is not yet updated even in master.
When creating a cloned volume, cinder-vlume is disconnected by RabbitMQ.
2020-10-06 15:19:05.697 55 DEBUG oslo.messaging. _drivers. impl_rabbit [-] [80610839- 4914-4df7- b1ab-d5d0efcc2f ae] Received recoverable error from kombu: on_error /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ impl_rabbit. py:765 _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 7/site- packages/ kombu/connectio n.py", line 494, in _ensured _drivers. impl_rabbit File "/usr/lib/ python2. 7/site- packages/ eventlet/ greenio/ base.py" , line 371, in _send_loop _drivers. impl_rabbit return send_method(data, *args) _drivers. impl_rabbit error: [Errno 104] Connection reset by peer _drivers. impl_rabbit _drivers. impl_rabbit [-] [80610839- 4914-4df7- b1ab-d5d0efcc2f ae] AMQP server on overcloud- controller- 0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
...
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
2020-10-06 15:19:05.697 55 ERROR oslo.messaging.
2020-10-06 15:19:05.698 55 ERROR oslo.messaging.
We observe "missed heartbeats from client, timeout: 60s" in RabbitMQ log and it seems that this disconnection is caused by missing heatbreats to client when flatten is caused.
The same issue was reported for volume- from-snapshot[ 1] and the fix for the issue made flatten call use tpool.Proxy[2]. /bugs.launchpad .net/cinder/ +bug/1658037 /review. opendev. org/#/c/ 423184/ 2/cinder/ volume/ drivers/ rbd.py
[1] https:/
[2] https:/
However the flatten call in create_ cloned_ volume is still executed in the same thread[3]. We need the same fix for this flatten call. /github. com/openstack/ cinder/ blob/6ad1ab0c72 98f0d2647ec42e2 f7bb101af135af7 /cinder/ volume/ drivers/ rbd.py# L743
[3] https:/
This issue was initially reported in downstream bug and reproduced with Queens. /bugzilla. redhat. com/show_ bug.cgi? id=1885734 cloned_ volume is not yet updated even in master.
https:/
However the call of flatten in create_