hello all, I run a queen version openstack with python2-oslo-messaging-5.35.5, rabbitmq 3.8.7, centos7. I found that after some other error happend, oslo-messaging may report a lot of AMQP server on 10.39.176.17:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe like below:
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6499, in _get_local_gb_info
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager info = LibvirtDriver._get_rbd_driver().get_pool_info()
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 368, in get_pool_info
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager with RADOSClient(self) as client:
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 102, in __init__
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager self.cluster, self.ioctx = driver._connect_to_rados(pool)
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 133, in _connect_to_rados
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager client.connect()
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager File "rados.pyx", line 893, in rados.Rados.connect (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/build/src/pybind/rados/pyrex/rados.c:9834)
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager PermissionError: [errno 1] error connecting to the cluster
2021-08-10 20:43:56.002 19626 ERROR nova.compute.manager
2021-08-10 20:43:56.007 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [cef69dca-f065-4b5e-8129-b42032d23717] AMQP server on 10.39.176.17:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.008 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [cf287c2d-0135-405a-9d83-784fc3cc1729] AMQP server on 10.39.176.9:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.008 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.010 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.010 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.011 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.012 19626 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.017 19626 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8a95348-8155-49d8-bc77-9d7c466cbea9] AMQP server on 10.39.176.2:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe
2021-08-10 20:43:56.018 19626 ERROR oslo.messaging._drivers.impl_rabbit [req-27e11dbc-6beb-4543-8768-df7a35fb7ed2 - - - - -] [d4886cfb-5645-409a-86b0-cf8702bedcd1] AMQP server on 10.39.176.9:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe
2021-08-10 20:43:57.041 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [b8a95348-8155-49d8-bc77-9d7c466cbea9] Reconnected to AMQP server on 10.39.176.2:5672 via [amqp] client with port 42738.
2021-08-10 20:43:57.045 19626 INFO oslo.messaging._drivers.impl_rabbit [req-27e11dbc-6beb-4543-8768-df7a35fb7ed2 - - - - -] [d4886cfb-5645-409a-86b0-cf8702bedcd1] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 49842.
2021-08-10 20:43:57.047 19626 INFO oslo.messaging._drivers.impl_rabbit [-] [cf287c2d-0135-405a-9d83-784fc3cc1729] Reconnected to AMQP server on 10.39.176.9:5672 via [amqp] client with port 49838.
At the same time, the rabbitmq's log hasn't any error info.
What does Broken pipe mean and why there is so many Broken pipe error?
I check the log again and find when oslo.messaging report Broken pipe, at the same time , mq report client unexpectedly closed TCP connection, the log time is exactly the same, I can't find out which log is first and which log is last:
nova-compute.log:
2021-08-10 20:43:57.047 19626 INFO oslo.messaging.
2021-08-10 20:43:57.084 19626 INFO oslo.messaging.
2021-08-10 20:44:18.665 19626 ERROR oslo.messaging.
2021-08-10 20:44:18.676 19626 ERROR oslo.messaging.
2021-08-10 20:44:18.679 19626 ERROR oslo.messaging.
the mq node 10.39.176.2's log:
2021-08-10 20:44:11.118 [info] <0.19291.3925> accepting AMQP connection <0.19291.3925> (10.39.
--
2021-08-10 20:44:18.531 [info] <0.19797.4111> Connection <0.19797.4111> (10.39.176.32:42654 -> 10.39.176.2:5672) has a client-provided name: nova-compute:
2021-08-10 20:44:18.536 [info] <0.19797.4111> connection <0.19797.4111> (10.39.176.32:42654 -> 10.39.176.2:5672 - nova-compute:
2021-08-10 20:44:18.665 [warning] <0.28284.4169> closing AMQP connection <0.28284.4169> (10.39.176.22:42948 -> 10.39.176.2:5672 - nova-compute:
client unexpectedly closed TCP connection
2021-08-10 20:44:18.676 [warning] <0.11227.3703> closing AMQP connection <0.11227.3703> (10.39.176.22:42942 -> 10.39.176.2:5672 - nova-compute:
client unexpectedly closed TCP connection
2021-08-10 20:44:18.758 [info] <0.2478.3652> accepting AMQP connection <0.2478.3652> (10.39.176.22:43216 -> 10.39.176.2:5672)
at the exactly same time 2021-08-10 20:44:18.665 and 2021-08-10 20:44:18.676, client report Broken pipe, mq report client unexpectedly closed TCP connection, why?