Seeing this in the gate:
http://logs.openstack.org/30/560930/1/gate/nova-next/dd44138/logs/screen-c-api.txt.gz?level=TRACE#_Apr_13_00_50_45_642761
Apr 13 00:50:45.642761 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments [req-a4a450bd-abb6-4812-a70b-b49edad3e469 req-1ae13a6c-f325-4a29-a0e3-b0a193b71869 tempest-ServerActionsTestJSON-1769432331 tempest-ServerActionsTestJSON-1769432331] Unable to update the attachment.: MessagingTimeout: Timed out waiting for a reply to message ID 90fe8ccba0a044f7a3a9770048c6d1b2
Apr 13 00:50:45.643045 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Apr 13 00:50:45.643251 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/opt/stack/new/cinder/cinder/api/v3/attachments.py", line 228, in update
Apr 13 00:50:45.643460 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments connector))
Apr 13 00:50:45.643673 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/opt/stack/new/cinder/cinder/volume/api.py", line 2162, in attachment_update
Apr 13 00:50:45.643876 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments attachment_ref.id))
Apr 13 00:50:45.644079 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/opt/stack/new/cinder/cinder/rpc.py", line 187, in _wrapper
Apr 13 00:50:45.644280 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments return f(self, *args, **kwargs)
Apr 13 00:50:45.644481 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/opt/stack/new/cinder/cinder/volume/rpcapi.py", line 442, in attachment_update
Apr 13 00:50:45.644694 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments attachment_id=attachment_id)
Apr 13 00:50:45.644897 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
Apr 13 00:50:45.645108 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments retry=self.retry)
Apr 13 00:50:45.645341 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
Apr 13 00:50:45.645596 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments timeout=timeout, retry=retry)
Apr 13 00:50:45.645820 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
Apr 13 00:50:45.646043 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments retry=retry)
Apr 13 00:50:45.646260 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send
Apr 13 00:50:45.646487 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments result = self._waiter.wait(msg_id, timeout)
Apr 13 00:50:45.646704 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait
Apr 13 00:50:45.646929 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments message = self.waiters.get(msg_id, timeout=timeout)
Apr 13 00:50:45.647139 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get
Apr 13 00:50:45.647351 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments 'to message ID %s' % msg_id)
Apr 13 00:50:45.647658 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments MessagingTimeout: Timed out waiting for a reply to message ID 90fe8ccba0a044f7a3a9770048c6d1b2
Apr 13 00:50:45.647884 ubuntu-xenial-inap-mtl01-0003494782 <email address hidden>[826]: ERROR cinder.api.v3.attachments
It's intermittent.
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unable%20to%20update%20the%20attachment.%3A%20MessagingTimeout%5C%22%20AND%20tags%3A%5C%22screen-c-api.txt%5C%22&from=7d
20 hits in 7 days, check and gate, master and stable/queens.
Hmm, not seeing corresponding logs on the c-vol side. I wonder if we are hitting rabbitmq issues?