oslo.messaging version: Train (however the code in question is still in master)
Messaging Server: Rabbit MQ
Prior to https://review.opendev.org/c/openstack/oslo.messaging/+/581417 we had a common message id cache mechanism that worked for both regular RPC messages and Notifications (because the common class was used). It prevented duplicate messages from processing.
However, now the cache is working only for Notifications. The code to check duplicates is still there but the cache is filled only in NotificationAMQPIncomingMessage class and not in AMQPIncomingMessage. This leads to the duplicate messages being processed sometimes, which may be not expected by the application-level code. It is possible if the connection gets broken while the server did not receive our ACK.
In particular, we observe duplicate operations in Trove DBaaS. Although the operation itself is idempotent, when the message is processed the second time, the calling code already got the reply to the first message and thinks that the agent is not busy anymore. The subsequent operations may clash with the processing of the second message. This reproduces fairly reliably on our CI.
After some strace and tcpdump, the whole flow looks like this:
1. We receive a new message and ACK it.
2. However, the connection to the Rabbit MQ server gets broken and the server cannot get our ACK.
3. The kernel retries the sending of ACK packets and the Rabbit MQ server keeps re-sending the message to us.
4. We process the message and try to send the reply.
5. The ensure machinery detects that the connection is broken and re-establishes it.
6. Our ACK and our reply finally reach the server, but at the same time we get the same message a second time.
7. The caller side of an RPC call gets the response and thinks that we are free now and ready to do another operation while we are actually busy processing the same message a second time.
8. Our processing conflicts with the subsequent operations sent to us.
One of the solutions could be to activate the message id caching for regular RPC messages, as a useful feature that once was there. However, we should not reject the duplicate message because it is likely already ACK'ed. The reject is happening if we let the DuplicateMessag