Hello,
I checked rabbitmq mirrored queue feature, it is great work! BTW, when I checked current implementation, I found a case that openstack services receives same messages twice. Here is a detail description.
In current implementation, rpc.call() flows below().
1 ) Producer(sender) sends a message using TopicProducer.
2 ) Consumer(receiver) accepts the message using TopicConsumer.
3) Consumer dispatch appropriate methods and got return value.
4) Consumer returns the result which is got at 3) using DirectProducer.
5) Producer receives the result using DirectConsumer.
( and Producer sends ack ... the result is deleted from rabbitmq)
6) Consumer returns the message "ending=True", that is a sign rpc.call() is now completed.
7) Producer receives the message "ending=True" using DirectConsumer.
( and Producer sends ack ... the message "ending=True" is deleted from rabbitmq)
8) finally, Producer sends ack for the message described at 2).
Once openstack services receive any messages, it has to send ack for it, otherwise, message is not deleted from rabbitmq.
Now we think about what if rabbitmq is down between 3) and 8) above. Openstack services have alredy handle the messages, but after reconnecting to slave rabbitmq, same message still remains. Then, the same messages have to be dispatched again. it causes an error.
I tested using below patch. Setting time.sleep() around 2) and 3) above, and while openstack services are sleeping, kill rabbitmq.
> diff --git a/nova/openstack/common/rpc/impl_kombu.py b/nova/openstack/common/rpc/impl_kombu.py
> index 3544fc..de41b33 100644
> --- a/nova/openstack/common/rpc/impl_kombu.py
> +++ b/nova/openstack/common/rpc/impl_kombu.py
> @@ -187,13 +187.29 @@ class ConsumerBase(object):
>
> def _callback(raw_message):
> message = self.channel.message_to_python(raw_message)
> try:
> + time.sleep(10)
> msg = rpc_common.deserialize_msg(message.payload)
Proposed fix is here: https:/