rabbitmq-server 3.5.7-1ubuntu0.16.04.2 security update dumped durable queues for autopkgtest.ubuntu.com

[Steve Langasek]

A rabbitmq-server security update was released today. This was auto-applied to the rabbitmq server that serves autopkgtest.ubuntu.com, and as a result, it appears that all the messages that were in the queue at the time were dropped.

According to Iain Lane, we are using durable queues in rabbitmq, so this should not have happened.

[Andreas Hasenack (ahasenack)]

Do you have access to the rabbit logs on autopkgtest.u.c and if yes, could you attach them to this bug please?

[Steve Langasek (vorlon)]

Attaching the logs that I could find. Let me know if there's something else I should look for.

Inconveniently, there don't appear to be log entries from around the time of the restart.

[Robie Basak (racb)]

Oh, perhaps not regression-update - rather a "when upgraded things go wrong" bug.

[Christian Ehrhardt  (paelzer)]

Since the logs had no content of when it happened and we had no other reports on it at all this is hard to action.

(Un?)Furtunately there also was no other upgrade of it since then.
The last still is: CVE-2016-9877 on Thu, 27 Jul 2017
Due to that we don't know if that would have happened again.

I tried to recreate this.
Based on [1] with this change for durable:
diff --git a/scripts/rabbitmq-server/receive.py b/scripts/rabbitmq-server/receive.py
index 80c7eb3e..cd5ce995 100755
--- a/scripts/rabbitmq-server/receive.py
+++ b/scripts/rabbitmq-server/receive.py
@@ -7,13 +7,13 @@ connection = pika.BlockingConnection(pika.ConnectionParameters(
 channel = connection.channel()

-channel.queue_declare(queue='hello')
+channel.queue_declare(queue='hellod', durable=True)

 def callback(ch, method, properties, body):
     print(" [x] Received %r" % body)

 channel.basic_consume(callback,
- queue='hello',
+ queue='hellod',
                       no_ack=True)

 #print(' [*] Waiting for messages. To exit press CTRL+C')
diff --git a/scripts/rabbitmq-server/send.py b/scripts/rabbitmq-server/send.py
index 56117d5a..8e4d4b7f 100755
--- a/scripts/rabbitmq-server/send.py
+++ b/scripts/rabbitmq-server/send.py
@@ -7,10 +7,10 @@ connection = pika.BlockingConnection(pika.ConnectionParameters(
 channel = connection.channel()

-channel.queue_declare(queue='hello')
+channel.queue_declare(queue='hellod', durable=True)

 channel.basic_publish(exchange='',
- routing_key='hello',
+ routing_key='hellod',
                       body='Hello World!')
 print(" [x] Sent 'Hello World!'")
 connection.close(

Test 1 - non durable queues
root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# ./receive.py
 [x] Received 'Hello World!'

Test 2 - non durable queues restarted
root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# systemctl restart rabbitmq-server
root@x:~# ./receive.py
(nothing - as expected)

Test 3 - durable queues [2]
root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# ./receive.py
 [x] Received 'Hello World!'
root@x:~# rabbitmqctl list_queues name durable messages
Listing queues ...
hello false 0
hellod true 0

Test 3 - durable queues restart
root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# rabbitmqctl list_queues name durable messages
Listing queues ...
hello false 0
hellod true 1
root@x:~# systemctl restart rabbitmq-server
root@x:~# ./receive.py
root@x:~# rabbitmqctl list_queues name durable messages
Listing queues ...
hellod true 0

So the messages are lost on a server restart - even on the durable queue.
I thought to understand that this should not happen.
I'm not entirely sure if this is a valid reproducer yet, but it should be with my limited rabbitmq-foo

[1]: https://git.launchpad.net/qa-regression-testing/tree/scripts/test-rabbitmq-server.py
[2]: https://www.rabbitmq.com/tutorials/tutorial-two-python.html

[Iain Lane (laney)]

I've since found out about delivery_mode=2 (persistence) and applied that when we insert messages from autopkgtest. I think it's possible that this is "not a bug" and we should have used this all along if we wanted messages to survive restarts?

https://www.rabbitmq.com/tutorials/amqp-concepts.html

"Durability of a queue does not make messages that are routed to that queue durable. If broker is taken down and then brought back up, durable queue will be re-declared during broker startup, however, only persistent messages will be recovered."

If you retry the test with delivery_mode=2, is everything okay?

[Christian Ehrhardt  (paelzer)]

Wit hthe help of the upstream IRC I found that the durable queue only makes the queue durable.
As outlined in my former [1].

TL;DR I need to persist the messages on publish.
In my example that is

 channel.basic_publish(exchange='',
- routing_key='hello',
- body='Hello World!')
+ routing_key='hellod',
+ body='Hello World!',
+ properties=pika.BasicProperties(
+ delivery_mode = 2, # make message persistent
+ ))
 print(" [x] Sent 'Hello World!'")
 connection.close()

With that it survives a restart

root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# ./send.py
 [x] Sent 'Hello World!'
root@x:~# rabbitmqctl list_queues name durable messages
Listing queues ...
hellod true 2
root@x:~# systemctl restart rabbitmq-server
root@x:~# rabbitmqctl list_queues name durable messages
Listing queues ...
hellod true 2
root@x:~# ./receive.py
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'

@Iain have we lost messages that were in persistent deliver mode or might that have been what happened?

[Christian Ehrhardt  (paelzer)]

Missed your reply before I finished mine ;-)
We came to the same conclusion, it is not a bug but needs to be used correctly.

That is good on autopkgtest.ubuntu.com now and we can close this bug.