As far as I can see from the environment (the logs from which were provided at the comment #14) there are issues with unconsumed AMQP messages exist:
For example, in logs it is clear that heat-engine at node-17,18 recovered AMQP connection at 2014-12-05T16:41:34.500079+00:00 info: 2014-12-05 16:41:34.495
But 'stack' OSTF checks are failing, queues notifications.error, info grow with unconsumed messages on each OSTF run
(See report.txt, report2.txt files attached):
<email address hidden> notifications.error false false [{"x-ha-policy","all"}] ha-all 43 0 43
<email address hidden> notifications.error false false [{"x-ha-policy","all"}] ha-all 49 0 49
<email address hidden> notifications.info false false [{"x-ha-policy","all"}] ha-all 0 19 19
ceilometer-agent-notification at node-17,18 AMQP issues - known issue https://bugs.launchpad.net/mos/+bug/1380800
and there are many unconsumed messages at ceilometer.collector.metering queue, and it grows on each OSTF run:
<email address hidden> ceilometer.collector.metering false false [{"x-ha-policy","all"}] ha-all 2903 722 3625
<email address hidden> ceilometer.collector.metering false false [{"x-ha-policy","all"}] ha-all 3053 722 3775
As far as I can see from the environment (the logs from which were provided at the comment #14) there are issues with unconsumed AMQP messages exist:
For example, in logs it is clear that heat-engine at node-17,18 recovered AMQP connection at 2014-12- 05T16:41: 34.500079+ 00:00 info: 2014-12-05 16:41:34.495 error, info grow with unconsumed messages on each OSTF run policy" ,"all"} ] ha-all 43 0 43 policy" ,"all"} ] ha-all 49 0 49 policy" ,"all"} ] ha-all 0 19 19
But 'stack' OSTF checks are failing, queues notifications.
(See report.txt, report2.txt files attached):
<email address hidden> notifications.error false false [{"x-ha-
<email address hidden> notifications.error false false [{"x-ha-
<email address hidden> notifications.info false false [{"x-ha-
ceilometer- agent-notificat ion at node-17,18 AMQP issues - known issue https:/ /bugs.launchpad .net/mos/ +bug/1380800 collector. metering queue, and it grows on each OSTF run: collector. metering false false [{"x-ha- policy" ,"all"} ] ha-all 2903 722 3625 collector. metering false false [{"x-ha- policy" ,"all"} ] ha-all 3053 722 3775
and there are many unconsumed messages at ceilometer.
<email address hidden> ceilometer.
<email address hidden> ceilometer.