Comment 25 for bug 1463433

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Re: [Bug 1463433] Re: [shaker] test failing due to multiple "Timed out waiting for reply to ID" events logged by Oslo.messaging after rabbitmq recovered from partitioning and kept running with AMQP publish got blocked because virt memory got exhausted at rabbit node

Alexander, Bogdan,

Is this the cause of the issue?
https://github.com/rabbitmq/rabbitmq-server/issues/62

On Tue, Jun 16, 2015 at 9:39 AM, Alexander Nevenchannyy
<email address hidden> wrote:
> This is mail thread looks like our problem
> https://groups.google.com/forum/#!topic/rabbitmq-users/Ja0iyfF0Szw
>
> --
> You received this bug notification because you are a member of MOS Oslo,
> which is a bug assignee.
> https://bugs.launchpad.net/bugs/1463433
>
> Title:
> [shaker] test failing due to multiple "Timed out waiting for reply to
> ID" events logged by Oslo.messaging after rabbitmq recovered from
> partitioning and kept running with AMQP publish got blocked because
> virt memory got exhausted at rabbit node
>
> Status in Fuel: OpenStack installer that works:
> In Progress
> Status in Fuel for OpenStack 6.1.x series:
> In Progress
> Status in Fuel for OpenStack 7.0.x series:
> In Progress
> Status in Mirantis OpenStack:
> Confirmed
>
> Bug description:
> During Shaker test (http://pyshaker.readthedocs.org/en/latest/index.html) we have found that ksoftirqd keep a lot of CPU:
> http://paste.openstack.org/show/277774/
> atop SHIFT+P ksoftirqd:
> http://paste.openstack.org/show/277891/
>
> At the time rabbitmq on this controller node (node-49) was down:
> from node-1
> =INFO REPORT==== 8-Jun-2015::23:48:07 ===
> rabbit on node 'rabbit@node-49' down
>
> from node-44
> =INFO REPORT==== 8-Jun-2015::23:46:55 ===
> rabbit on node 'rabbit@node-49' down
>
> Configuration:
> Baremetal,Centos,fedora_kernel,IBP,HA, Neutron-vlan,Ceph-all,Nova-debug,Nova-quotas, 6.1-521
> Controllers:3 Computes:47
>
> net_ticktime parameter has been added:
> http://paste.openstack.org/show/278020/
>
> api: '1.0'
> astute_sha: 7766818f079881e2dbeedb34e1f67e517ed7d479
> auth_required: true
> build_id: 2015-06-08_06-13-27
> build_number: '521'
> feature_groups:
> - mirantis
> fuel-library_sha: f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d
> fuel-ostf_sha: 7c938648a246e0311d05e2372ff43ef1eb2e2761
> fuelmain_sha: bcc909ffc5dd5156ba54cae348b6a07c1b607b24
> nailgun_sha: 4340d55c19029394cd5610b0e0f56d6cb8cb661b
> openstack_version: 2014.2.2-6.1
> production: docker
> python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
> release: '6.1'
>
> Diagnostic Snapshot: http://mos-scale-share.mirantis.com/fuel-
> snapshot-2015-06-09_09-55-58.tar.xz
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1463433/+subscriptions

--
Davanum Srinivas :: https://twitter.com/dims