Error in /usr/lib/ocf/resource.d/fuel/rabbitmq-server when much memory used

[Mikhail Chernik]

When running NovaServers.resize_server Rally test following error occurred:

Oct 05 07:39:08 [25956] node-203.domain.tld crmd: notice: process_lrm_event: node-203.domain.tld-p_rabbitmq-server_monitor_30000:329 [ /usr/lib/ocf/resource.d/fuel/rabbitmq-server: line 1274: 2.15997e+09 / 1048576 : syntax erro
r: invalid arithmetic operator (error token is ".15997e+09 / 1048576 ")\n ]

Environment:
3 controllers, 198 computes

api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'

[Davanum Srinivas (DIMS) (dims-v)]

Mikhail,

Can you please make a local change to grab the value of this variable "queues"?
https://github.com/stackforge/fuel-library/blob/master/files/fuel-ha-utils/ocf/rabbitmq#L1258

We do not seem to be logging this anywhere.

Thanks,
Dims

[Davanum Srinivas (DIMS) (dims-v)]

Mikhail,

Can you please try this? http://paste.openstack.org/show/475762/

awk prints out in exponent/scientific notation which bash division does not like at all. so we should do the conversion from bytes to mb using awk itself.

Thanks,
Dims

[Davanum Srinivas (DIMS) (dims-v)]

Fix in upstream: https://review.openstack.org/233179

[Alexey Khivin (akhivin)]

https://review.openstack.org/#/c/233179/

[Polina Petriuk (ppetriuk)]

This issue will exist in MOS6.1-mu-3, since https://bugs.launchpad.net/fuel/+bug/1479815 pretty much backported the changes to OCF rabbit script from MOS7.0. And #1479815 is targeted for MOS6.1-mu-3.

Also confirmed that issue exists on MOS6.1 with patches from 1479815.

[Vitaly Sedelnik (vsedelnik)]

Fix Committed for 6.1, there CR is https://review.openstack.org/#/c/238421/

[Mikhail Chernik (mchernik)]

Verified on 7.0-mu1, fixed.

[Mikhail Chernik (mchernik)]

Verified on MOS 8.0 ISO 482, fixed.