Rabbit server runs out of memory
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Charm Test Infra |
Fix Released
|
High
|
Liam Young |
Bug Description
It looks like the tests (stable_to_next_ha mainly) does not allocate machines for the rabbit servers with enough RAM. This mainly shows itself as a rabbit charm update status charm hook failing.
From a recent mojo run of stable to next ha on xenial:
/var/log/
2020-02-13 16:52:25 DEBUG update-status Traceback (most recent call last):
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status rabbit.
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status assess_
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status services=
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status state, message, lambda: charm_func(
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status charm_state, charm_message = charm_func_
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status state, message, lambda: charm_func(
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status if not clustered():
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status res = func(*args, **kwargs)
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status if len(running_
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status res = func(*args, **kwargs)
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status return nodes(get_
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status res = func(*args, **kwargs)
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status out = rabbitmqctl_
2020-02-13 16:52:25 DEBUG update-status File "/var/lib/
2020-02-13 16:52:25 DEBUG update-status .check_output(cmd, stderr=
2020-02-13 16:52:25 DEBUG update-status File "/usr/lib/
2020-02-13 16:52:25 DEBUG update-status **kwargs).stdout
2020-02-13 16:52:25 DEBUG update-status File "/usr/lib/
2020-02-13 16:52:25 DEBUG update-status output=stdout, stderr=stderr)
2020-02-13 16:52:25 DEBUG update-status subprocess.
2020-02-13 16:52:25 ERROR juju.worker.
syslog:
Feb 13 16:47:20 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
Feb 13 16:47:21 juju-996685-
/<email address hidden>
=INFO REPORT==== 13-Feb-
vm_memory_
=WARNING REPORT==== 13-Feb-
memory resource limit alarm set on node 'rabbit@
*******
*** Publishers will be blocked until this alarm clears ***
*******
=ERROR REPORT==== 13-Feb-
Discarding message {'$gen_
=ERROR REPORT==== 13-Feb-
closing AMQP connection <0.13201.3> (172.17.
Missed heartbeats from client, timeout: 60s
# ls -l /var/log/
-rw-r--r-- 1 rabbitmq rabbitmq 74 Feb 13 16:47 /var/log/
# cat /var/log/
eheap_alloc: Cannot allocate 762886488 bytes of memory (of type "heap").
Changed in charm-test-infra: | |
status: | New → Confirmed |
importance: | Undecided → High |
assignee: | nobody → Liam Young (gnuoy) |
Changed in charm-test-infra: | |
status: | Confirmed → Fix Released |