ceilometer-agent-central doesn't get paused/stopped

[Eduardo Sousa]

I'm porting the Amulet tests to the Zaza framework and while doing the migration found that not all services are paused when the pause action is ran.

When running the pause_resume test, the ceilometer-agent-central doesn't get stopped and for this reason the test fails due to a timeout while waiting for the service to stop. Currently I'm testing against the bionic-stein target.

To confirm that it was this service that was blocking the test I stopped the service while the test was waiting for it to stop, using systemctl stop ceilometer-agent-central.service inside the ceilometer unit. This made the test continue and finish successfully.

The test is test_901_pause_resume and can be found here:
https://github.com/openstack-charmers/zaza-openstack-tests/blob/83ea8d94a1a0984851578bedb2b49c3acc7f6b02/zaza/openstack/charm_tests/ceilometer/tests.py#L110

I'm attaching to this bug the crashdump concerning the model under test.

These are the services running while the test is waiting before timeout:

root@juju-096e49-zaza-458993dff86b-13:/home/ubuntu# ps aux | grep ceilometer
root 3694 0.0 0.1 21768 3500 ? Ss 17:53 0:00 bash /lib/systemd/system/jujud-unit-ceilometer-0/exec-start.sh
root 3700 0.3 4.6 745760 94204 ? Sl 17:53 0:09 /var/lib/juju/tools/unit-ceilometer-0/jujud unit --data-dir /var/lib/juju --unit-name ceilometer/0 --debug
ceilome+ 23897 0.0 3.8 454980 78360 ? Ssl 18:10 0:01 ceilometer-polling: master process [/usr/bin/ceilometer-polling --config-file=/etc/ceilometer/ceilometer.conf --polling-namespaces central --log-file=/var/log/ceilometer/ceilometer-agent-central.log]
ceilome+ 23922 0.0 3.3 614224 68140 ? Sl 18:10 0:00 ceilometer-polling: AgentManager worker(0)
root 28076 0.0 0.0 14856 1076 pts/0 S+ 18:35 0:00 grep --color=auto ceilometer

[Eduardo Sousa (edsousa)]

This problem also affects trusty-mitaka for the ceilometer-collector service.

[Alex Kavanagh (ajkavanagh)]

What did the original amulet test do regarding these services for the ubuntu-openstack combinations you've mentioned.

[Eduardo Sousa (edsousa)]

It ran the pause action for that charm and verified if it shows the pause message or not. It didn't verify if the services inside it were running or not.

The test can be checked here: https://github.com/openstack/charm-ceilometer/blob/7998b10d93b865f104524d61b97efcf4421e7c19/tests/basic_deployment.py#L667

In the change for Zaza, we need to specify the services to see if they are running or not.

[Corey Bryant (corey.bryant)]

Hi Eduardo,

The charm code removes ceilometer-agent-central from the list of services for all of the releases that we currently support (trusty-mitaka and above) [1], so is it possible that the test is improperly expecting ceilometer-agent-central to be paused/resumed?

[1] https://review.opendev.org/#/c/435540/

[Eduardo Sousa (edsousa)]

Hi Corey,

Yes, it is possible. There is still begs the question: does the charm need to handle the service or not? If it does, then test is right and the charm needs to be fixed. If it doesn't, the test is wrong and the test should be fixed.

[Felipe Reyes (freyes)]

this issue was taken care of by PRs https://github.com/openstack-charmers/zaza-openstack-tests/pull/80/files and https://github.com/openstack-charmers/zaza-openstack-tests/pull/163