[hirsute] NeutronApiTest.test_901_pause_resume fails on waiting for services to stop

[Aurelien Lourot]

Hirsute-only for now: consistently happening on hirsute-wallaby, but not focal-wallaby nor groovy-victoria. Visible in the gate:
https://review.opendev.org/c/openstack/charm-neutron-api/+/794522
https://openstack-ci-reports.ubuntu.com/artifacts/e3c/794522/2/check/hirsute-wallaby/e3cde0c/job-output.txt

ERROR: test_901_pause_resume (zaza.openstack.charm_tests.neutron.tests.NeutronApiTest)
Run pause and resume tests.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 372, in test_901_pause_resume
    pgrep_full=pgrep_full):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/test_utils.py", line 514, in pause_resume
    pgrep_full=pgrep_full)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
    return run(_run_it())
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
    return task.result()
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
    return await f(*args, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 1537, in async_block_until_service_status
    await async_block_until(_check_service, timeout=timeout)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-neutron-api/.tox/func-target/lib/python3.6/site-packages/zaza/model.py", line 1605, in async_block_until
    await asyncio.wait_for(_block(), timeout, loop=loop)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 362, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

[Hemanth Nakkina (hemanth-n)]

Additional info on failure:

From Hirsute, apache2 service installation resulted in start of apache2 and apache-htcacheclean service. And during the test, apache-htcacheclean service is not stopped and so pgrep -o -f apache2 resulted in returning process id of apache-htcacheclean service. This resulted in test case blocked assuming services are not stopped.

# ps -aef | grep apache2
www-data 8847 1 0 13:14 ? 00:00:00 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n
root 37837 34961 0 13:37 pts/0 00:00:00 grep --color=auto apache2

I am not sure if apache-htcacheclean service should be started during initialisation from hirsute and so raised a bug on apache2 [1].

[1] https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1942745

[Corey Bryant (corey.bryant)]

Thanks for triaging this Hemanth.

[Billy Olsen (billy-olsen)]

The htcacheclean process runs as the www-data user, so I think this can likely be resolved by expanding the test to be for pgrep -u root -o -f apache2 to filter by process run for root user. Alternatively, the charm can be made aware of the apache-htcacheclean service as well.

[Corey Bryant (corey.bryant)]

I ended up just using the full binary paths so that the pgreps don't match unexpected processes. Let me know what you think. https://github.com/openstack-charmers/zaza-openstack-tests/pull/686

For example:
ubuntu@juju-e9cc66-zaza-906d385905ca-4:~$ pgrep -o -f /usr/bin/neutron-server
40738
ubuntu@juju-e9cc66-zaza-906d385905ca-4:~$ pgrep -o -f /usr/sbin/apache2
40424
ubuntu@juju-e9cc66-zaza-906d385905ca-4:~$ pgrep -o -f /usr/sbin/haproxy
40605
ubuntu@juju-e9cc66-zaza-906d385905ca-4:~$ ps aux | grep -E '40738|40424|40605'
root 40424 0.0 0.2 6860 4880 ? Ss 21:14 0:00 /usr/sbin/apache2 -k start
root 40605 0.0 0.6 95416 12456 ? Ss 21:14 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
neutron 40738 2.2 6.6 146460 134912 ? Ss 21:14 0:03 /usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api/+/822202

[Corey Bryant (corey.bryant)]

I think the change above might be ready to land. The tests weren't all successful at once but were successful at various times.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api/+/822202
Committed: https://opendev.org/openstack/charm-neutron-api/commit/59ce3d96cec1c61fb31b67bb0ba53e02d08e79ad
Submitter: "Zuul (22348)"
Branch: master

commit 59ce3d96cec1c61fb31b67bb0ba53e02d08e79ad
Author: Corey Bryant <email address hidden>
Date: Fri Dec 17 16:31:02 2021 -0500

    Re-enable voting for wallaby tests

    The pgrep behavior in test_901_pause_resume has been fixed in
    zaza-openstack-tests.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/686
    Closes-Bug: #1933338
    Change-Id: I6e5a1384939e25f9bec4a019be6788432cea6b34