Comment 5 for bug 1940866

Revision history for this message
yatin (yatinkarel) wrote :

<< Not sure the root cause of the issue yet.

Had noticed the failures on https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805029 which failed at multiple recheck. All the failures are caused by too much memory consumption leading to oom and pcs timeouts which resulted into tempest or some other failure. Inspecting atop log i see ceph process are consuming much memory and seems https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/803493/ has triggered the issue.

Symptoms and timings lead to the patch, Can verify by the revert of https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/803493/(corresponding depends-on tripleo-common patches may also need to be reverted) and running multiple scenarion001 job as job is failing randomly. If revert confirms the issue then can go with the revert and someone from ceph can check why ceph process consuming much memory.