Description of problem:
Since NSS_SDB_USE_CACHE=no is not set before calling curl in the container health checks, the dentry cache on controller nodes grows continually. On controllers with a larger amount of RAM, this can lead to soft lockups once memory pressure forces a reclamation of the extraneous cache entries.
See RHBZ 1044666 for background.
Version-Release number of selected component (if applicable):
13.0
How reproducible:
Run the following systemtap script on a containerized OSP13 controller:
probe kernel.function("d_alloc").return { log(reverse_path_walk($return)) }
Observe many repeated calls referencing lib/docker/overlay2/<some_id>/diff/etc/pki/nssdb/.<some_number>_dOeSnotExist_.db
Steps to Reproduce:
1. Deploy an OSP13 environment with containerized control plane
2. Run the systemtap script above for 1 minute
3. Observe many calls for nonexistent NSS DB files
4. Run the following to add NSS_SDB_USE_CACHE=no to the healthcheck function:
docker ps -q | xargs -I {} docker exec -u root {} sed -i '/^healthcheck_curl/a \ \ export NSS_SDB_USE_CACHE=no' /usr/share/openstack-tripleo-common/healthcheck/common.sh
5. Re-run the systemtap script
6. Observe a large reduction (90%+) in dentry cache calls over 1 minute
Actual results:
[root@ctl01 ~]# stap -o test1.out -T 60 dentry.stap
[root@ctl01 ~]# wc -l test1.out
186526 test1.out
[root@ctl01 ~]# grep dOeSnotExist test1.out | wc -l
158649
Expected results:
[root@ctl01 ~]# stap -o test2.out -T 60 dentry.stap
[root@ctl01 ~]# wc -l test2.out
17544 test2.out
[root@ctl01 ~]# grep dOeSnotExist test2.out | wc -l
0
Fix proposed to branch: master
Review: https:/