| 2020-09-24 03:55:06 |
Michael Chapman |
description |
In a local test environment I misconfigured the pools.yaml to point at bind servers that don't exist, then ran tempest. The result was about 1500 rndc processes that don't appear to ever disappear, causing significant load on the system. rndc doesn't have a timeout option that I can see in the man page, so it probably makes sense to add one via oslo processutils.
The environment is deployed using tripleo, so the processes are all in the designate-worker container:
[root@controller-1 ~]# podman stats designate_worker
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
18103f368b4e designate_worker 1.73% 19.13GB / 33.56GB 57.02% -- / -- 4.882GB / 8.536MB 2076 |
In a local test environment I misconfigured the pools.yaml to point at bind servers that don't exist, then ran tempest. The result was about 1500 rndc processes that don't appear to ever disappear, causing significant load on the system.
The environment is deployed using tripleo, so the processes are all in the designate-worker container:
[root@controller-1 ~]# podman stats designate_worker
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
18103f368b4e designate_worker 1.73% 19.13GB / 33.56GB 57.02% -- / -- 4.882GB / 8.536MB 2076
Steps to reproduce:
1. Deploy devstack with the bind backend.
2. Edit /etc/designate/pools.yaml so rndc_host doesn't point at a bind server
3. Update the pools: designate-manage pool update --file /etc/designate/pools.yaml
4. Run tempest: tox -e all-plugin -- designate
This is despite designate worker only having 2 configured workers:
/etc/designate.conf:
[service:worker]
workers = 2
It might make sense to maintain a task queue for the rndc commands so that only a limited number can be active at any given time.
rndc doesn't have a timeout option that I can see in the man page, so it might makes sense to add one via oslo processutils. I think the built in timeout is about 30 seconds. |
|
