large number of interfaces cause slow puppet executions due to fact generation
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Alex Schultz |
Bug Description
When a compute node (or controller) has many interfaces (bridges/tun), during the application of updates the puppet processes start thrashing trying to generate the network puppet facts. This affects both facter 2 and facter 3 based systems due to the fact that multiple processes end up trying to execute ip commands to gather the networking facts. This can be worked around by caching the facts for the life of the container-puppet.py (or docker-puppet.py) executions. This affects all versions of TripleO since Ocata. Prior to Ocata, facter 2 is super slow when there are more than than 1000 interfaces on a host. At ~1500, it can take > 10 minutes to generate the facts.
for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done
$ time facter
facter2
real 9m51.817s
user 7m8.936s
sys 2m42.702s
facter3
real 0m2.954s
user 0m1.111s
sys 0m1.721s
$ time puppet facts
facter2
real 12m10.936s
user 8m16.478s
sys 3m54.138s
facter3
real 0m11.169s
user 0m5.522s
sys 0m4.002s
tags: | added: queens-backport-potential rocky-backport-potential |
Changed in tripleo: | |
milestone: | none → train-2 |
Changed in tripleo: | |
status: | Triaged → In Progress |
https:/ /review. opendev. org/#/c/ 669011/