Contrail 2.21-14 : Very high CPU for contrail-control and collector processes

[Deepak Jeyaraman]

Noticing very high/excessive CPU utilixation for contrail-control and collector processes. This is the case even when the system is idle (read not creating stacks/nova instances or deleting them)

Tasks: 838 total, 3 running, 834 sleeping, 0 stopped, 1 zombie
%Cpu(s): 39.3 us, 12.1 sy, 1.8 ni, 46.1 id, 0.4 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 26404764+total, 26331049+used, 737148 free, 77072 buffers
KiB Swap: 26832486+total, 1299424 used, 26702544+free. 13073200 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 2907 contrail 20 0 51.445g 0.039t 5680 R 2047 16.0 14369:24 contrail-collec
 2823 contrail 20 0 0.152t 0.149t 7936 S 1020 60.6 16223:19 contrail-contro
 2838 root 20 0 0.150t 0.017t 1.070g S 626.5 7.0 6065:06 java
 2846 root 20 0 1341940 572044 5808 R 101.4 0.2 520:44.76 contrail-api
 2839 heat 20 0 323764 137384 4936 S 29.8 0.1 50:36.82 heat-engine
40039 root 20 0 127792 47320 2852 S 17.9 0.0 0:00.52 puppet
  362 root 20 0 0 0 0 S 11.9 0.0 1:54.43 kswapd0

There were 150 stacks/instances during this time.

Setup:

1 control / 11 computes- enough resource to handle the scale.

root@ccra-06:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-14 14
contrail-config 2.21.1-14 14
contrail-config-openstack 2.21.1-14 14
contrail-control 2.21.1-14 14
contrail-dns 2.21.1-14 14
contrail-f5 2.21.1-14 14