Activity log for bug #1835044

Date Who What changed Old value New value Message
2019-07-02 11:02:26 Candido Campos Rivas bug added bug
2019-07-02 12:08:01 Candido Campos Rivas description Description of problem: Memory leak in privsep-helper(neutron agents) when create/destroy action are executed. Version-Release number of selected component (if applicable): How reproducible: Rocky and Queens are affected Steps to Reproduce: 1. Deploy Osp 14 2.create the next scripts: (overcloud) [stack@undercloud-0 ~]$ cat create2.sh set -x ips=(0 10.0.0.215 10.0.0.249 10.0.0.223 10.0.0.222 10.0.0.218 10.0.0.247 10.0.0.210 10.0.0.220 10.0.0.246 10.0.0.213 10.0.0.224 10.0.0.212 10.0.0.217 10.0.0.221 10.0.0.216) ips=(0 10.0.0.220 10.0.0.216 10.0.0.235 10.0.0.232 10.0.0.245 10.0.0.226 10.0.0.217 10.0.0.211 10.0.0.221 10.0.0.230 10.0.0.248 10.0.0.228 10.0.0.223 10.0.0.212 10.0.0.225) openstack network create net_$1 openstack subnet create --network net_$1 --dns-nameserver 10.0.0.1 --gateway 10.$1.0.1 --subnet-range 10.$1.0.0/16 net_$1 openstack router create router_$1 openstack router add subnet router_$1 net_$1 #openstack router set router_$1 --external-gateway public #openstack server create --flavor cirros --image cirros --nic net-id=net$1 --security-group test --key-name mykey vm$1 #openstack server add floating ip vm$1 ${ips[$1]} #ping ${ips[$1]} -c 10 (overcloud) [stack@undercloud-0 ~]$ cat delete2.sh #openstack server delete vm$1 openstack router remove subnet router_$1 net_$1 openstack network delete net_$1 openstack router delete router_$1 3. Execute: for j in $(seq 1 10); do for i in $(seq 1 10) ; do ./create2.sh $i ; done ; for i in $(seq 1 10) ; do ./delete2.sh $i ; done ;echo "####LOOP $j ####" ;done Chech the memory usage of the privsep-helper in the controllers: [root@controller-1 heat-admin]# top -c -p 125052 -p 278912 top - 22:37:22 up 9 days, 4:59, 1 user, load average: 4.45, 3.71, 3.77 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.8 us, 4.1 sy, 0.0 ni, 75.2 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st KiB Mem : 32779936 total, 3150768 free, 16209200 used, 13419968 buff/cache KiB Swap: 0 total, 0 free, 0 used. 15146444 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 125052 root 20 0 2497576 2.2g 2468 S 0.0 7.2 76:06.44 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+ 278912 root 20 0 1048908 897004 2468 S 0.0 2.7 13:55.34 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+ The important colunm is RES Actual results: Memory usage increases in every loop. Expected results: Memory usage should be stable Additional info: Leak is in pyroute2--> necessary update version to 0.5.2.x More info : https://bugzilla.redhat.com/show_bug.cgi?id=1723597 Description of problem: Memory leak in privsep-helper(neutron agents) when create/destroy action are executed. Version-Release number of selected component (if applicable): How reproducible: Queens is affected Steps to Reproduce: 1. Deploy Queens 2.create the next scripts: (overcloud) [stack@undercloud-0 ~]$ cat create2.sh set -x ips=(0 10.0.0.215 10.0.0.249 10.0.0.223 10.0.0.222 10.0.0.218 10.0.0.247 10.0.0.210 10.0.0.220 10.0.0.246 10.0.0.213 10.0.0.224 10.0.0.212 10.0.0.217 10.0.0.221 10.0.0.216) ips=(0 10.0.0.220 10.0.0.216 10.0.0.235 10.0.0.232 10.0.0.245 10.0.0.226 10.0.0.217 10.0.0.211 10.0.0.221 10.0.0.230 10.0.0.248 10.0.0.228 10.0.0.223 10.0.0.212 10.0.0.225) openstack network create net_$1 openstack subnet create --network net_$1 --dns-nameserver 10.0.0.1 --gateway 10.$1.0.1 --subnet-range 10.$1.0.0/16 net_$1 openstack router create router_$1 openstack router add subnet router_$1 net_$1 #openstack router set router_$1 --external-gateway public #openstack server create --flavor cirros --image cirros --nic net-id=net$1 --security-group test --key-name mykey vm$1 #openstack server add floating ip vm$1 ${ips[$1]} #ping ${ips[$1]} -c 10 (overcloud) [stack@undercloud-0 ~]$ cat delete2.sh #openstack server delete vm$1 openstack router remove subnet router_$1 net_$1 openstack network delete net_$1 openstack router delete router_$1 3. Execute: for j in $(seq 1 10); do for i in $(seq 1 10) ; do ./create2.sh $i ; done ; for i in $(seq 1 10) ; do ./delete2.sh $i ; done ;echo "####LOOP $j ####" ;done Chech the memory usage of the privsep-helper in the controllers: [root@controller-1 heat-admin]# top -c -p 125052 -p 278912 top - 22:37:22 up 9 days, 4:59, 1 user, load average: 4.45, 3.71, 3.77 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.8 us, 4.1 sy, 0.0 ni, 75.2 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st KiB Mem : 32779936 total, 3150768 free, 16209200 used, 13419968 buff/cache KiB Swap: 0 total, 0 free, 0 used. 15146444 avail Mem     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  125052 root 20 0 2497576 2.2g 2468 S 0.0 7.2 76:06.44 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+  278912 root 20 0 1048908 897004 2468 S 0.0 2.7 13:55.34 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+ The important colunm is RES Actual results: Memory usage increases in every loop. Expected results: Memory usage should be stable Additional info: Leak is in pyroute2--> necessary update version to 0.5.2.x
2019-07-02 13:04:15 Rodolfo Alonso summary [Rocky]Memory leak in privsep-helper(neutron agents) [Queens] Memory leak in pyroute2 0.4.21
2019-07-02 15:23:01 Bernard Cafarelli neutron: status New In Progress
2019-07-02 15:23:05 Bernard Cafarelli neutron: importance Undecided High
2019-07-02 15:23:20 Bernard Cafarelli neutron: assignee Rodolfo Alonso (rodolfo-alonso-hernandez)
2019-07-02 15:24:43 Bernard Cafarelli tags loadimpact
2019-07-02 15:49:40 Bernard Cafarelli bug added subscriber Bernard Cafarelli
2019-07-08 10:15:16 Slawek Kaplonski neutron: status In Progress Won't Fix