nova-compute memory leak

[Takenori MATSUMOTO]

[Impact]

 * nova-compute service which uses python-libvirt continuously grows the heap size

[Test Case]

  * using valgrind tool can detect memory leak, valgrind --leak-check=full /usr/bin/python ./d.py
    after patch applied, should be 0 bytes definitely lost.

==11243== LEAK SUMMARY:
==11243== definitely lost: 0 bytes in 0 blocks
==11243== indirectly lost: 0 bytes in 0 blocks
==11243== possibly lost: 1,064 bytes in 2 blocks
==11243== still reachable: 984,453 bytes in 1,757 blocks
==11243== suppressed: 0 bytes in 0 blocks

[Regression Potential]

 * patch backported from upstream, and has been fixed in utopic.

[Other Info]

 * This bug fix has been fixed upstream http://libvirt.org/git/?p=libvirt-python.git;a=commitdiff;h=4acfb169400497da2a82a849dc8aaa65f88ac6d1

We found a symptom like memory leak of nova-compute.
Even though there are no activities of OpenStack, the heap of
nova-compute continuously grows as approximately 100KB/sec.

After running for a week the virtual space size of nova-compute
almost reached 2.5GB. The initial size was around 1.6GB.

Is this bug at upstream? And if this is bug, is there workaround or solution to avoid memory leak.
The following is detail information

root# while true;
> do
> date
> pmap -x 8506 | head
> sleep 60
> done
Tue Sep 2 08:32:28 UTC 2014
8506: /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1832 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 368 252 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000000cdc000 19948 19708 19708 rw--- [ anon ] *** heap
00007f47b4000000 132 132 132 rw--- [ anon ]
00007f47b4021000 65404 0 0 ----- [ anon ]
00007f47bb7ff000 4 0 0 ----- [ anon ]
Tue Sep 2 08:33:28 UTC 2014
8506: /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1832 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 368 252 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000000cdc000 20080 19840 19840 rw--- [ anon ]
00007f47b4000000 132 132 132 rw--- [ anon ]
00007f47b4021000 65404 0 0 ----- [ anon ]
00007f47bb7ff000 4 0 0 ----- [ anon ]
Tue Sep 2 08:34:28 UTC 2014
8506: /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1832 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 368 252 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000000cdc000 20080 19848 19848 rw--- [ anon ]
00007f47b4000000 132 132 132 rw--- [ anon ]
00007f47b4021000 65404 0 0 ----- [ anon ]
00007f47bb7ff000 4 0 0 ----- [ anon ]
Tue Sep 2 08:35:28 UTC 2014
8506: /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1832 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 368 252 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000000cdc000 20232 19992 19992 rw--- [ anon ]
00007f47b4000000 132 132 132 rw--- [ anon ]
00007f47b4021000 65404 0 0 ----- [ anon ]
00007f47bb7ff000 4 0 0 ----- [ anon ]

- Software versions
ii nova-common 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - common files
ii nova-compute 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node libvirt support

ii python 2.7.5-5ubuntu3 amd64 interactive high-level object-oriented language (default version)

- nova-compute.conf
[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=kvm

- nova.conf
[DEFAULT]
scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,CoreFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,DifferentHostFilter,SameHostFilter
dhcpbridge_flagfile = /etc/nova/nova.conf
dhcpbridge = /usr/bin/nova-dhcpbridge
logdir = /var/log/nova
state_path = /var/lib/nova
lock_path = /var/lock/nova
force_dhcp_release = True
iscsi_helper = tgtadm
libvirt_use_virtio_for_bridges = True
connection_type = libvirt
root_helper = sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose = True
ec2_private_dns_show_ip = True
api_paste_config = /etc/nova/api-paste.ini
volumes_path = /var/lib/nova/volumes
enabled_apis = ec2,osapi_compute,metadata
auth_strategy = keystone
rpc_backend = rabbit
rabbit_host = xxx.xxx.xxx.xxx
rabbit_password = *********
my_ip = yyy.yyy.yyy.yyy
vnc_enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = yyy.yyy.yyy.yyy
novncproxy_base_url = http://zzz.zzz.zzz.zzz:6080/vnc_auto.html
glance_host = xxx.xxx.xxx.xxx
network_api_class = nova.network.api.API
security_group_api = nova
firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
allow_same_net_traffic = False
multi_host = True
send_arp_for_ha = True
share_dhcp_address = True
public_interface = bond0
network_size = 256
block_device_allocate_retries = 360
block_device_allocate_retries_interval = 5
network_manager = nova.network.manager.FlatDHCPManager
flat_network_bridge = br100
flat_interface = bond1
libvirt_images_type = rbd
libvirt_images_rbd_pool = volumes
libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

[database]
connection = *********

[keystone_authtoken]
auth_uri = http://xxx.xxx.xxx.xxx:5000
auth_host = xxx.xxx.xxx.xxx
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = *********

[osapi_v3]
enabled = True

[Takashi Aosawa (t-aosawa)]

At least python-libvirt, which nova-compute uses, seems to leak memory when handling XML data. Find the file d.py attached which reproduces a similar problem.

ii python-libvirt 1.2.2-0ubuntu1 amd64 libvirt Python bindings

valgrind can detect "definitely lost".

# ps -ef|grep d.py
root 25578 17888 24 14:40 pts/0 00:00:03 python ./d.py 100000
root 25594 25581 0 14:41 pts/2 00:00:00 grep --color=auto d.py
# pmap -x 25578|head
25578: python ./d.py 100000
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1448 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 260 176 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000001a82000 72328 72296 72296 rw--- [ anon ]
00007f0851a3a000 28 8 0 r-x-- libffi.so.6.0.1
00007f0851a41000 2044 0 0 ----- libffi.so.6.0.1
00007f0851c40000 4 4 4 r---- libffi.so.6.0.1
# pmap -x 25578|head
25578: python ./d.py 100000
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1448 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 260 176 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000001a82000 87900 87848 87848 rw--- [ anon ]
00007f0851a3a000 28 8 0 r-x-- libffi.so.6.0.1
00007f0851a41000 2044 0 0 ----- libffi.so.6.0.1
00007f0851c40000 4 4 4 r---- libffi.so.6.0.1
# pmap -x 25578|head
25578: python ./d.py 100000
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 2804 1448 0 r-x-- python2.7
00000000008bc000 4 4 4 r---- python2.7
00000000008bd000 468 260 176 rw--- python2.7
0000000000932000 72 64 64 rw--- [ anon ]
0000000001a82000 120676 120636 120636 rw--- [ anon ]
00007f0851a3a000 28 8 0 r-x-- libffi.so.6.0.1
00007f0851a41000 2044 0 0 ----- libffi.so.6.0.1
00007f0851c40000 4 4 4 r---- libffi.so.6.0.1

# valgrind --leak-check=full /usr/bin/python ./d.py
...
==25610== 12,682 bytes in 4 blocks are definitely lost in loss record 227 of 236
==25610== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25610== by 0x5183844: xdr_string (xdr.c:788)
==25610== by 0x67E1A8D: ??? (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x67E4CF8: ??? (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x67FDA61: virNetMessageDecodePayload (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x67F2AEB: virNetClientProgramCall (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x67C7025: ??? (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x67D0174: ??? (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x6787907: virDomainGetXMLDesc (in /usr/lib/libvirt.so.0.1002.2)
==25610== by 0x642E41D: libvirt_virDomainGetXMLDesc (in /usr/lib/python2.7/dist-packages/libvirtmod.so)
==25610== by 0x52C6D4: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==25610== by 0x55C593: PyEval_EvalCodeEx (...

[Takashi Aosawa (t-aosawa)]

python-libvirt memory leak will be fixed on upstream.

https://www.redhat.com/archives/libvir-list/2014-September/msg00751.html

I hope the fix will be merged soon.

[Yaguang Tang (heut2008)]

This has been fixed in upstream http://libvirt.org/git/?p=libvirt-python.git;a=commitdiff;h=4acfb169400497da2a82a849dc8aaa65f88ac6d1

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt-python - 1.2.8-0ubuntu2

---------------
libvirt-python (1.2.8-0ubuntu2) utopic; urgency=medium

  * debian/patches/libvirt-memleak.patch: Fix memory leak.
    (LP: #1365250)
 -- Chuck Short <email address hidden> Fri, 19 Sep 2014 08:51:50 -0400

[Chris J Arges (arges)]

Hello Takenori, or anyone else affected,

Accepted libvirt-python into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt-python/1.2.2-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Yaguang Tang (heut2008)]

I have tested using valgrind , the new version fixes memory leak issue.

 python /tmp/d.py 100

==8200== LEAK SUMMARY:
==8200== definitely lost: 546,282 bytes in 99 blocks
==8200== indirectly lost: 0 bytes in 0 blocks
==8200== possibly lost: 6,582 bytes in 3 blocks
==8200== still reachable: 839,804 bytes in 1,221 blocks
==8200== suppressed: 0 bytes in 0 blocks
==8200== Rerun with --leak-check=full to see details of leaked memory
==8200==
==8200== For counts of detected and suppressed errors, rerun with: -v
==8200== Use --track-origins=yes to see where uninitialised values come from
==8200== ERROR SUMMARY: 639 errors from 36 contexts (suppressed: 0 from 0)
ubuntu@moon:~$ dpkg -l |grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.2 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.2 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu1 amd64 libvirt Python bindings

 python /tmp/d.py 100

==10116== LEAK SUMMARY:
==10116== definitely lost: 0 bytes in 0 blocks
==10116== indirectly lost: 0 bytes in 0 blocks
==10116== possibly lost: 1,064 bytes in 2 blocks
==10116== still reachable: 839,926 bytes in 1,221 blocks
==10116== suppressed: 0 bytes in 0 blocks
==10116== Rerun with --leak-check=full to see details of leaked memory
==10116==
==10116== For counts of detected and suppressed errors, rerun with: -v
==10116== Use --track-origins=yes to see where uninitialised values come from
==10116== ERROR SUMMARY: 706 errors from 36 contexts (suppressed: 0 from 0)
ubuntu@moon:~$ dpkg -l |grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.4 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.4 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt-python - 1.2.2-0ubuntu2

---------------
libvirt-python (1.2.2-0ubuntu2) trusty-proposed; urgency=medium

  * debian/patches/libvirt-memleak.patch: Fix memory leak.
    (LP: #1365250)
 -- Serge Hallyn <email address hidden> Tue, 23 Sep 2014 12:45:38 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for libvirt-python has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.