sahara-all leaks memory

[Francois Deppierraz]

OpenStack kilo deployment using the Ubuntu Cloud Archive under Ubuntu 14.04 and running Sahara with the direct infrastructure engine (as the heat one didn't worked in this setup).

This deployment was done with the official puppet-sahara modules and there are currently only 2 clusters running (4 and 16 nodes). Many test clusters have been launched and deleted before though.

The 'sahara-all' process has been running for 21 days and now consumes around 5.5 GB of RAM.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
$ apt-cache policy sahara-common
sahara-common:
  Installed: 2015.1.0-0ubuntu1~cloud0
  Candidate: 2015.1.0-0ubuntu1~cloud0
  Version table:
 *** 2015.1.0-0ubuntu1~cloud0 0
       1000 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/kilo/main amd64 Packages
        100 /var/lib/dpkg/status
$ top -b -n 1 | grep sahara-all
 1057 sahara 20 0 5876080 5.506g 5376 S 0.0 17.5 552:40.64 sahara-all
$

While trying to understand what all this memory is being used for, I've discovered many instance of SSH private key in the process memory. It seems that ~3 GB of memory (1.7 mio instance at 1.7 KB each) is being used for these keys, this may be a hint on where the memory leaks actually comes from.

$ sudo gcore 1057
[...]
Saved corefile core.1057
$ strings core.1057 | sort | uniq -c | sort -rn | head
1771726 -----END RSA PRIVATE KEY-----
[...]
$ echo $[1771490*1679]
2974331710
$

[Evgeny Sikachev (esikachev)]

please add method for deploy openstack to description

[Sergey Reshetnyak (sreshetniak)]

same problem with sahara-engine

[Evgeny Sikachev (esikachev)]

bug reproduced on stable/liberty

[Francois Deppierraz (francois-ctrlaltdel)]

Hi Vitaly, Do you have any pointer to commits which solved this issue?

Thanks!

[Vitalii Gridnev (vgridnev)]

Francois, we didn't came up with a solution of this problem, actually. So, I did reset of assignee for this bug and removed liberty series for awhile (until valid solution will be found)