Potential memory leak in Ceilometer compute agent

[Fabrizio Soppelsa]

Observed on IceHouse 2014.1.3.

The value of RSS increased by nearly 500% from 51996 at start of the test to 303788 when the test was ended for the ceilometer-agent-compute process on the compute nodes.
This arose after some Rally benchmarking scenarios (Nova and Cinder - create/destroy instances/storages) ran for several days.

See also https://bugs.launchpad.net/ceilometer/+bug/1369836

[Ivan Berezovskiy (iberezovskiy)]

We are starting to research this issue

[Fabrizio Soppelsa (fsoppelsa)]

Thank you Ivan. Please contact me for any information request (logs are not available now, but if the problem will be reproduced, should be available within 10 days or so).

[Ilya Tyaptin (ityaptin)]

Hi, Fabrizio. Our researches show what memory leak is caused by novaclient. It's been registered in bug https://bugs.launchpad.net/python-novaclient/+bug/1433491. We'll provide a fix for this bug in MOS 5.1.2 novaclient as soon as possible.

[OSCI Robot (oscirobot)]

RPM package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.18.1, package release == fuel5.1.2.mira2.git.3e40940.58e13bf

Changeset: https://review.fuel-infra.org/5270
project: openstack/python-novaclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: patchset-created

Files placed on repository:
python-novaclient-2.18.1-fuel5.1.2.mira2.git.3e40940.58e13bf.noarch.rpm
python-novaclient-doc-2.18.1-fuel5.1.2.mira2.git.3e40940.58e13bf.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-5270/centos

[OSCI Robot (oscirobot)]

DEB package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.18.1, package release == fuel5.1.2~mira2+git.3e40940.58e13bf

Changeset: https://review.fuel-infra.org/5270
project: openstack/python-novaclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: patchset-created

Files placed on repository:
python-novaclient_2.18.1-fuel5.1.2~mira2+git.3e40940.58e13bf_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-5270/ubuntu

[OSCI Robot (oscirobot)]

RPM package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.18.1, package release == fuel5.1.2.mira2

Changeset: https://review.fuel-infra.org/5270
project: openstack/python-novaclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: change-merged

Files placed on repository:
python-novaclient-2.18.1-fuel5.1.2.mira2.noarch.rpm
python-novaclient-doc-2.18.1-fuel5.1.2.mira2.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

[OSCI Robot (oscirobot)]

DEB package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.18.1, package release == fuel5.1.2~mira2

Changeset: https://review.fuel-infra.org/5270
project: openstack/python-novaclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: change-merged

Files placed on repository:
python-novaclient_2.18.1-fuel5.1.2~mira2_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.20.0, package release == fuel6.0.1.mira18.git.d7ef367.bd15f98

Changeset: https://review.fuel-infra.org/5807
project: openstack/python-novaclient
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: patchset-created

Files placed on repository:
python-novaclient-2.20.0-fuel6.0.1.mira18.git.d7ef367.bd15f98.noarch.rpm
python-novaclient-doc-2.20.0-fuel6.0.1.mira18.git.d7ef367.bd15f98.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-5807/centos

[OSCI Robot (oscirobot)]

DEB package python-novaclient has been built for project openstack/python-novaclient
Package version == 2.20.0, package release == fuel6.0.1~mira18+git.d7ef367.bd15f98

Changeset: https://review.fuel-infra.org/5807
project: openstack/python-novaclient
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Avoid saving request times than timing is False
status: patchset-created

Files placed on repository:
python-novaclient_2.20.0-fuel6.0.1~mira18+git.d7ef367.bd15f98_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5807/ubuntu

[Ilya Tyaptin (ityaptin)]

Hi, Fabrizio.

This bug fixes for MOS 6.0.2 and MOS 6.1 have been send to review and they wait review from Nova team and approve from our QA engineers.

[Fabrizio Soppelsa (fsoppelsa)]

Thank you Ilya. How about the 5.1.2 version?

[Ilya Tyaptin (ityaptin)]

Hi, Fabrizio.

Fix for MOS 5.1.2 have been already merged. The packages with this commit have signature *mira2* and currently available only in internal package repository [1]. In the nearest future packages in public mirror will be updated too.

[1]
RPM:
http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos/noarch/python-novaclient-2.18.1-fuel5.1.2.mira2.noarch.rpm.

DEB:
http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu/all/python-novaclient_2.18.1-fuel5.1.2~mira2_all.deb

[Vasilios Tzanoudakis (vtzanoudakis)]

Dear Team,

When this package will be merged on http://fuel-repository.mirantis.com ?

thank you

[OSCI Robot (oscirobot)]

Reviewed: https://review.fuel-infra.org/5718
Committed: https://review.fuel-infra.org/gitweb?p=openstack/python-novaclient.git;a=commitdiff;h=bceb7080b892448c909a37bc8c75e7ac0bb3ba78
Submitter: mos-infra-ci
Branch: openstack-ci/fuel-6.1/2014.2

commit bceb7080b892448c909a37bc8c75e7ac0bb3ba78
Author: Ilya Tyaptin <email address hidden>

Avoid saving request times than timing is False

Otherwise it makes provide unexpected memory leak.
This approach is taken from openstack/common/apiclient.

Closes-bug: 1439278
Change-Id: I3df89764cabdfa864805085d20e7ec676acf10b2

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/6434

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6434
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: d1296fefa9443355c32f5db3b31e17d1a48960dd
Author: Alex Ermolov <email address hidden>
Date: Wed May 6 10:26:22 2015

Avoid saving request times than timing is False

Otherwise it makes provide unexpected memory leak.
This approach is taken from openstack/common/apiclient.

Closes-bug: 1439278
Change-Id: I3df89764cabdfa864805085d20e7ec676acf10b2

[Vitaly Gusev (vgusev)]

Verified on ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "395"
  build_id: "2015-05-08_11-08-49"
  nailgun_sha: "46f55c293e4540d31bcaa6ca3fba77235fb27537"
  python-fuelclient_sha: "af6c9c3799b9ec107bcdc6dbf035cafc034526ce"
  astute_sha: "6a4dcd11c67af2917815f3678fb594c7412a4c97"
  fuel-library_sha: "f385d6a58298c702f8d4f14c452dcffdc0b1e2a3"
  fuel-ostf_sha: "740ded337bb2a8a9b3d505026652512257375c01"
  fuelmain_sha: "3eca5e8f7ca6a83faff5feeca92c21cff31c0af1"

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/8308

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8308
Reason: In favor of https://review.fuel-infra.org/9775