The local cache mechanism of compute resource may calse cause memory leak

Bug #1594285 reported by Liusheng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
Liusheng

Bug Description

The ceilometer-polling service support a local mechanism to cache the instances info from Nova API of local node. this mechanism is used to reduce the workload of Nova API, in every polling cycle, the ceilometer-polling will query the instances from Nova API by passing a "change-since" parameter and filtered by local host. By this, only the updated instance since last polling will be retrieved and updated the local cache. when an instance has been migrated (cold or live migrated) from one node to another, the result of Nova API query won't include this this instance info but is has gone, the legacy instance info in the local cache of this local node will never be cleaned unless the service being restarted.

Liusheng (liusheng)
Changed in ceilometer:
assignee: nobody → Liusheng (liusheng)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/333129

Changed in ceilometer:
status: New → In Progress
Revision history for this message
Zhengwei Gao (multi-task) wrote :

In case of: When booting instance and schedulered to host, ceilometer-compute will collect sample for the instance's meter. The instance may error before instance's state active, the instance's host will be changed, ceilometer-compute won't know instances's state forever and still trying to collect instance meter.

Changed in ceilometer:
assignee: Liusheng (liusheng) → gordon chung (chungg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/333129
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=3ae997e3d6c9648292969366d25362297474b815
Submitter: Jenkins
Branch: master

commit 3ae997e3d6c9648292969366d25362297474b815
Author: liusheng <email address hidden>
Date: Tue Jun 21 09:19:22 2016 +0800

    Add support of refereshing the resource info in local cache

    The local resource caching mechanism may cause legacy resource info
    resided in the local cache and never be cleaned, uless the service
    being restarted. e.g. an instance has been migrated to another host, the
    instance info will reside in the cache on the ceilometer-polling
    service running on the old compute host.

    Change-Id: I55f12ea9f6f6104de6a38b98ad935d458a7b0cc8
    Closes-Bug: #1594285
    Closes-Bug: #1656166

Changed in ceilometer:
status: In Progress → Fix Released
gordon chung (chungg)
Changed in ceilometer:
assignee: gordon chung (chungg) → Liusheng (liusheng)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 8.0.0

This issue was fixed in the openstack/ceilometer 8.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/467012

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/newton)

Reviewed: https://review.openstack.org/467012
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=66dd8ab65e2d9352de86e47056dea0b701e21a15
Submitter: Jenkins
Branch: stable/newton

commit 66dd8ab65e2d9352de86e47056dea0b701e21a15
Author: liusheng <email address hidden>
Date: Tue Jun 21 09:19:22 2016 +0800

    Add support of refereshing the resource info in local cache

    The local resource caching mechanism may cause legacy resource info
    resided in the local cache and never be cleaned, uless the service
    being restarted. e.g. an instance has been migrated to another host, the
    instance info will reside in the cache on the ceilometer-polling
    service running on the old compute host.

    Change-Id: I55f12ea9f6f6104de6a38b98ad935d458a7b0cc8
    Closes-Bug: #1594285
    Closes-Bug: #1656166

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 7.0.5

This issue was fixed in the openstack/ceilometer 7.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.