ceilometer continues to record nova instances 'exists' events after receiving 'delete.start' event

[François Charlier]

Ceilometer currently filters the events received from nova to try to bracket the lifespan of metering an instance from the end of creation (event 'compute.instance.create.end') to the start of its deletion (event 'compute.instance.delete.start').

It happens that if the instance_usage_audit job runs after the start of deletion, but before the end of the deletion a 'compute.instance.exists' event is recorded whereras the instance is no longer accessible.

The appearance frequency of this problem will largely depend on the value of 'instance_usage_audit_period' in nova.conf, but might happen more if it's set to 'hour'. Supposing that deleting an instance will last at least one hour, there's still the chance to record up to one hour of data that should not be recorded.

[1] https://github.com/openstack/ceilometer/blob/ddc37e6e9abd018405beea72576e759297318879/ceilometer/compute/notifications.py#L101-L106

[Julien Danjou (jdanjou)]

I've checked how Nova works about this, and actually the last meters should have a metadata.state field set to 'soft-delete'. Could you confirm that François, based on your data?

Based on that, I think this is the same problem someone might have with metadata.state being set to 'suspended' when VM are suspended. It's not up to the collection event nor Nova to deal with that, but with the API to allow more advanced filtering and computing on duration based on range where the metadata.state value of this meter is set to 'active'.

[François Charlier (francois-charlier)]

In the ceilometer version I'm using (Debian package "2013.1~g2-1"), I unfortunately can't find any meter with a metadata.state field.

If you can confirm this field is here now I'm ok with this answer and bug #1187843 is half resolved, but then there's no point in limiting the recorded events as done in https://github.com/openstack/ceilometer/blob/ddc37e6e9abd018405beea72576e759297318879/ceilometer/compute/notifications.py#L101-L106.

[Julien Danjou (jdanjou)]

After checking master version, I don't think the vm_state field from notifications is properly captured currently by our notifications. I think I'll tackle this by capturing and flattening all metadata.

[Julien Danjou (jdanjou)]

Adding a note to say that you're also right about limiting events.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/33057

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/33228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33057
Committed: http://github.com/openstack/ceilometer/commit/00531f9de040f43c643dd843ada986cfbcf91e87
Submitter: Jenkins
Branch: master

commit 00531f9de040f43c643dd843ada986cfbcf91e87
Author: Julien Danjou <email address hidden>
Date: Fri Jun 14 13:51:39 2013 +0200

    Unify Counter generation from notifications

    This will make sure we don't lose any metadata information from
    notifications into the Counter/samples we publish.

    This is part of fixing bug #1187833 and bug #1187843

    Change-Id: I315581907be1dcec33f43a5d5fd7b7876a7889f6
    Signed-off-by: Julien Danjou <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33228
Committed: http://github.com/openstack/ceilometer/commit/2dffc1fc51878fcfe4d8e71e30c15b8cf184f5a8
Submitter: Jenkins
Branch: master

commit 2dffc1fc51878fcfe4d8e71e30c15b8cf184f5a8
Author: Julien Danjou <email address hidden>
Date: Mon Jun 17 11:25:16 2013 +0200

    Register all interesting events

    As stated in bug #1187833, there's no need to not record all of this events.
    We need to filter events on the state anyway.

    Also adding update events and documentation on how to enable them.

    Change-Id: Ic5845e1a3dc5ed531114e7ccdbe17bb98cb049fb
    Signed-off-by: Julien Danjou <email address hidden>