ImagePollster record duplicate counter during one poll

[xingzhou]

I've established the integration between central agent and glance, and upload some images to glance, then start central agent to poll the image info, below is the image info in glance in my local env:

| 0c8d025d-8a23-4f1c-82e1-b4627fbe8e7a | | | | | queued |
| 65da8737-aaf0-44de-84b9-7344b71d8b32 | c1 | qcow2 | bare | 13147648 | active |
| df665c76-bd11-4553-8ce0-d0b6f8f56ec5 | c2 | qcow2 | bare | 13147648 | active |
| 254a1156-0a40-44e5-9327-a19af7a86090 | c3 | qcow2 | bare | 13147648 | active |
| 215fcbf6-81f1-4487-b55c-5b345083feaa | c4 | qcow2 | bare | 13147648 | active |
| 987449ef-912a-49ff-af4c-cbf62dd089ad | c4 | qcow2 | bare | 13147648 | active |
| c30812b8-ae60-417e-9b3c-0b32d92c10de | c5 | qcow2 | bare | 13147648 | active |
| 21ecd717-fa53-4778-b4c1-ad82140ca016 | c6 | qcow2 | bare | 13147648 | active |
| f05a65f5-4b65-4919-9cf3-3c5ac56e0010 | cirros-0.3.1-x86_64-uec | ami | ami | 25165824 | active |
| 4d35d4d7-313b-4194-a89b-6a5d466774da | cirros-0.3.1-x86_64-uec-kernel | aki | aki | 4955792 | active |
| 8ef96a84-8dc5-4b9e-ba6d-d4c165add053 | cirros-0.3.1-x86_64-uec-ramdisk | ari | ari | 3714968 | active |
| 451674e1-4646-4dea-afe1-d0ea39164669 | dd | qcow2 | bare | 13147648 | active |

there are 12 images at present, but seems in each central agent's image pollster poll, 21 image info are recorded down, I've added a print call in the glance.ImagePollster#get_counters method to print the image id that is handled in each poll, and the result is:

##processing image, its id is 21ecd717-fa53-4778-b4c1-ad82140ca016
##processing image, its id is c30812b8-ae60-417e-9b3c-0b32d92c10de
##processing image, its id is 987449ef-912a-49ff-af4c-cbf62dd089ad
##processing image, its id is 215fcbf6-81f1-4487-b55c-5b345083feaa
##processing image, its id is 254a1156-0a40-44e5-9327-a19af7a86090
##processing image, its id is df665c76-bd11-4553-8ce0-d0b6f8f56ec5
##processing image, its id is 65da8737-aaf0-44de-84b9-7344b71d8b32
##processing image, its id is 451674e1-4646-4dea-afe1-d0ea39164669
##processing image, its id is 0c8d025d-8a23-4f1c-82e1-b4627fbe8e7a
##processing image, its id is f05a65f5-4b65-4919-9cf3-3c5ac56e0010
##processing image, its id is 8ef96a84-8dc5-4b9e-ba6d-d4c165add053
##processing image, its id is 4d35d4d7-313b-4194-a89b-6a5d466774da
##processing image, its id is 21ecd717-fa53-4778-b4c1-ad82140ca016
##processing image, its id is c30812b8-ae60-417e-9b3c-0b32d92c10de
##processing image, its id is 987449ef-912a-49ff-af4c-cbf62dd089ad
##processing image, its id is 215fcbf6-81f1-4487-b55c-5b345083feaa
##processing image, its id is 254a1156-0a40-44e5-9327-a19af7a86090
##processing image, its id is df665c76-bd11-4553-8ce0-d0b6f8f56ec5
##processing image, its id is 65da8737-aaf0-44de-84b9-7344b71d8b32
##processing image, its id is 451674e1-4646-4dea-afe1-d0ea39164669
##processing image, its id is 0c8d025d-8a23-4f1c-82e1-b4627fbe8e7a

in the above message, there are some duplicate ids there, and I checked the backend mongodb, there are 42 meters recorded(21*2, indicating that there are two counterrs, image and image.size), I think there are duplicate things stored in the process, please correct me if there is anything in my steps is wrong

[Julien Danjou (jdanjou)]

Which images in the list are public or private?

[xingzhou (xingzhou)]

all of the images in my local env are private ones

[Julien Danjou (jdanjou)]

Could you hack around iter_images to see what returns each calls done to client.images.list()?

The thing is that I'm afraid Glance returns an image twice without taking into account the is_public filter we use.

[xingzhou (xingzhou)]

Hi Julien, yes, I've printed the image ids that responded from glance, seems glance just ignore the is_Public filter(or may be we are using wrong filter grammar) and return the images twice, but this is still strange that if a double image-list is returned, there should be 12*2 image meter recorded down, but seems it is 21 at present, I'll keep on debugging this issue and try to find the cause

[Julien Danjou (jdanjou)]

Yeah, I'm pretty sure the problem is with Glance or Ceilometer usage of the API. Could you tell me what version is your Glance server running?

[xingzhou (xingzhou)]

Hi Julien, I'm using the dev edition, and I also found that there are some lines in glance that may explain the reason:

def _get_images(self, context, filters, **params):
        """
        Get images, wrapping in exception if necessary.
        """
        # NOTE(markwash): for backwards compatibility, is_public=True for
        # admins actually means "treat me as if I'm not an admin and show me
        # all my images"
        if context.is_admin and params.get('is_public') is True:
            context.is_admin = False
            del params['is_public']
        try:
            return self.db_api.image_get_all(context, filters=filters,
                                             **params)
        except exception.NotFound as e:
            msg = _("Invalid marker. Image could not be found.")
            raise exc.HTTPBadRequest(explanation=msg)

I'm also using admin user to run ceilometer and glance, in the above api we can see that the is_public param will be removed in some situations, that's will help to explain why all the images have been returned even if we have set the is_public=True. I think this should be an Glance API usage issue from Ceilometer side. I'll try to help fix it for my first Ceilometer defect:)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29877

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29877
Committed: http://github.com/openstack/ceilometer/commit/dfcd0dd7c4dad64b282843f681213b335ab735f4
Submitter: Jenkins
Branch: master

commit dfcd0dd7c4dad64b282843f681213b335ab735f4
Author: xingzhou <email address hidden>
Date: Tue May 21 02:11:18 2013 -0400

    ImagePollster record duplicate counter during one poll

    When using ImagePollster to poll image status from Glance, there might
    be duplicate image status event recorded down if there are private
    images together with public images stored in Glance.

    In Glance, the get image list API looks like:
        def _get_images(self, context, filters, **params):
            ...
     # NOTE(markwash): for backwards compatibility, is_public=True for
     # admins actually means "treat me as if I'm not an admin and show
     # all my images"
            if context.is_admin and params.get('is_public') is True:
                context.is_admin = False
                del params['is_public']
            ...
    The above code shows that if a user is of 'admin_role'(this option is
    configured in the glance-api.conf) and querying for public images,
    Glance will return all the public images together with the private
    ones

    While in ceilometer/image/glance.py#_Base.iter_images method,
    ceilometer get the image list by querying public and private images
    respectively and then chain the result lists together. This causes
    the duplication of image in the image list.

    This fix changes the code in ImagePollster.get_counters method to
    filter out the duplicate images based on their image id.

    Change-Id: Ib5e213161043033c20195efc0c6e7edd78982bd6
    Fixes: Bug 1180630

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31198
Committed: http://github.com/openstack/ceilometer/commit/2d79ca92ac3714ebfbe92aa01782b23f095f9baa
Submitter: Jenkins
Branch: stable/grizzly

commit 2d79ca92ac3714ebfbe92aa01782b23f095f9baa
Author: xingzhou <email address hidden>
Date: Tue May 21 02:11:18 2013 -0400

    ImagePollster record duplicate counter during one poll

    When using ImagePollster to poll image status from Glance, there might
    be duplicate image status event recorded down if there are private
    images together with public images stored in Glance.

    In Glance, the get image list API looks like:
        def _get_images(self, context, filters, **params):
            ...
     # NOTE(markwash): for backwards compatibility, is_public=True for
     # admins actually means "treat me as if I'm not an admin and show
     # all my images"
            if context.is_admin and params.get('is_public') is True:
                context.is_admin = False
                del params['is_public']
            ...
    The above code shows that if a user is of 'admin_role'(this option is
    configured in the glance-api.conf) and querying for public images,
    Glance will return all the public images together with the private
    ones

    While in ceilometer/image/glance.py#_Base.iter_images method,
    ceilometer get the image list by querying public and private images
    respectively and then chain the result lists together. This causes
    the duplication of image in the image list.

    This fix changes the code in ImagePollster.get_counters method to
    filter out the duplicate images based on their image id.

    Change-Id: Ib5e213161043033c20195efc0c6e7edd78982bd6
    Fixes: Bug 1180630
    (cherry picked from commit dfcd0dd7c4dad64b282843f681213b335ab735f4)