duplicate samples for network.services.lb.member

[Mike Spreitzer]

For some of the network.services... meters, each Sample is doubled. That is, Samples come in identical pairs. This happens for network.services.lb.member, network.services.lb.health_monitor, network.services.lb.pool, and network.services.lb.vip; it does NOT happen for network.services.lb.incoming.bytes nor network.services.lb.total.connections.

If I edit the first source in pipeline.yaml --- the source that says it matches all meters --- to explicitly exclude one of the doubled meters, and restart the relevant agent, then the doubling stops.

[Pradeep Kilambi (pkilambi)]

hmm I verified in the DB and i don't see duplicate samples in mysql. For example,

mysql> select * from sample where meter_id = 13\G;
*************************** 1. row ***************************
               id: 83
          user_id: NULL
       project_id: 05808ee2515e4623ad33420a94500907
      resource_id: 0bdd7b5a-b92a-44cb-a351-b7c78938cc3a
resource_metadata: {"status": "ACTIVE", "status_description": null, "protocol": "HTTP", "description": "", "admin_state_up": true, "subnet_id": "fd782ef4-c502-46aa-a04e-0ea77e0f23cc", "connection_limit": -1, "pool_id": "648d29af-767a-4cc6-9f0b-2344f7503339", "session_persistence": {"type": "SOURCE_IP"}, "address": "10.0.0.5", "protocol_port": 80, "port_id": "a03648c3-ad65-415b-a47d-d292924dc8e0", "name": "myvip"}
           volume: 1
message_signature: a5d223d19ad689cc85a915532dfc81d5d1dd5c21d987fee204f13f39e516dd6d
       message_id: 5bb22bb4-26ec-11e4-9077-000c295e3c02
        timestamp: 1408375732.049707
      recorded_at: 1408375732.065283
         meter_id: 13
        source_id: openstack
*************************** 2. row ***************************
               id: 84
          user_id: NULL
       project_id: 05808ee2515e4623ad33420a94500907
      resource_id: 0bdd7b5a-b92a-44cb-a351-b7c78938cc3a
resource_metadata: {"status": "ACTIVE", "status_description": null, "protocol": "HTTP", "description": "", "admin_state_up": true, "subnet_id": "fd782ef4-c502-46aa-a04e-0ea77e0f23cc", "connection_limit": -1, "pool_id": "648d29af-767a-4cc6-9f0b-2344f7503339", "session_persistence": {"type": "SOURCE_IP"}, "address": "10.0.0.5", "protocol_port": 80, "port_id": "a03648c3-ad65-415b-a47d-d292924dc8e0", "name": "myvip"}
           volume: 1
message_signature: a5d223d19ad689cc85a915532dfc81d5d1dd5c21d987fee204f13f39e516dd6d
       message_id: 5bb22bb4-26ec-11e4-9077-000c295e3c02
        timestamp: 1408375732.049707
      recorded_at: 1408375732.169598
         meter_id: 13
        source_id: openstack
*************************** 3. row ***************************
               id: 166
          user_id: NULL
       project_id: 05808ee2515e4623ad33420a94500907
      resource_id: 0bdd7b5a-b92a-44cb-a351-b7c78938cc3a
resource_metadata: {"status": "ACTIVE", "status_description": null, "protocol": "HTTP", "description": "", "admin_state_up": true, "subnet_id": "fd782ef4-c502-46aa-a04e-0ea77e0f23cc", "connection_limit": -1, "pool_id": "648d29af-767a-4cc6-9f0b-2344f7503339", "session_persistence": {"type": "SOURCE_IP"}, "address": "10.0.0.5", "protocol_port": 80, "port_id": "a03648c3-ad65-415b-a47d-d292924dc8e0", "name": "myvip"}
           volume: 1
message_signature: 8204c3320aa20fc1e1d25bf98fb47af375a576afda9363ef5affee7cc7f1ef12
       message_id: 4b0a9d18-26ed-11e4-8521-000c295e3c02
        timestamp: 1408376133.604812
      recorded_at: 1408376133.625570
         meter_id: 13
        source_id: openstack
*************************** 4. row ***************************
               id: 167
          user_id: NULL
       project_id: 05808ee2515e4623ad33420a94500907
      resource_id: 0bdd7b5a-b92a-44cb-a351-b7c78938cc3a
resource_metadata: {"status": "ACTIVE", "status_description": null, "protocol": "HTTP",...

[Pradeep Kilambi (pkilambi)]

The message id does seem to be the same for couple of rows in the above query. Though the id and timestamp are different. The resources discovered are returning only one value So not sure yet. Will continue investigating.

[Nejc Saje (nejc-saje)]

The dictionary holding the static resources and discovery configuration for pollsters is keyed by pollster name - without source name. So anywhere in the pipeline config a pollster has some resources or discoveries configured, they will be amalgamated for all pipelines. So unless we massively refactor the pipeline code, I suggest for now we put negations into the catch-all source where necessary.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124027

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124027
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=d8317189e554c8378eefc615b73726f4b89791cb
Submitter: Jenkins
Branch: master

commit d8317189e554c8378eefc615b73726f4b89791cb
Author: Eoghan Glynn <email address hidden>
Date: Wed Sep 24 09:56:09 2014 +0000

    Per-source separation of static resources & discovery

    Previously, the amalgamation of static resources and discovery
    extensions defined for all matching pipeline sources were passed
    to each pollster on each polling cycle.

    This led to unintended duplication of the samples emitted when
    an individual pollster matched several sources.

    Now, we relate the static resources and discovery extensions to
    the originating sources and only pass these when a pollster is
    traversed in the context of that source.

    Similarly, sinks are now related to the originating source and
    samples are only published over the sinks corresponding to the
    current sources.

    Closes-Bug: #1357869

    Change-Id: Ie973625325ba3e25c76c90e4792eeaf466ada657