Pagination returns the wrong offset/element id

[Ryan Bak]

In some cases pagination is either returning the wrong offset or element.id I believe this was introduced with the changes in offset for the addition of group_by. The following is an runthrough of an example using the provided offset to generate the next call. Note the changes in the results of the third call. Element name was constant, and no dimensions were returned due to the merge statement.

Query 1:
monasca --debug measurement-list --dimensions region=NCE --merge_metrics --endtime 2016-05-15T05:59:59.999Z keystone.time.token_get 2016-05-08T06:00:00.000Z

Result 1:
offset=c957e59b3a7e2f008108dd20f0676e590eefc60e_2016-05-10T13%3A33%3A01.672Z
"elements":[{"id":"c957e59b3a7e2f008108dd20f0676e590eefc60e"

Query 2:
monasca --debug measurement-list --dimensions region=NCE --merge_metrics --endtime 2016-05-15T05:59:59.999Z --offset c957e59b3a7e2f008108dd20f0676e590eefc60e_2016-05-10T13:33:01.672Z keystone.time.token_get 2016-05-08T06:00:00.000Z

Result 2:
offset=c957e59b3a7e2f008108dd20f0676e590eefc60e_2016-05-12T21%3A06%3A01.883Z
"elements":[{"id":"c957e59b3a7e2f008108dd20f0676e590eefc60e"

Query 3:
monasca --debug measurement-list --dimensions region=NCE --merge_metrics --endtime 2016-05-15T05:59:59.999Z --offset c957e59b3a7e2f008108dd20f0676e590eefc60e_2016-05-12T21:06:01.883Z keystone.time.token_get 2016-05-08T06:00:00.000Z

Result 3:
offset=067d4ecdc15d4f301a10b66810c8bb2ddadf0f7b_2016-05-15T04%3A40%3A01.331Z
"elements":[{"id":"067d4ecdc15d4f301a10b66810c8bb2ddadf0f7b"

Query 4:
monasca --debug measurement-list --dimensions region=NCE --merge_metrics --endtime 2016-05-15T05:59:59.999Z --offset 067d4ecdc15d4f301a10b66810c8bb2ddadf0f7b_2016-05-15T04:40:01.331Z keystone.time.token_get 2016-05-08T06:00:00.000Z

Result 4:
offset N/A (No next link provided)
"elements":[{"id":"0d5cbbd2b0f702bedaffb84008e427869d4fbb17"

[Ryan (ryan-brandt)]

This does look like a bug, I will try to reproduce. The expected behavior with 'merge-metrics=true' should be for no ids to be returned in the offset and elements.

[Ryan Bak (ryanmbak)]

Ok, so maybe not a bug, just inconsistent design that's causing problems for Grafana. When I make multiple queries I have to reconstruct the paged data for grafana to display properly. What you're saying is that depending on whether I use group_by or merge_metrics, I have to behave completely different. For merge I have to just assume all the elements returned are part of a single dataset regardless of element.id, and for group_by I can't assume any elements are the same unless they have the same element.id.

This could be handled by grafana, it's just messy. However once group_by allows more than, queries will be able to merge on some dimensions and group on others, meaning that I'll need a consistent element.id to reconstruct each of the dataset groups, regardless of the fact that some metrics are merged within a single group.

[Ryan (ryan-brandt)]

The trouble with the ids is that they apply only to complete dimension sets, so any merging invalidates the ids. However, each element returned should be a separate dataset, so I'm confused as to what you mean by 'dataset groups'.

The case where some groups are merged and some are not is not handled yet and will require some thought regarding ids...

[Ryan Bak (ryanmbak)]

I realize the group and merge isn't implemented, and you can ignore my term dataset groups. The point there was just that eventually we'll need the id to be consistent for each dataset when we are returning multiple elements in multiple pages, and those datasets may be merged.

However I would still argue this is a bug. It's non-obvious enough that two people with a lot of monasca experience thought it is was, and customers won't know details like what a dimension set is, and shouldn't need to know that level of detail in order to interpret the results of a query. And given that, plus the issue that this will need to be solved eventually, it seems like this is something that should be solved sooner rather than later.