_compare_results incorrectly reports extra list items when items in lists don't match

[Augustina Ragwitz]

I discovered this bug while refactoring functional tests. This happens in the current Nova master.

The _compare_results function is defined in nova/tests/functional/api_samples_test_base.py

To reproduce, pass two lists to _compare_results that are the same length but have one item not equal.

list1 = [{u'foo': u'bar', u'bar': u'baz'}]

list2 = [{u'foo': u'baz', u'bar': u'baz'}]

_compare_results returns the following:

nova.tests.functional.api_samples_test_base.NoMatch: Extra list items in template:
{u'foo': u'bar', u'bar': u'baz'}
Extra list items in Test:
{u'foo': u'baz', u'bar': u'baz'}

While this is "technically" accurate, it's really hard to debug if you can imagine a lengthy, nested json file dumped into that response.

_compare_results is already checking each item in the list, it should just be returning a list of keys that don't match, not the whole thing!

[Sarafraj Singh (sarafraj-singh)]

Auggy,
Are you working on the fix? Please change status to Inprogress if you are, otherwise change Assigned to ->nobody.

[Takashi Natsume (natsume-takashi)]

A bug report with an assignee should be 'In Progress' status.
So set 'In Progress'.

[Augustina Ragwitz (auggy)]

This is on my TODO list but I haven't made working on it a priority so moving to unassigned in case anyone else wants to give it a shot.

[Prateek Arora (parora)]

The code is not written to handle cases that have nested data structures and in case the nested list won't match word for word, it would fully return the elements as they have not found any match. IMO the compare results need to call itself in case it encounters a dict or a list and retry matching.

[Prateek Arora (parora)]

Auggy, IMO the nested list(or dicts) comparison is possible only when we have one extra list in the result as well as expected keys, but in case we have more than 1 in both, it is computationally not good to check the closest dictionary in the expected dictionaries and result dictionaries.

Assuming we get 3 not matching lists in result
and 2 not matching lists in expected result

we would have to match 2 lists for each of the 3 lists in the result dict, find the closest companion and print.

However it would be good if we have 1 not matching dict in result as well as expected result and then find the differing keys.I would try to work on this case.