test_calculate_delta fails with message "TypeError: unhashable type: 'list'"

[Abed Abu dbai]

log:
http://logs.openstack.org/51/406951/16/check/gate-octavia-python35/27292e5/console.html

==============================
2017-01-11 10:00:28.246355 | Failed 1 tests - output below:
2017-01-11 10:00:28.246366 | ==============================
2017-01-11 10:00:28.246371 |
2017-01-11 10:00:28.246395 | octavia.tests.unit.controller.worker.tasks.test_network_tasks.TestNetworkTasks.test_calculate_delta
2017-01-11 10:00:28.246420 | ---------------------------------------------------------------------------------------------------
2017-01-11 10:00:28.246426 |
2017-01-11 10:00:28.246434 | Captured traceback:
2017-01-11 10:00:28.246443 | ~~~~~~~~~~~~~~~~~~~
2017-01-11 10:00:28.246457 | b'Traceback (most recent call last):'
2017-01-11 10:00:28.246488 | b' File "/home/jenkins/workspace/gate-octavia-python35/.tox/py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched'
2017-01-11 10:00:28.246502 | b' return func(*args, **keywargs)'
2017-01-11 10:00:28.246539 | b' File "/home/jenkins/workspace/gate-octavia-python35/octavia/tests/unit/controller/worker/tasks/test_network_tasks.py", line 100, in test_calculate_delta'
2017-01-11 10:00:28.246560 | b' self.assertEqual(empty_deltas, net.execute(self.load_balancer_mock))'
2017-01-11 10:00:28.246591 | b' File "/home/jenkins/workspace/gate-octavia-python35/octavia/controller/worker/tasks/network_tasks.py", line 110, in execute'
2017-01-11 10:00:28.246609 | b' delta = calculate_amp.execute(loadbalancer, amphora)'
2017-01-11 10:00:28.246639 | b' File "/home/jenkins/workspace/gate-octavia-python35/octavia/controller/worker/tasks/network_tasks.py", line 57, in execute'
2017-01-11 10:00:28.246658 | b' desired_network_ids.add(CONF.controller_worker.amp_network)'
2017-01-11 10:00:28.246681 | b"TypeError: unhashable type: 'list'"
2017-01-11 10:00:28.246689 | b''

******* note: could not reproduce locally **********

patch:
https://review.openstack.org/#/c/406951/16

[Michael Johnson (johnsom)]

Can you include a link to the patch you are seeing this on? This test is passing fine on master.

[Abed Abu dbai (abeda)]

It was the patch 16:

https://review.openstack.org/#/c/406951/16

I also notice that after the last merge with master (patch 17) the problem disappeared but it doesn't say it was solved

[Michael Johnson (johnsom)]

This is hard to debug a bit.

It may be configuration contamination from another test, i.e. the test environment wasn't setup properly.

You could try running the following for a period of time (overnight for example) and see if you can reproduce the issue:

tox -e py35 -- --until-failure

[Abed Abu dbai (abeda)]

Hi Michael Johnson,

Were to run this? and what does it do?

"""You could try running the following for a period of time (overnight for example) and see if you can reproduce the issue:

tox -e py35 -- --until-failure """

[Michael Johnson (johnsom)]

This runs one of the tox tests (which you should run before you check in code) until it detects a failure.
In this case I specified that it only run the py35 test that failed in your report.

Tox randomizes the order of the tests to catch any poorly written tests that contaminate or depend on other tests.

By running until a failure occurs it will cycle through a number of test orders to help reproduce the issue.

[Michael Johnson (johnsom)]

I just found that the recent change to ostestr is breaking --until-failure: https://bugs.launchpad.net/os-testr/+bug/1472395

So, I edited my tox.ini back to the old testr way:
https://github.com/openstack/octavia/commit/4edcf3a8dfd22166f54da66a74835ccf9cdf4b6b

I am running that on master now. My suspicion, since we have not seen this outside the active/active chain is the problem was introduced in the act/act patches.
Running this with until-failure on master will prove that one way or the other.

[Abed Abu dbai (abeda)]

Thanks Michael,

The problem is currently seen in the "Active-Active Topology - Cluster DB Tasks" only
https://review.openstack.org/#/c/409764/20

[Michael Johnson (johnsom)]

So I ran tox -e py35 -- --until-failure all day on the master code and did not encounter this error.
It must be an issue introduced in one of the patches in the patch chain. Please test the patches in the chain to find which one is introducing the error.

[Launchpad Janitor (janitor)]

[Expired for octavia because there has been no activity for 60 days.]