Periodic tasks share instance-specific data in a class attribute
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
oslo-incubator |
Fix Released
|
High
|
Ben Nemec |
Bug Description
On a fairly consistent basis, the periodic tasks test cases fail when run in parallel. For example:
2014-01-31 20:13:31.925 | FAIL: tests.unit.
2014-01-31 20:13:31.925 | tags: worker-1
2014-01-31 20:13:31.926 | -------
2014-01-31 20:13:31.926 | Empty attachments:
2014-01-31 20:13:31.926 | stderr
2014-01-31 20:13:31.926 | stdout
2014-01-31 20:13:31.926 |
2014-01-31 20:13:31.926 | pythonlogging:'': {{{
2014-01-31 20:13:31.926 | ERROR [openstack.
2014-01-31 20:13:31.927 | Traceback (most recent call last):
2014-01-31 20:13:31.927 | File "openstack/
2014-01-31 20:13:31.927 | task(self, context)
2014-01-31 20:13:31.927 | File "tests/
2014-01-31 20:13:31.927 | raise AnException('urg')
2014-01-31 20:13:31.927 | AnException: urg
2014-01-31 20:13:31.927 | ERROR [openstack.
2014-01-31 20:13:31.928 | Traceback (most recent call last):
2014-01-31 20:13:31.928 | File "openstack/
2014-01-31 20:13:31.928 | task(self, context)
2014-01-31 20:13:31.928 | File "tests/
2014-01-31 20:13:31.928 | raise AnException('urg')
2014-01-31 20:13:31.928 | AnException: urg
2014-01-31 20:13:31.928 | }}}
2014-01-31 20:13:31.929 |
2014-01-31 20:13:31.929 | Traceback (most recent call last):
2014-01-31 20:13:31.929 | File "tests/
2014-01-31 20:13:31.929 | self.assertTrue
2014-01-31 20:13:31.929 | File "/usr/lib/
2014-01-31 20:13:31.929 | raise self.failureExc
2014-01-31 20:13:31.929 | AssertionError: False is not true
I have spent a fair amount of time trying to track down the problem and have found that I can only reproduce this if I run the entire test_periodic.py file in parallel. Running just the failing set of tests or trying to narrow it down in any other way resulted in no failures.
The unit tests that are failing are not particularly good in the first place, so I'll probably go ahead and submit a change to improve them, but I still don't have a full solution to the parallel problem.
Changed in oslo: | |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in oslo: | |
milestone: | none → icehouse-3 |
Changed in oslo: | |
status: | Fix Committed → Fix Released |
Changed in oslo: | |
milestone: | icehouse-3 → 2014.1 |
I believe I have tracked this down to some data that is specific to an instance of a periodic task class but is stored in a class attribute so it leaks out to all other instances of the same class. I'm unsure whether this is a problem in actual use of this code, but it's unintuitive and wrong IMHO so I'm going to propose a change.