test_tagged_attachment randomly fails with: sequence or collection is not empty: [{u'type': u'disk', u'bus': u'pci', u'address': u'0000:00:07.0', u'serial': u'56e9a4a0-51fc-4319-845b-02c6a21d5b16', u'tags': [u'volume-tag']}]
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tempest |
Fix Released
|
High
|
Ghanshyam Mann |
Bug Description
Seen here:
2018-06-06 22:51:16.149472 | controller | {2} tempest.
2018-06-06 22:51:16.149532 | controller |
2018-06-06 22:51:16.149586 | controller | Captured traceback:
2018-06-06 22:51:16.149637 | controller | ~~~~~~~~~~~~~~~~~~~
2018-06-06 22:51:16.149726 | controller | Traceback (most recent call last):
2018-06-06 22:51:16.149855 | controller | File "tempest/
2018-06-06 22:51:16.149938 | controller | return f(*func_args, **func_kwargs)
2018-06-06 22:51:16.150112 | controller | File "tempest/
2018-06-06 22:51:16.150189 | controller | self.verify_
2018-06-06 22:51:16.150356 | controller | File "tempest/
2018-06-06 22:51:16.150443 | controller | CONF.compute.
2018-06-06 22:51:16.150678 | controller | File "tempest/
2018-06-06 22:51:16.150764 | controller | if func(*args, **kwargs):
2018-06-06 22:51:16.150924 | controller | File "tempest/
2018-06-06 22:51:16.151000 | controller | verify_
2018-06-06 22:51:16.151170 | controller | File "tempest/
2018-06-06 22:51:16.151276 | controller | self.assertEmpt
2018-06-06 22:51:16.151380 | controller | File "tempest/test.py", line 863, in assertEmpty
2018-06-06 22:51:16.151454 | controller | self.assertFals
2018-06-06 22:51:16.151649 | controller | File "/opt/stack/
2018-06-06 22:51:16.151732 | controller | raise self.failureExc
2018-06-06 22:51:16.152319 | controller | AssertionError: [{u'type': u'disk', u'bus': u'pci', u'address': u'0000:00:07.0', u'serial': u'56e9a4a0-
87 hits in 7 days, check and gate, all failures.
The test that is failing was merged on May 31 so this is pretty recent and a reasonably high failure rate:
https:/
This is in the tempest logs before it fails:
There is a sleep in this test so it's obviously racy:
https:/
That should probably sleep until the verify method passes or we timeout.
Changed in tempest: | |
status: | New → Confirmed |
importance: | Undecided → High |
yes, it is easily reproducible on local env by increasing the nova's config option metadata_ cache_expiratio n value to something higher (i did 50 sec).
It fail in both place in verify_ empty_devices( )[1] as well as in verify_ device_ metadata( )[2]
[1] https:/ /github. com/openstack/ tempest/ blob/5aeb551a58 078abd34884583a c70ad425f4590ba /tempest/ api/compute/ servers/ test_device_ tagging. py#L310
[2] https:/ /github. com/openstack/ tempest/ blob/5aeb551a58 078abd34884583a c70ad425f4590ba /tempest/ api/compute/ servers/ test_device_ tagging. py#L306
as matt mentioned in gerrit, we should go with wait loop here than hard coded sleep.