tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing

Bug #1775947 reported by Slawek Kaplonski on 2018-06-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Medium
Unassigned

Bug Description

Since few days I see that tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment in neutron-tempest-dvr job.

Example of failure: http://logs.openstack.org/90/572690/2/check/neutron-tempest-dvr/45ec391/logs/testr_results.html.gz

It happened at least 3 times on 8.06: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22exit%20status%3A%20137%2C%20stderr%3A%20Killed%5C%22%20AND%20build_name%3A%5C%22neutron-tempest-dvr%5C%22

Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Slawek Kaplonski (slaweq) wrote :

It looks like it is some issue with tempest test probably.
For now it looks like it detaches interface and then "too fast" tries to get metadata on instance. It's in tempest/api/compute/servers/test_device_tagging.py

Maybe some waif for detached state first could do the job? I need to test it.

Deepak Mourya (mourya007) wrote :

Hi, yes i have tried this and got some test case failed in the tempest side. https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_device_tagging.py#L350

Changed in neutron:
assignee: nobody → Deepak Mourya (mourya007)
affects: neutron → tempest
Slawek Kaplonski (slaweq) wrote :

After some analysis of logs from failed run it looks for me that this "exit code 137" is because test reached timeout and was just cleaned.
I think that in verify_empty_devices() method there is still some device existing and because f that this test is not finishing.
I don't know yet if that is volume or nic device which wasn't removed from this metadata properly but will investigate...

Ghanshyam Mann (ghanshyammann) wrote :

In verify_empty_devices(), we can check the non presence of specified tags ['nic-tag', 'volume-tag'] instead of no device so that if any other devices is there we do not keep trying in wait loop.

Deepak Mourya (mourya007) wrote :

@slaweq This test is now passed on my environment. Can you please check the same on your environment or Is it still reproducable?

python -m testtools.run tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment
Tests running...

Ran 1 test in 61.800s
OK

Reviewed: https://review.openstack.org/578765
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Submitter: Zuul
Branch: master

commit eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 28 13:03:04 2018 +0200

    Add additional log of meta_data devices content

    In TaggedAttachmentsTest.test_tagged_attachment test there is
    now added some additional debug log to check what
    devices are still in meta_data's devices field when it should
    be empty already.

    This patch changes also from handling all exeptions to only
    AssertionError in verify_empty_devices() method.
    If exeption other than AssertionError occur, it will not be silently
    ignored anymore.

    Change-Id: I95016e35102569fba79fc58f9695e480234619dc
    Related-Bug: #1775947

Slawek Kaplonski (slaweq) wrote :

I just spotted this issue again and it looks that volume device is not removed from this meta_data.json file:
http://logs.openstack.org/69/567369/15/check/tempest-full/528bc75/job-output.txt.gz#_2018-07-19_10_06_09_273919

Reviewed: https://review.openstack.org/586292
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Submitter: Zuul
Branch: master

commit 4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Author: Matt Riedemann <email address hidden>
Date: Thu Jul 26 12:28:48 2018 -0400

    Skip verifying empty devices in test_tagged_attachment until bug is fixed

    We have a non-trivial amount of failures due to bug 1775947 [1]
    and so far attempts at debugging the issue haven't proved
    fruitful, so let's skip the last (problematic) portion of the
    test until we can get a recreate with proper debug to help
    isolate the problem.

    Related nova debug patch: https://review.openstack.org/584032

    [1] http://status.openstack.org/elastic-recheck/#1775947

    Change-Id: I3104041cb66f912727aabd6c5894600dc87a1229
    Related-Bug: #1775947

Changed in tempest:
assignee: Deepak Mourya (mourya007) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers