tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing

Bug #1775947 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Expired
Medium
Unassigned

Bug Description

Since few days I see that tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment in neutron-tempest-dvr job.

Example of failure: http://logs.openstack.org/90/572690/2/check/neutron-tempest-dvr/45ec391/logs/testr_results.html.gz

It happened at least 3 times on 8.06: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22exit%20status%3A%20137%2C%20stderr%3A%20Killed%5C%22%20AND%20build_name%3A%5C%22neutron-tempest-dvr%5C%22

Tags: gate-failure
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

It looks like it is some issue with tempest test probably.
For now it looks like it detaches interface and then "too fast" tries to get metadata on instance. It's in tempest/api/compute/servers/test_device_tagging.py

Maybe some waif for detached state first could do the job? I need to test it.

Revision history for this message
Deepak Mourya (mourya007) wrote :

Hi, yes i have tried this and got some test case failed in the tempest side. https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_device_tagging.py#L350

Changed in neutron:
assignee: nobody → Deepak Mourya (mourya007)
affects: neutron → tempest
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tempest (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/578765

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

After some analysis of logs from failed run it looks for me that this "exit code 137" is because test reached timeout and was just cleaned.
I think that in verify_empty_devices() method there is still some device existing and because f that this test is not finishing.
I don't know yet if that is volume or nic device which wasn't removed from this metadata properly but will investigate...

Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

In verify_empty_devices(), we can check the non presence of specified tags ['nic-tag', 'volume-tag'] instead of no device so that if any other devices is there we do not keep trying in wait loop.

Revision history for this message
Deepak Mourya (mourya007) wrote :

@slaweq This test is now passed on my environment. Can you please check the same on your environment or Is it still reproducable?

python -m testtools.run tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment
Tests running...

Ran 1 test in 61.800s
OK

Revision history for this message
Miguel Lavalle (minsel) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tempest (master)

Reviewed: https://review.openstack.org/578765
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Submitter: Zuul
Branch: master

commit eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 28 13:03:04 2018 +0200

    Add additional log of meta_data devices content

    In TaggedAttachmentsTest.test_tagged_attachment test there is
    now added some additional debug log to check what
    devices are still in meta_data's devices field when it should
    be empty already.

    This patch changes also from handling all exeptions to only
    AssertionError in verify_empty_devices() method.
    If exeption other than AssertionError occur, it will not be silently
    ignored anymore.

    Change-Id: I95016e35102569fba79fc58f9695e480234619dc
    Related-Bug: #1775947

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I just spotted this issue again and it looks that volume device is not removed from this meta_data.json file:
http://logs.openstack.org/69/567369/15/check/tempest-full/528bc75/job-output.txt.gz#_2018-07-19_10_06_09_273919

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tempest (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/586292

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tempest (master)

Reviewed: https://review.openstack.org/586292
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Submitter: Zuul
Branch: master

commit 4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Author: Matt Riedemann <email address hidden>
Date: Thu Jul 26 12:28:48 2018 -0400

    Skip verifying empty devices in test_tagged_attachment until bug is fixed

    We have a non-trivial amount of failures due to bug 1775947 [1]
    and so far attempts at debugging the issue haven't proved
    fruitful, so let's skip the last (problematic) portion of the
    test until we can get a recreate with proper debug to help
    isolate the problem.

    Related nova debug patch: https://review.openstack.org/584032

    [1] http://status.openstack.org/elastic-recheck/#1775947

    Change-Id: I3104041cb66f912727aabd6c5894600dc87a1229
    Related-Bug: #1775947

Changed in tempest:
assignee: Deepak Mourya (mourya007) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tempest (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/742444

Revision history for this message
Martin Kopec (mkopec) wrote :

What's the status here? I see several fixes were merged, is the issue still reproducible? If so, please, share new info (tracebacks, steps to reproduce, links to failed jobs etc) and let's discuss what next, until then I'll mark is as Incomplete.

Changed in tempest:
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for tempest because there has been no activity for 60 days.]

Changed in tempest:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers