vif doesn't exist Error message in tempest runs

[Matthew Treinish]

On a successful tempest run the following error message was in the nova logs:

ERROR nova.network.manager [req-1ef0c266-bec4-4414-ae27-f6937e6baa2a ListServersNegativeTestXML-tempest-117344783-user ListServersNegativeTestXML-tempest-117344783-tenant] Unable to release 10.1.0.2 because vif doesn't exist.

This causes the test job to fail.

Logs are located here:
http://logs.openstack.org/78/60578/2/check/check-tempest-dsvm-full/0cc61b2/

[Abhishek Chanda (abhishek-i)]

Marking as critical since it's a tempest failure

[Matt Riedemann (mriedem)]

There is a recent change in tempest to test_list_servers_negative but I'm not sure it would be a suspect here since it's supposed to fix a race with allow_tenant_isolation=false which shouldn't be set in the gate tests using nova-network (as far as I know).

https://review.openstack.org/#/c/57384/

[Matt Riedemann (mriedem)]

Pushed a patch for an elastic-recheck query here: https://review.openstack.org/#/c/61158/

34 hits in the last 7 days:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiYmVjYXVzZSB2aWYgZG9lc24ndCBleGlzdFwiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc2NyZWVuLW4tbmV0LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2NjkyMTMxNjMwfQ==

[Matt Riedemann (mriedem)]

From the logstash results, it's not just in test_list_servers_negative, it also fails in test_servers.

[Matt Riedemann (mriedem)]

This seems to have really spiked since 1/28, not sure what merged in nova around that time but this is all over the place now.

[Matt Riedemann (mriedem)]

Actually, from logstash nearly 81% of the time this shows up now it's on a successful run, so the query isn't very good anymore.

[Russell Bryant (russellb)]

Testing a revert based on a change that merged 1/28 ... https://review.openstack.org/69987

[Russell Bryant (russellb)]

That patch hit this bug a bunch during test runs while up for review. The spike of this bug started at the time this patch merged.

[Sean Dague (sdague)]

I'm not convinced this is a root cause bug, as we're hitting this 90% of the time on success runs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/69987
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6b7304d8d38c0f04643bdcd031d682c688c91b28
Submitter: Jenkins
Branch: master

commit 6b7304d8d38c0f04643bdcd031d682c688c91b28
Author: Russell Bryant <email address hidden>
Date: Wed Jan 29 16:47:18 2014 -0500

    Revert "Allow deleting instances while uuid lock is held"

    This reverts commit 05f4f7170ab67c20e7f7b3f9f304ccc7ca163490.

    We're seeing a huge increase in the occurrence of bug 1258848 since
    this patch merged. This patch effectively allows a delete to occur,
    even if the creation hasn't finished yet, which would explain the race
    we're seeing leading to the new flood of these errors.

    Related-bug: #1258848
    Change-Id: I6e44ac788426ba41f1674a5df3d04300eb7c05cc

[Matt Riedemann (mriedem)]

I think we can remove the e-r query for this one since this removes the error message:

https://review.openstack.org/#/c/70187/