test_create_router_set_gateway_with_fixed_ip is flaky

Bug #1676207 reported by Jordan Pittier on 2017-03-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
High
Unassigned

Bug Description

test_create_router_set_gateway_with_fixed_ip starts by allocating an IP from the public_network, then it deletes that IP (which releases the IP back to the pool), then it tries to create a router with a fixed external IP set to the IP value it got earlier.

But in the mean time, another test could have claimed the very same IP because there's a fraction of time when the IP is again available. In that case, the create router call fails with

Body: {"NeutronError": {"message": "IP address 172.24.5.2 already allocated in subnet 41f6fb6a-f10d-4a9a-9b9b-235bbbde0891", "type": "IpAddressAlreadyAllocated", "detail": ""}}

Jordan Pittier (jordan-pittier) wrote :
Prateek Arora (parora) wrote :

Hi Jordan

Thanks for reporting the issue. I would like to know how did you hit this bug.

My guess is you put a sleep at the end of test and in the meanwhile claimed the IP using command line.
Can you please confirm this or if you did something please let us know.

Changed in tempest:
status: New → Incomplete
tags: added: gate-failure
Changed in tempest:
importance: Undecided → High

Reviewed: https://review.openstack.org/450032
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=7252ce0789d6eaaa14474d853a50f25ef5c9878a
Submitter: Jenkins
Branch: master

commit 7252ce0789d6eaaa14474d853a50f25ef5c9878a
Author: Jordan Pittier <email address hidden>
Date: Sun Mar 26 20:48:14 2017 +0200

    Skip flaky test_create_router_set_gateway_with_fixed_ip

    test_create_router_set_gateway_with_fixed_ip starts by allocating an IP
    from the public_network, then it deletes that IP (which releases the
    IP back to the pool), then it tries to create a router with a fixed
    external IP set to the IP value it got earlier.

    But in the mean time, another test could have claimed the very same IP
    because there's a fraction of time when the IP is again available. In
    that case, the create router call fails with

    Body: {"NeutronError": {"message": "IP address 172.24.5.2 already allocated
    in subnet 41f6fb6a-f10d-4a9a-9b9b-235bbbde0891", "type":
    "IpAddressAlreadyAllocated", "detail": ""}}

    Change-Id: I9761b094351709f97e5b1524935d070796c64273
    Related-Bug: #1676207

Changed in tempest:
status: Confirmed → In Progress
zhufl (zhu-fanglei) on 2017-07-14
Changed in tempest:
status: In Progress → Fix Committed
Dr. Jens Harbott (j-harbott) wrote :

The test is being skipped, but I don't think that this counts as a fix. Can we unskip it an see whether we still are hitting this issue?

Changed in tempest:
status: Fix Committed → New
zhufl (zhu-fanglei) wrote :

"because there's a fraction of time when the IP is again available", it seems the fraction is very small, so I also don't understand why this testcases will fail so often. I guess, maybe ip is not released immediately after delete_port?

Slawek Kaplonski (slaweq) wrote :

I agree with Jens, maybe we can revert this skip and check Neutron logs when this issue happens again to check what is the root cause of this already allocated IP address

Any update so far on this issue?

Dr. Jens Harbott (j-harbott) wrote :

I think the proper fix would be to create a dedicated network in this test and use that as gateway instead of the public network. That will avoid any races.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments