DefaultSubnetPoolsTest is racy

Bug #1709938 reported by Jakub Libosvar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Jakub Libosvar

Bug Description

Default subnet can exist only once in cloud and there are two tests that create default and one that updates to default, it happens that tests are running in parallel while the check for existing subnet is at the class level. So it happens that:

 1) class checks for default subnet, it's not there
 3) test1 creates default subnet -> it's fine, we now have our unique resource
 4) test2 creates default subnet -> the error we see cause test1 has the default

From the tempest logs:

Step1:
2017-08-10 07:03:12.341 3008 INFO tempest.lib.common.rest_client [req-07271c2f-6725-4f77-b676-a55a95adbf7b ] Request (DefaultSubnetPoolsTest:setUpClass): 200 GET http://10.0.0.103:9696/v2.0/subnetpools 0.418s
ion/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '18', 'content-location': 'http://10.0.0.103:9696/v2.0/subnetpools', u'date': 'Thu, 10 Aug 2017 11:03:12 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-07271c2f-6725-4f77-b676-a55a95adbf7b'}
        Body: {"subnetpools":[]} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:425

Step2:
2017-08-10 07:03:12.998 3008 INFO tempest.lib.common.rest_client [req-6322524f-d1d8-4c7c-abd4-f08862bcec60 ] Request (DefaultSubnetPoolsTest:test_admin_create_default_subnetpool): 201 POST http://10.0.0.103:9696/v2.0/subnetpools 0.655s
2017-08-10 07:03:12.998 3008 DEBUG tempest.lib.common.rest_client [req-6322524f-d1d8-4c7c-abd4-f08862bcec60 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"subnetpool": {"is_default": true, "prefixes": ["10.11.12.0/24"], "name": "tempest-smoke-subnetpool-2026337716", "min_prefixlen": "29"}}
    Response - Headers: {'status': '201', u'content-length': '508', 'content-location': 'http://10.0.0.103:9696/v2.0/subnetpools', u'date': 'Thu, 10 Aug 2017 11:03:12 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-6322524f-d1d8-4c7c-abd4-f08862bcec60'}
        Body: {"subnetpool":{"is_default":true,"description":"","default_quota":null,"tenant_id":"542c5acbca3f49a0bc89d0903eb5c7e5","created_at":"2017-08-10T11:03:12Z","tags":[],"updated_at":"2017-08-10T11:03:12Z","prefixes":["10.11.12.0/24"],"min_prefixlen":"29","max_prefixlen":"32","address_scope_id":null,"revision_number":0,"ip_version":4,"shared":false,"default_prefixlen":"29","project_id":"542c5acbca3f49a0bc89d0903eb5c7e5","id":"dd1b15f4-0dc1-4582-9435-394a5b2bdea9","name":"tempest-smoke-subnetpool-2026337716"}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:425

Step3:
2017-08-10 07:03:15.667 3008 INFO tempest.lib.common.rest_client [req-24e48bfe-473f-44e0-aaf4-8f2debf81a0e ] Request (DefaultSubnetPoolsTest:test_convert_subnetpool_to_default_subnetpool): 400 PUT http://10.0.0.103:9696/v2.0/subnetpools/fb199e24-a9e2-443f-81cc-3c07c3bd7a20 0.842s
2017-08-10 07:03:15.668 3008 DEBUG tempest.lib.common.rest_client [req-24e48bfe-473f-44e0-aaf4-8f2debf81a0e ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"subnetpool": {"is_default": true}}
    Response - Headers: {'status': '400', u'content-length': '203', 'content-location': 'http://10.0.0.103:9696/v2.0/subnetpools/fb199e24-a9e2-443f-81cc-3c07c3bd7a20', u'date': 'Thu, 10 Aug 2017 11:03:15 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-24e48bfe-473f-44e0-aaf4-8f2debf81a0e'}
        Body: {"NeutronError": {"message": "Invalid input for operation: A default subnetpool for this IP family has already been set. Only one default may exist per IP family.", "type": "InvalidInput", "detail": ""}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:425

Revision history for this message
Boden R (boden) wrote :

I was trying to get a handle on potential priority for this one, but I can't find any failures from it in the last month [1]. Is this for a new test, or perhaps I'm missing something in [1] to locate existing failures?

[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22default%20subnetpool%20for%20this%20IP%20family%20has%20already%20been%20set%5C%22

Changed in neutron:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/492653

Changed in neutron:
status: Triaged → In Progress
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I am not sure what is the root cause of the race. These tests were introduced by [1], but that couldn't have happened in the gate [2], because the default subnetpool is already available to test 'get-me-a-network'.

[1] https://review.openstack.org/#/c/468257/
[2] http://logs.openstack.org/57/468257/6/gate/gate-neutron-dsvm-api-ubuntu-xenial/a8bd940/testr_results.html.gz

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

...and thus the tests are skipped.

Changed in neutron:
importance: Undecided → High
milestone: none → pike-rc1
importance: High → Undecided
milestone: pike-rc1 → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/492653
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=637734c1b6d5a158374576ce27941a04363fc8bb
Submitter: Jenkins
Branch: master

commit 637734c1b6d5a158374576ce27941a04363fc8bb
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 10 17:04:45 2017 +0000

    Fix DefaultSubnetPool API test

    As default subnetpool is a unique resource in the cloud, it needs to be
    cleaned after each test is done. This patch adds a cleanup call to
    DefaultSubnetPool tests in order to delete created default subnet pool.

    Change-Id: I4c963d0d0e9910f7047061b51feb36c8a19de65c
    Closes-bug: #1709938

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/495467

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/495467
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f010efcd0965690ca603dcec0b387a3600b98abe
Submitter: Jenkins
Branch: stable/pike

commit f010efcd0965690ca603dcec0b387a3600b98abe
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 10 17:04:45 2017 +0000

    Fix DefaultSubnetPool API test

    As default subnetpool is a unique resource in the cloud, it needs to be
    cleaned after each test is done. This patch adds a cleanup call to
    DefaultSubnetPool tests in order to delete created default subnet pool.

    Change-Id: I4c963d0d0e9910f7047061b51feb36c8a19de65c
    Closes-bug: #1709938
    (cherry picked from commit 637734c1b6d5a158374576ce27941a04363fc8bb)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.1

This issue was fixed in the openstack/neutron 11.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.0.0b1

This issue was fixed in the openstack/neutron 12.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.