[compute] "create_test_server" if networks is undefined and more than one network is present

[Rodolfo Alonso]

This problem was detected in "ServersNegativeTestJSON" [1]. When a server is created ("cls.create_test_server"), if a network is not defined but several networks are present in this project, Nova raises the exception "NetworkAmbiguous", a seen in this log snippet:

Sep 13 23:44:15.520980 ubuntu-bionic-limestone-regionone-0011283625 <email address hidden>[27339]: DEBUG nova.network.neutronv2.api [None req-c95ecec2-8d10-4984-8ba9-b608161dd645 tempest-ServersNegativeTestJSON-445859222 tempest-ServersNegativeTestJSON-445859222] validate_networks() for None {{(pid=27340) validate_networks /opt/stack/nova/nova/network/neutronv2/api.py:2251}}
Sep 13 23:44:15.754945 ubuntu-bionic-limestone-regionone-0011283625 <email address hidden>[27339]: INFO nova.api.openstack.wsgi [None req-c95ecec2-8d10-4984-8ba9-b608161dd645 tempest-ServersNegativeTestJSON-445859222 tempest-ServersNegativeTestJSON-445859222] HTTP exception thrown: Multiple possible networks found, use a Network ID to be more specific.

We can see that the network information provided to the server creation is empty but Nova tries to assign a default single network for this server. However Nova does not assign this default network because several networks are present for this project ID. In this case, the server creation should be specific passing the network information.

[1]https://58a87e825b9766115d07-cec36eea8e90c9127fc5a72b798cfeab.ssl.cf2.rackcdn.com/670177/9/check/networking-ovn-tempest-dsvm-ovs-release/b58638a/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682964

[Jakub Libosvar (libosvar)]

BTW it seems the bug I reported some time ago is duplicate of this one: https://bugs.launchpad.net/nova/+bug/1831048

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

I talked to Jakub via IRC and he decided to mark 1831048 as a duplicate of this one, even the current bug is newer than 1831048. I'll continue updating the patch to solve this problem.

Thanks Jakub.

[Matt Riedemann (mriedem)]

There is an AttachInterfacesTestJSON that expects and handles the 400 error so we have to filter that out to get a decent logstash query with mostly failures:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22HTTP%20exception%20thrown%3A%20Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific.%5C%22%20AND%20NOT%20message%3A%5C%22AttachInterfacesTestJSON%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

[Matt Riedemann (mriedem)]

This mostly hits in the nova-next job and one difference with that job from something like tempest-full is that the tempest-full job runs the tempest API tests concurrently and then the tempest scenario tests serially. The nova-next job uses the 'all' tox env target which runs all tests concurrently. The nova-next job runs the tempest.api.compute and most scenario tests concurrently:

      # We use the "all" environment for tempest_test_regex and
      # tempest_black_regex.
      tox_envlist: all
      # Run all compute API tests and most scenario tests at the default
      # concurrency (nproc/2 which is normally 4 in the gate).
      tempest_test_regex: ^tempest\.(scenario|api\.compute)
      # The tempest.scenario.test_network* tests are skipped because they
      # (1) take a long time and (2) are already covered in the
      # tempest-slow* job. If this regex gets more complicated use
      # tempest_test_blacklist.
      tempest_black_regex: ^tempest.scenario.test_network

So there is likely a scenario test that is creating a network at the same time as the compute API tests are running and that's causing the intermittent failures to show up more often in the nova-next job. Another strategy for working around this bug is blacklisting more scenario tests that are creating networks but not doing anything with the compute service so nova-next wouldn't need to run them.

[Matt Riedemann (mriedem)]

tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest.test_qos_min_bw_allocation_basic, which was enabled in the nova-next job this week:

https://review.opendev.org/#/c/697180/

is likely causing the spike in failures in the nova-next job. See this failed job:

https://zuul.opendev.org/t/openstack/build/b7872ac3886e4cc7a060c1376a2a3395/log/job-output.txt#79124

The failing compute api tests are running while that test is going and that scenario test creates a public network, so that's likely the cause of the failure.

The nova-next job should probably blacklist that test until this bug is fixed.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/697491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697491
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=15c59f1897771e696cbce167379b9ac968c255db
Submitter: Zuul
Branch: master

commit 15c59f1897771e696cbce167379b9ac968c255db
Author: Matt Riedemann <email address hidden>
Date: Thu Dec 5 09:55:28 2019 -0500

    Skip test_minbw_allocation_placement in nova-next job

    We just recently enable this scenario test [1] but it's
    causing the nova-next job to intermittently fail because
    the nova-next job runs the tempest compute API and scenario
    tests concurrently (unlike the tempest-full job which runs
    the API tests concurrently and then the scenario tests serially)
    and the scenario test is creating a public shared network which
    causes the compute API tests, when creating a server without
    specifying a network, to fail since nova doesn't know which
    network to use. So for now just blacklist the scenario test
    until the related tempest bug is fixed.

    [1] https://review.opendev.org/#/c/697180/

    Change-Id: Ie0e6136e88bedb2c4c6d0b742319627d58da147e
    Related-Bug: #1844568

[XiaojueGuan (xiaojuegaun)]

what we expect nova to behave when there is no network defined in creating a server while there are more than one network.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/701792

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/701794

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/701797

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/701799

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Fried (<email address hidden>) on branch: master
Review: https://review.opendev.org/701792
Reason: in favor of https://review.opendev.org/701797

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Fried (<email address hidden>) on branch: master
Review: https://review.opendev.org/701794
Reason: This has served its purpose.

We're going to use the big-hammer approach instead https://review.opendev.org/701797

[Ghanshyam Mann (ghanshyammann)]

Tempest Dynamic credential by default create the network for each tenant. For compute tests we disabled it in base class and assumed each test class creating server successfully and require network will request the network resource creation. Any tests creating the server should(must) create network explicitly to avoid this bug which does not harm much.

We can either identify the tests creating server successfully but not passing network to nova and change OR enable the network creation in base class but explicitly disable the network creation in below cases:

1. Not all tests derive from BaseV2ComputeTest create server so does not need network like quotas tests. We have to explicitly mention not to create network in each such class by set_network_resources().

2. Few tests like negative test classes create server but does not need network as they are supposed to fail early by nvoa so we have to explicit disable the network creation for those too.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/701797
Reason: Abandoned in favor of https://review.opendev.org/#/c/701792/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/701799

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Fried (<email address hidden>) on branch: master
Review: https://review.opendev.org/701794
Reason: This passed three in a row with the latest dep PS, which is good enough for me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701792
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=bfaa50f18bfdc02470a44d5e209e66085f0dcb05
Submitter: Zuul
Branch: master

commit bfaa50f18bfdc02470a44d5e209e66085f0dcb05
Author: Eric Fried <email address hidden>
Date: Thu Jan 9 12:04:54 2020 -0600

    Create default network for several compute tests

    Chipping away at the referenced bug, this commit introduces a
    class-level variable on BaseV2ComputeTest that triggers
    setup_credentials to create a default network, and sets it for several
    classes where the problem has been observed. The class variable is so we
    can whack-a-mole with one line instead of four whenever this problem
    pops up.

    Change-Id: Ia34fe3b41261de02918713c2a948e6c56554bf6f
    Partial-Bug: #1844568

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702553
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=1f546532c32a1cade765685eaac180c57258f352
Submitter: Zuul
Branch: master

commit 1f546532c32a1cade765685eaac180c57258f352
Author: Eric Fried <email address hidden>
Date: Tue Jan 14 17:11:58 2020 -0600

    Create default network for more compute tests

    Chipping away at the referenced bug, this commit uses the
    create_default_network class-level variable introduced via [1] to
    trigger setup_credentials to create a default network for
    a handful more test cases in which the problem was observed.

    [1] Ia34fe3b41261de02918713c2a948e6c56554bf6f
    Partial-Bug: #1844568

    Change-Id: Ic38c74c245fddf29826e6ca0cc469f919076355e

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 23.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/704033

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704033
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=e85f029abf864a3a671f25cd6f468e2e186b542f
Submitter: Zuul
Branch: master

commit e85f029abf864a3a671f25cd6f468e2e186b542f
Author: Eric Fried <email address hidden>
Date: Thu Jan 23 10:42:27 2020 -0600

    Create default network for more compute tests

    Chipping away at the referenced bug, this commit uses the
    create_default_network class-level variable introduced via [1] to
    trigger setup_credentials to create a default network for another test
    case in which the problem was observed.

    [1] Ia34fe3b41261de02918713c2a948e6c56554bf6f
    Partial-Bug: #1844568

    Change-Id: Ic500aff5adda66328c292c936eaa097253360ea4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/711049

[melanie witt (melwitt)]

Just saw the failure with the following tests too just now in nova-next [1]:

Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'

tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_suspended_server [0.427611s] ... FAILED
tempest.api.compute.images.test_images.ImagesTestJSON.test_delete_saving_image [0.757112s] ... FAILED
setUpClass (tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON) [0.000000s] ... FAILED

[1] https://b0f9ae0491e974b2315d-45537cc5c7120f43f6e626c6c78dc0c0.ssl.cf5.rackcdn.com/713035/14/gate/nova-next/3bc683b/job-output.txt

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711049
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=7d91b690818016815d688efc4df30c8058a0a45e
Submitter: Zuul
Branch: master

commit 7d91b690818016815d688efc4df30c8058a0a45e
Author: Ghanshyam Mann <email address hidden>
Date: Tue Mar 3 10:21:50 2020 -0600

    Create default network for compute volume and negative metadata tests

    server metadata and volume tests also create server without passing
    the network to nova which further fail if more than one test network exists.

    http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-03-03.log.html#t2020-03-03T15:57:34

    Partial-Bug: #1844568
    Change-Id: I9956946a4309b9a013fd53b0db06718372a9213c

[melanie witt (melwitt)]

Another failure in the nova-next job [1]:

tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps

  File "/opt/stack/tempest/tempest/scenario/test_server_advanced_ops.py", line 50, in test_server_sequence_suspend_resume
  Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'

[1] https://zuul.opendev.org/t/openstack/build/3f24c2428eac43fcb99ff4f5d7087354

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716809

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/721767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721767
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=2c22e283311fc17e787d8fa9d4cfb1c2d51135f7
Submitter: Zuul
Branch: master

commit 2c22e283311fc17e787d8fa9d4cfb1c2d51135f7
Author: Ghanshyam Mann <email address hidden>
Date: Tue Apr 21 21:37:54 2020 -0500

    Create default network for compute rescue tests

    server rescue tests also create server without passing
    the network to nova which further fail if more than one test network exists.

    https://03399ed3bb928f8e37fb-954a4196d912d707c769d8596124df5e
    .ssl.cf1.rackcdn.com/720042/1/check/nova-next/ba84dfc/testr_results.html

    Change-Id: Ic9930172a1b6937cd038707afd4d0f6e3730efa1
    Partial-Bug: #1844568

[Martin Kopec (mkopec)]

After discussion with Ghanshyam we can consider this bug done.

kopecmartin gmann: thanks .. btw, is this still in progress? i see bunch of merged reviews, isn't it done already? https://bugs.launchpad.net/tempest/+bug/1844568 14:22
openstack Launchpad bug 1844568 in tempest "[compute] "create_test_server" if networks is undefined and more than one network is present" [Medium,In progress] - Assigned to Ghanshyam Mann (ghanshyammann) 14:22
gmann kopecmartin: we covered most of the tests and we kept this to find and fix more tests but i have not seen network failure recently. 14:36
gmann kopecmartin: we can close this and if something comes up then just fix it. 14:36
gmann without this bug link or just with ref to bug 14:36
kopecmartin ok, makes sense

[Martin Kopec (mkopec)]

Seems like this is not fixed everywhere:

https://zuul.opendev.org/t/openstack/build/f1ae8eeb986a42e2b45d1a79e2e300e2

[Ghanshyam Mann (ghanshyammann)]

nova-next job again hitting it for scenario tests

https://zuul.opendev.org/t/openstack/build/ce402610f88a45928ff808bbb4a6303f

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2021-01-07 15:29:24,382 102340 INFO [tempest.lib.common.rest_client] Request (TestServerAdvancedOps:test_server_sequence_suspend_resume): 201 POST https://10.210.33.154/identity/v3/auth/tokens 0.065s
2021-01-07 15:29:24,383 102340 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
        Body: <omitted>
    Response - Headers: {'date': 'Thu, 07 Jan 2021 15:29:24 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '3712', 'x-subject-token': '<omitted>', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-62b32058-d22f-416f-bd88-439b0bfdae75', 'connection': 'close', 'status': '201', 'content-location': 'https://10.210.33.154/identity/v3/auth/tokens'}
        Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "8e0b42bf518043e689a20a3744dfe658", "name": "tempest-TestServerAdvancedOps-806623797", "password_expires_at": null}, "audit_ids": ["Sh-4IigZToOJw0rdssZI8g"], "expires_at": "2021-01-07T16:29:24.000000Z", "issued_at": "2021-01-07T15:29:24.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "99af8a86968f4cf882b52d3c3aefe402", "name": "tempest-TestServerAdvancedOps-806623797"}, "is_domain": false, "roles": [{"id": "346a280b9ee44a858f45b11e7d4a421a", "name": "member"}, {"id": "567eafdc17584b328fe001d552a7f324", "name": "reader"}], "catalog": [{"endpoints": [{"id": "36d0a9d7f4cb414fbc378efa743cb59f", "interface": "public", "region_id": "RegionOne", "url": "https://10.210.33.154/compute/v2/99af8a86968f4cf882b52d3c3aefe402", "region": "RegionOne"}], "id": "0fbb134a690b47d3ba2dce69a73b1f28", "type": "compute_legacy", "name": "nova_legacy"}, {"endpoints": [{"id": "d2a910f62abc400ea54e0ce0ad3eea30", "interface": "public", "region_id": "RegionOne", "url": "https://10.210.33.154/volume/v2/99af8a86968f4cf882b52d3c3aefe402", "region": "RegionOne"}], "id": "1140c3aa77ed4d5eb00ac74882f7056d", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "bb3bb75c92c949a993ead2517b755909", "interface": "public", "region_id": "RegionOne", "url": "https://10.210.33.154/placement", "region": "RegionOne"}], "id": "1f357359dd744a6d8b3dff1008d0d89a", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "aed68456925a47e9aa9c530d946b5dbb", "interface": "public", "region_id": "RegionOne", "url": "https://10.210.33.154/identity", "region": "RegionOne"}, {"id": "cbff0f9b243c480bb40d5fa096d006c0", "interface": "admin", "region_id": "RegionOne", "url": "https://10.210.33.154/identity", "region": "RegionOne"}], "id": "22d037f8edb848cb9efda32c7a310acc", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "5320ee4fc57240d69ab0e254120712d7", "interface": "admin", "region_id": "RegionOne", "url": "https://10.210.33.154:8080", "region": "RegionOne"}, {"id": "dfd66b04916f4708a4e4f490feb55f49", "interface": "public", "region_id": "RegionOne", "url": "https://10.210.33.154:8...

[Ghanshyam Mann (ghanshyammann)]

By debugging more it for scenario tests (where I thought I cannot occur as scenario tests do create a network for each test) I found it can still happen for scenario test too when 'port_vnic_type' and 'port_profile' are not configured in Tempest.

If these two config are not present then Tempest scenario manager does not fetch the network from neutron to pass in nova post server API.
- https://github.com/openstack/tempest/blob/fec2c93cdcc14ad08d0a35136ee287525e7a4879/tempest/scenario/manager.py#L242

In that case, tenant network is used, which is not present as scenario manager does not create the default network resource for credential and tenant.

- https://github.com/openstack/tempest/blob/fec2c93cdcc14ad08d0a35136ee287525e7a4879/tempest/scenario/manager.py#L298

We need to setup the default network resource in ScenarioTest so that a specific network can be passed to nova if test does not create their own network.

[Ghanshyam Mann (ghanshyammann)]

here is the log where no network is passed for failing tests test_server_sequence_suspend_resume

2021-01-07 15:29:24.383 102340 INFO tempest.lib.common.fixed_network [-] (TestServerAdvancedOps:test_server_sequence_suspend_resume) Found network None available for tenant

https://zuul.opendev.org/t/openstack/build/ce402610f88a45928ff808bbb4a6303f/log/controller/logs/tempest_log.txt#53439

[Ghanshyam Mann (ghanshyammann)]

fix for scenario tests which should fix this bug completely.

https://review.opendev.org/c/openstack/tempest/+/770169

[Ghanshyam Mann (ghanshyammann)]

marking invalid from nova side as no fix needed on nova side for this.

[Martin Kopec (mkopec)]

https://review.opendev.org/c/openstack/tempest/+/770169 got merged.

[melanie witt (melwitt)]

This bug is back [1]:

ft1.1: setUpClass (tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/test.py", line 181, in setUpClass
    raise value.with_traceback(trace)
  File "/opt/stack/tempest/tempest/test.py", line 174, in setUpClass
    cls.resource_setup()
  File "/opt/stack/tempest/tempest/api/compute/images/test_list_image_filters.py", line 109, in resource_setup
    cls.server1 = cls.create_test_server()
  File "/opt/stack/tempest/tempest/api/compute/base.py", line 266, in create_test_server
    body, servers = compute.create_test_server(
  File "/opt/stack/tempest/tempest/common/compute.py", line 197, in create_test_server
    body = clients.servers_client.create_server(name=name, imageRef=image_id,
  File "/opt/stack/tempest/tempest/lib/services/compute/servers_client.py", line 105, in create_server
    resp, body = self.post('servers', post_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/opt/stack/tempest/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'code': 409, 'message': 'Multiple possible networks found, use a Network ID to be more specific.'}

[1] https://zuul.opendev.org/t/openstack/build/87cdec6b52d54cbfb0b314a0ee13fc21/log/job-output.txt#63828

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/846419

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/846419
Committed: https://opendev.org/openstack/tempest/commit/de258f9956dd5eb4f57a4031c5785d5f2664bc0f
Submitter: "Zuul (22348)"
Branch: master

commit de258f9956dd5eb4f57a4031c5785d5f2664bc0f
Author: Balazs Gibizer <email address hidden>
Date: Fri Jun 17 17:01:10 2022 +0200

    Create default net in ImagesNegativeTestJSON

    As the test result [1] showed we missed ImagesNegativeTestJSON when
    fixed https://bugs.launchpad.net/tempest/+bug/1844568

    Related-Bug: #1844568

    [1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7a3/846187/3/check/nova-next/7a30291/testr_results.html

    Change-Id: Ibd12542ee08f5020eddafcb253a978ac50f9e53a

[yatin (yatinkarel)]

Few more instances being fixed with https://review.opendev.org/c/openstack/tempest/+/890242