create_test_server doesn't specify default network, so fails in multi network environments

[Derek Higgins]

In tripleo we have two networks on the overcloud

# neutron net-show default-net
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | cd8b1087-068a-4510-937c-9d2b2bbcf034 |
| name | default-net |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | False |
| shared | True |
| status | ACTIVE |
| subnets | f9e73be1-43da-493a-907e-bfdb6a9d79e3 |
| tenant_id | 9569ac5250b04c73b220b87bc43870de |
+---------------------------+--------------------------------------+
# neutron net-show ext-net
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 8e4d4115-c1a1-404b-aa6f-d3241f12b3a0 |
| name | ext-net |
| provider:network_type | local |
| provider:physical_network | |
| provider:segmentation_id | |
| router:external | True |
| shared | False |
| status | ACTIVE |
| subnets | ca360565-9220-4706-bb7c-c9215f1f3273 |
| tenant_id | 9569ac5250b04c73b220b87bc43870de |
+---------------------------+--------------------------------------+

The admin user has access to both networks and needs to specify which network to start a instance on

This Errors
# nova boot --image cirros --flavor 1 qt
ERROR: Multiple possible networks found, use a Network ID to be more specific. (HTTP 400) (Request-ID: req-7e99b3f7-c201-4295-9cbf-e7c6430f9818)

This works
# nova boot --image cirros --flavor 1 --nic net-id=cd8b1087-068a-4510-937c-9d2b2bbcf034 qt

In tempest we have the same problem for example (there are others)
# testr run tempest.api.compute.admin.test_quotas.QuotasAdminTestJSON
  File "/opt/stack/tempest/tempest/services/compute/json/servers_client.py", line 82, in create_server
Details: {u'badRequest': {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}}

From what I can see its because BaseComputeTest.create_test_server doesn't specify a network, if I hard code the network id into create_server the test passes
diff --git a/tempest/api/compute/base.py b/tempest/api/compute/base.py
index d9ede31..0b0105f 100644
--- a/tempest/api/compute/base.py
+++ b/tempest/api/compute/base.py
@@ -101,7 +101,7 @@ class BaseComputeTest(tempest.test.BaseTestCase):
         image_id = kwargs.get('image_id', cls.image_ref)

         resp, body = cls.servers_client.create_server(
- name, image_id, flavor, **kwargs)
+ name, image_id, flavor, networks=[{"uuid": "cd8b1087-068a-4510-937c-9d2b2bbcf034"}], **kwargs)

         # handle the case of multiple servers
         servers = [body]

Should this option be exposed in the config file? e.g. default network for instance if none is provided?

[Robert Collins (lifeless)]

Isn't the answer 'don't run tempest as admin' ?

[Derek Higgins (derekh)]

Seting status to wishlist and will disable these tests for tripleo.

[Zhikun Liu (zhikunliu)]

default-net is a shared network, so testing user could see this network.
| shared | True |

To fix the problem:
neutron net-update default-net --shared=false

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/61037

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/68904
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=9c052aade86409327e03bbcdf98afb4a579712f4
Submitter: Jenkins
Branch: master

commit 9c052aade86409327e03bbcdf98afb4a579712f4
Author: JordanP <email address hidden>
Date: Fri Jan 24 13:05:00 2014 +0000

    Scenario : Start instances using fixed network when possible

    Uses the fixed_network_name option to select which network
    to boot test instances on. This allows Tempest to be run
    in environments where there are multiple networks available.

    Change-Id: I64ad6ce6bee3fe33537ac4f8d8d913856fe46554
    Partial-Bug: #1250866

[Victor Stinner (vstinner)]

Hi, we have this issue (message "Multiple possible networks found, use a Network ID to be more specific") at eNovance in our internal Continuous Integration service (basically Jenkins running Tempest tests). We have 2 networks "nova" and "private". I'm working on a fix for the following test:
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip

Then I will try to fix other tests raising the same error:

test_add_remove_fixed_ip
test_admin_delete_servers_of_others
test_aggregate_add_host_as_user
test_aggregate_add_host_create_server_with_az
test_authorization
test_create_image_from_deleted_server
test_create_image_from_stopped_server
test_create_list_show_delete_interfaces
test_create_server
test_create_server_with_admin_password
test_create_specify_keypair
test_create_with_existing_server_name
test_delete_active_server
test_delete_saving_image
test_delete_server_while_in_attached_volume
test_delete_server_while_in_building_state
test_delete_server_while_in_error_state
test_delete_server_while_in_pause_state
test_delete_server_while_in_shelved_state
test_delete_server_while_in_shutoff_state
test_disk_config.ServerDiskConfigTestJSON
test_disk_config.ServerDiskConfigTestXML
test_floating_ips_actions
test_image_metadata
test_images.ImagesV3Test.test_delete_queued_image
test_images.ListSnapshotImagesTest
test_images_oneserver
test_instance_actions
test_list_image_filters
test_list_server_filters
test_list_servers_negative
test_multiple_create
test_rebuild_server_with_auto_disk_config
test_rebuild_server_with_manual_disk_config
test_resize_server_from_auto_to_manual
test_resize_server_from_manual_to_auto
test_server_actions
test_server_addresses_negative.ServerAddressesNegativeTestJSON
test_server_addresses_negative.ServerAddressesNegativeTestXML
test_server_addresses_negative.ServerAddressesV3NegativeTest
test_server_addresses.ServerAddressesTestJSON
test_server_addresses.ServerAddressesTestXML
test_server_addresses.ServerAddressesV3Test
test_server_metadata
test_server_password.ServerPasswordTestJSON
test_server_password.ServerPasswordTestXML
test_server_password.ServerPasswordV3Test
test_server_personality
test_server_rescue
test_server_rescue_negative
test_server_rescue_negative.ServerRescueNegativeTestJSON
test_server_rescue_negative.ServerRescueNegativeTestXML
test_server_rescue_negative.ServerRescueNegativeV3Test
test_server_rescue.ServerRescueTestJSON
test_server_rescue.ServerRescueTestXML
test_server_rescue.ServerRescueV3Test
test_servers
test_server_security_groups
test_servers_negative
test_simple_tenant_usage
test_snapshot_create_with_volume_in_use
test_update_access_server_address
test_update_server_name
test_virtual_interfaces.VirtualInterfacesTestJSON
test_virtual_interfaces.VirtualInterfacesTestXML
test_volumes_actions
test_volumes_negative

[Ben Nemec (bnemec)]

Victor,

I actually just revisited this last Friday, and I'm hoping to be able to push a new version of the fix soon. I'm verifying that https://review.openstack.org/#/c/61037 still works and then I'll ask someone to reopen the change so I can update it to address the comments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96131

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96135

[Victor Stinner (vstinner)]

Ben: I completly missed your patch :-/ It looks almost done and it fixes more issues than my patch https://review.openstack.org/96131 . Do you still plan to work on the patch (ex: to address my issues)?

My second patch is still needed ( https://review.openstack.org/96135 ).

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/61037
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/96135
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/96131
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Ben Nemec (bnemec)]

I'm afraid this has gotten pushed far enough down my priority list that I don't think I'll be able to continue with it any time soon. If someone wants to take over https://review.openstack.org/#/c/61037/ now I would appreciate it.

[Catherine Diep (cdiep)]

Ben, I am a contributor to the refstack-client stackforge project. Many of the DefCore defined potential "core" tests would fail for "Multiple possible networks found" in the test clouds that have shared networks. We would like to re-open and work on https://review.openstack.org/#/c/61037/.

[git-harry (git-harry)]

@Simeon Monov - this bug is a blocker where I work. How long do you think it will take you to get a new patch set uploaded? Do you need any assistance with this?

[Evan Callicoat (diopter)]

In case it's useful to anyone affected by this bug, we did find one way to ease the pain a bit when you're working with a simple two network setup, with one external and one internal. Here's the steps to workaround this issue without changing upstream tempest:

* Make the external (external=True) network not-shared (public=False) and owned by the admin tenant
* Make the internal ("tenant network") be owned by the demo tenant
* Configure tempest to use the demo tenant for testing
* Configure tempest to use tenant isolation (allow_tenant_isolation = true)

Because the external network isn't shared and is owned by another tenant, the demo tenant can see it but can't use it (due to policy restrictions), so nova boot doesn't get confused about which network to use since only one is available to it. This also doesn't preclude tempest from assigning floats from the external network because it does so as admin, which isn't restricted from attaching floats to other tenants' instances.

This has been tested using our AIO script on our own servers as well as on openstack-ci and tempest is successful, as seen here for example: http://logs.openstack.org/53/152753/13/check/check-os-ansible-deployment-dsvm-aio-build-voting/358a516/console.html#_2015-02-06_20_58_16_908

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/159776

[yuchuan zhang (yuchuan-zhang)]

when to merge this fix to master?

[git-harry (git-harry)]

@yuchuan zhang - I can't say if or when that patch will go it. It is still being reviewed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159776
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=9ce97dfa353e3659aa0ec0a7f62ed3a7e54c4bdf
Submitter: Jenkins
Branch: master

commit 9ce97dfa353e3659aa0ec0a7f62ed3a7e54c4bdf
Author: Rohan Kanade <email address hidden>
Date: Tue Dec 10 18:59:35 2013 +0530

    Start instances using fixed network when possible

    Uses the fixed_network_name option to select which network
    to boot test instances on. This allows Tempest to be run
    in environments where there are multiple networks available.

    This problem does not apply to nova-network environments because
    it will happily start instances with multiple networks attached,
    so the behavior is left as before in that case.

    Co-Authored-By: Ben Nemec <email address hidden>
    Co-Authored-By: Vincent Untz <email address hidden>
    Co-Authored-By: git-harry <email address hidden>

    Change-Id: Ib5b84b59e3d182d8b9cc83954537c32f3eb9e388
    Closes-Bug: #1250866

[Yulia (j-kopkova)]

Hi, we observe the issue with "Multiple possible networks found, use a Network ID to be more specific" at Verizon in our internal CI/CD service.
Tempest tests affected:
tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest
tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest
tempest.api.compute.servers.test_create_server.ServersTestJSON
tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON

[Matthew Treinish (treinish)]

So, this bug was closed over 6months ago and you should have all the knobs necessary to configure things to specify the correct network when creating a server. It is quite likely the issue you're hitting is just a tempest configuration issue. I recommend reading:

http://docs.openstack.org/developer/tempest/configuration.html#networking

which covers how to configure tempest in different network configurations.

If you still have trouble I recommend asking in the #openstack-qa IRC channel and we can work through specifically what is happening in your use case.

[Steven Hardy (shardy)]

This bug was reported against an old version of TripleO, and may no longer be valid.

Since it was reported before the start of the liberty cycle (and our oldest stable
branch is stable/liberty), I'm marking this incomplete.

Please reopen this (change the status from incomplete) if the bug is still valid
on a current supported (stable/liberty, stable/mitaka or trunk) version of TripleO,
thanks!

[Emilien Macchi (emilienm)]

This bug is > 365 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.