Neutron jobs won't pass tempest running in parallel

[Matthew Treinish]

When running tempest in parallel and neutron is enabled the tests do not work. You hit this failure:

ft5.1: setUpClass (tempest.api.volume.test_volumes_actions.VolumesActionsTest)_StringException: Traceback (most recent call last):
  File "tempest/api/volume/test_volumes_actions.py", line 37, in setUpClass
    cls.flavor_ref)
  File "tempest/services/compute/json/servers_client.py", line 81, in create_server
    resp, body = self.post('servers', post_body, self.headers)
  File "tempest/common/rest_client.py", line 259, in post
    return self.request('POST', url, headers, body)
  File "tempest/common/rest_client.py", line 387, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 437, in _error_checker
    raise exceptions.BadRequest(resp_body)
BadRequest: Bad request
Details: {u'badRequest': {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}}

(It occurs on a few tests but the error message is the same, see the sample logs below.)

Logs can be found here: http://logs.openstack.org/97/41797/1/check/gate-tempest-devstack-vm-neutron/5702aa5/

Additionally, by default in devstack tenant isolation is disabled when neutron is enabled. This is a requirement for running tempest in parallel. If parallel tempest is going to be used for neutron gate jobs this needs to be changed

[Attila Fazekas (afazekas)]

This looks like a duplicate of https://bugs.launchpad.net/tempest/+bug/1092009.

[Miguel Lavalle (minsel)]

Mathew,

Did you add a network to the tennant you are using to create this server? That error happens when you boot a server, you don't specify networks in the boot operation and the tenant has acces to 2 or more networks. In that situation, nova doesn't know what network to connect the server to.

In any case, I have a proposal to solve the tenant isolation / parallel execution situation. It boilds down to implement in devstack the use case "Single Flat Network", as described here

http://docs.openstack.org/grizzly/openstack-network/admin/content/use_cases_single_flat.html

and here

http://docs.openstack.org/grizzly/openstack-network/admin/content/demo_flat_logical_network_config.html

 A devstack instance, when created with neutron as the networking service, creates a network called 'private' for tenant 'demo'. That network is private to that tenant. We need to make that network shared. For this to happen, we have to change in devstack/lib/neutron line 332 from:

NET_ID=$(neutron net-create --tenant_id $TENANT_ID "$PRIVATE_NETWORK_NAME" | grep ' id ' | get_field 2)

to:

NET_ID=$(neutron net-create --tenant_id $TENANT_ID --shared "$PRIVATE_NETWORK_NAME" | grep ' id ' | get_field 2)

Once we do this, I think we can enable tenant isolation in Tempest and all the tenants that we create will have access to this shared network. And the end result is that we will be able to run Tempest in parallel.

I just created a devstack instance in my home lab. It works fine. I tested the shared network booting servers with tenants demo and alt_demo and it works fine.

I didn't try parallel execution because I am still running nosetests. But you can test this idea tomorrow. I will ping you as soon as I get to the office

If this works, we will also have to change line 328 in devstack/lib/neutron to create a shared network. Also, over time, we may want to change the name of the network from private to something else.

[Attila Fazekas (afazekas)]

Thank you.

I do not know what is the reason behind the 'private' network does not created as shared at the first place.

The other option is adding new option for tempest for handling the environment without shared network and create a router+net+subnet for every 'randomly' created tenant.

It can be made more smarter by adding '_new_net_for' set for every test class with the , 'primary', 'admin', 'alt' , and just create networks for the explicitly mentioned tenants.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45578

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45578
Committed: http://github.com/openstack/tempest/commit/b8fabc5614f2648a297cb3933723cf37e0418861
Submitter: Jenkins
Branch: master

commit b8fabc5614f2648a297cb3933723cf37e0418861
Author: Miguel Lavalle <email address hidden>
Date: Fri Aug 23 11:19:57 2013 -0500

    Handling network resources in tenant isolation

    Functionality is added to create and delete isolated network, subnet and router
    when isolated credentials are created and deleted for a primary, admin or alt
    tenant.

    When a set of isolated credentials are created, the tenant gets also a private
    network, subnet and router. The router connects the tenat's network to the
    public network designated by public_network_id in etc/tempest.conf

    Change-Id: I7587c85017cca09f7a67eae0670f67b2bceacb60
    Fixes: bug #1216076

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/46314
Committed: http://github.com/openstack-dev/devstack/commit/3d84cf2d7c323750971cf2d27f3a4eaa26cb7a9f
Submitter: Jenkins
Branch: master

commit 3d84cf2d7c323750971cf2d27f3a4eaa26cb7a9f
Author: Matthew Treinish <email address hidden>
Date: Thu Sep 12 13:25:54 2013 -0400

    Enable tenant isolation to tempest for neutron

    This commit re-enables tenant isolation in tempest for neutron. This
    is a requirement for running tempest in parallel. This commit depends
    on tempest change I7587c85017cca09f7a67eae0670f67b2bceacb60

    Fixes bug 1216076

    Change-Id: I63a30bacd48cecd110fb90e1fc718249c2b1904b