Functional tests (with neutron) failing with "ERROR (Conflict): Multiple possible networks found, use a Network ID to be more specific. (HTTP 409)" since 1/6

[Matt Riedemann]

I'm seeing unrelated patches failing with this since 1/6:

http://logs.openstack.org/65/417265/2/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/d351b68/testr_results.html.gz

ft3.1: novaclient.tests.functional.v2.test_servers.TestServersBootNovaClient.test_boot_server_with_legacy_bdm_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "novaclient/tests/functional/v2/legacy/test_servers.py", line 55, in test_boot_server_with_legacy_bdm
    self._boot_server_with_legacy_bdm(bdm_params=params)
  File "novaclient/tests/functional/v2/legacy/test_servers.py", line 44, in _boot_server_with_legacy_bdm
    "bdm_params": bdm_params}))
  File "novaclient/tests/functional/base.py", line 315, in nova
    endpoint_type, merge_stderr)
  File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 130, in nova
    'nova', action, flags, params, fail_ok, merge_stderr)
  File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 374, in cmd_with_auth
    self.cli_dir, prefix=self.prefix)
  File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 71, in execute
    result_err)
tempest.lib.exceptions.CommandFailed: Command '['/opt/stack/new/python-novaclient/.tox/functional/bin/nova', '--os-username', 'admin', '--os-tenant-name', 'admin', '--os-password', 'secretadmin', '--os-auth-url', 'http://10.18.197.89/identity_admin', '--os-compute-api-version', '2.latest', '--os-endpoint-type', 'publicURL', 'boot', '127e19d5-d202-494a-9211-853e82713810', '--flavor', '1', '--poll', '--block-device-mapping', 'vda=61cdaf41-aa40-4bb7-aef5-bcc7392f9256:::1']' returned non-zero exit status 1.
stdout:

stderr:
/opt/stack/new/python-novaclient/novaclient/client.py:272: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
ERROR (Conflict): Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-de79de6b-5a7d-49f9-b7c5-985bd6165d34)

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%20(Conflict)%3A%20Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific.%20(HTTP%20409)%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fpython-novaclient%5C%22&from=7d

160 hits since 1/6, check and gate, all failures.

[Matt Riedemann (mriedem)]

Revert of a possible related devstack change to see if that fixes it:

https://review.openstack.org/#/c/417643/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/417645

[Matt Riedemann (mriedem)]

These are the multiple networks when this fails:

http://logs.openstack.org/45/417645/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/cc21d78/logs/screen-n-api.txt.gz#_2017-01-08_03_35_22_386

[
   {
      u'provider: physical_network':u'public',
      u'ipv6_address_scope':None,
      u'revision_number':8,
      u'port_security_enabled':True,
      u'provider: network_type':u'flat',
      u'id':u'02df12e9-4b26-48f1-a1c3-993f6461c158',
      u'router: external':True,
      u'availability_zone_hints':[

      ],
      u'availability_zones':[
         u'nova'
      ],
      u'ipv4_address_scope':None,
      u'shared':False,
      u'project_id':u'4c62a3fde1ba4ad3b03ddf8a986a5d88',
      u'status':u'ACTIVE',
      u'subnets':[
         u'5bc09e37-0404-455d-a9fe-63ac5bf2d97c',
         u'9d5d5fc4-884f-47bf-ae23-77f7195f115f'
      ],
      u'description':u'',
      u'tags':[

      ],
      u'updated_at': u'2017-01-08T03:28:16 Z',
      u'is_default':True,
      u'provider: segmentation_id':None,
      u'name':u'public',
      u'admin_state_up':True,
      u'tenant_id':u'4c62a3fde1ba4ad3b03ddf8a986a5d88',
      u'created_at': u'2017-01-08T03:28:03 Z',
      u'mtu':1500
   },
   {
      u'provider: physical_network':None,
      u'ipv6_address_scope':None,
      u'revision_number':7,
      u'port_security_enabled':True,
      u'provider: network_type':u'vxlan',
      u'id':u'6d8eaf5f-a1f9-442d-aa41-e9b600d3d7d3',
      u'router: external':False,
      u'availability_zone_hints':[

      ],
      u'availability_zones':[
         u'nova'
      ],
      u'ipv4_address_scope':None,
      u'shared':False,
      u'project_id':u'4c62a3fde1ba4ad3b03ddf8a986a5d88',
      u'status':u'ACTIVE',
      u'subnets':[
         u'75dffadb-322c-4c35-bcfe-dbdb965fa884',
         u'05af59d7-caf3-483d-a8ff-a1438649d73a'
      ],
      u'description':u'',
      u'tags':[

      ],
      u'updated_at': u'2017-01-08T03:27:55 Z',
      u'provider: segmentation_id':37,
      u'name':u'private',
      u'admin_state_up':True,
      u'tenant_id':u'4c62a3fde1ba4ad3b03ddf8a986a5d88',
      u'created_at': u'2017-01-08T03:27:52 Z',
      u'mtu':1450
   }
]

[Matt Riedemann (mriedem)]

So it looks like we're getting both a vxland and flat private tenant network, which has to be due to one of those devstack changes on 1/6.

[Matt Riedemann (mriedem)]

I suspect this is the fix on the devstack side:

https://review.openstack.org/#/c/417686/

[Matt Riedemann (mriedem)]

Of the two networks in comment 3, the public one has is_default=True, I'm not sure what that means though, or if Nova should be picking that one if there are multiple available.

[Sean M. Collins (scollins)]

DevStack has always had both a private and public network that is created. I don't see any changes around that timeframe that changed anything. I'll keep looking

[Sean M. Collins (scollins)]

The only thing I can think of, is that because this test is running as the admin user (see the args being passed to the nova boot command) - there are two networks present. The public and the private network.

Notice how the public net's shared property is set to false. If this test were running as a tenant user, there'd only be one network available for use - the private net

[Matt Riedemann (mriedem)]

Yeah it could be that this is running as admin but I'm not sure if it always has or what, but apparently this wasn't an issue before 1/6 so it'd be nice to know what changed to trigger the two networks to now be exposed (and maybe this was just always broken in novaclient).

[Matt Riedemann (mriedem)]

It looks like https://review.openstack.org/#/c/413231/ in neutron was the change that broke the tests in novaclient. It changed something with the policy processing such that the admin user in the novaclient functional tests is now listing more networks than it was before. Maybe that was just always a bug in the novaclient tests and the fix in neutron is correct.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418190

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/417645
Reason: https://review.openstack.org/#/c/418190/ is the fix.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/417686
Reason: https://review.openstack.org/#/c/418190/ is the fix.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/417643
Reason: https://review.openstack.org/#/c/418190/ is the fix.

[Sean M. Collins (scollins)]

clarkb just pointed me to https://bugs.launchpad.net/manila/+bug/1652317 - so looks like OSC has a lot of problems .....

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/417645

[Matt Riedemann (mriedem)]

Apparently this was fixed by this change to blacklist openstacksdk 0.9.12 from global-requirements:

https://github.com/openstack/requirements/commit/6da1fafee9ec30d3d9653c64cd27f61bbfb21862

So this is a duplicate of bug 1652317.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418190
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=ae699768023bb0a8d60798eca7f0cc8aebfe3355
Submitter: Jenkins
Branch: master

commit ae699768023bb0a8d60798eca7f0cc8aebfe3355
Author: Matt Riedemann <email address hidden>
Date: Mon Jan 9 22:09:22 2017 -0500

    Fix functional tests to deal with multiple networks

    There was a change in the openstacksdk 0.9.11 which
    python-openstackclient uses to create networks in a devstack
    run with Neutron. Because of this change, the admin tenant
    has access to both the 'public' and 'private' network setup
    in devstack which before used to just be the 'public' network,
    which exposed a bug in the novaclient functional testing where
    the admin user is attempting to create a server but not
    specify a specific network to use, but multiple networks
    are available to the admin (it can list multiple networks).
    In this case the networks are the standard public and private
    networks that are created in devstack. Since a network isn't
    specified when creating the server, the nova API fails with a
    409 error because it can't determine which network to use.

    This patch fixes the testing in novaclient by checking to see
    if there are multiple networks available and if so, specifies
    one for the legacy BDM tests that weren't specifying a network
    ID (those tests don't really care about the networking).

    The auto-network test is skipped if there are multiple networks
    available because passing in a specific network would defeat
    the purpose of that test.

    Change-Id: I22ee148581a94b153cf7e733563cfafaa56b1ffd
    Closes-Bug: #1654806

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-novaclient 7.1.0 release.