periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images

[Marios Andreou]

The periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master is failing nova related tempest at [1] [2] with a trace like:

 2019-03-26 04:18:06 | tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_multi_create_auto_allocate[id-2e6cf129-9e28-4e8a-aaaa-045ea826b2a6]
 2019-03-26 04:18:06 | Body: {"badRequest": {"message": "Unable to automatically allocate a network for project 34df197c8cc848c68c22b9f7845dbe68", "code": 400}}

 2019-03-26 04:18:06 | tearDownClass (tempest.api.compute.images.test_images.ImagesTestJSON)
 2019-03-26 04:18:06 | testtools.runtest.MultipleExceptions: (<class 'tempest.lib.exceptions.DeleteErrorException'>, Resource 33730221-6e22-4d16-acf7-77a6e0f64b35 failed to delete and is in ERROR status, <traceback object at 0x7effe234cc20>)
 2019-03-26 04:18:06 | testtools.runtest.MultipleExceptions: (<class 'tempest.lib.exceptions.DeleteErrorException'>, Resource 33730221-6e22-4d16-acf7-77a6e0f64b35 failed to delete and is in ERROR status, <traceback object at 0x7effe234cc20>)

[1] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/274a91e/logs/undercloud/home/zuul/tempest.log.txt.gz
[2] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/0f410a5/logs/undercloud/home/zuul/tempest.log.txt.gz

[Marios Andreou (marios-b)]

related to https://review.openstack.org/#/c/641368 ?

    09:24 < ykarel> marios_|ruck, is fs020 failure known? 09:24 < ykarel> 1 tempest test is failing since tempestconf patch:- https://review.openstack.org/#/c/641368/
    09:28 < kopecmartin> ykarel|lunch, marios_|ruck if the test was skipped before, was it because microversions weren't set for compute? because that's what was added to python-tempestconf .. if that's the case, add those tests to blacklist

[Marios Andreou (marios-b)]

10:41 < kopecmartin> ykarel, so it was skipped because microversions weren't configured, now they are configured by python-tempestconf so if you want to keep the test skipped, add it to the blacklist
10:41 < kopecmartin> or figure out, why the tests couldn't allocate network for project
10:41 < ykarel> kopecmartin, ok that i think someone from nova can check
10:42 < ykarel> after tempestconf patch, min_version=2.1

[Arx Cruz (arxcruz)]

Seeing these errors on neutron:

2019-03-25 08:39:06.441 30 ERROR neutron.plugins.ml2.managers [req-6ea12291-2833-4b22-b07c-32c56e597be6 76d3f135dbb946e9a369bc67801a9db1 9147a486efe84d2f8e7a89ec0eaaf9da - default default] Failed to bind port 95429802-e1e3-46bb-ac73-7230d0314132 on host undercloud.localdomain for vnic_type baremetal using segments [{'network_id': '4747ff20-26df-496a-b0fc-31a1ee0e19eb', 'segmentation_id': None, 'physical_network': u'ctlplane', 'id': 'bf9777d8-62ed-4d7d-a48b-226c1da5b28d', 'network_type': u'flat'}]

[Arx Cruz (arxcruz)]

Also I found this:

2019-03-25 08:39:05.852 20 INFO nova.api.openstack.requestlog [req-1978a004-7caa-4f2d-9165-5ab026e682fe 69f4059d6181471aaec5704e323fcf5e 9147a486efe84d2f8e7a89ec0eaaf9da - default default] 192.168.24.1 "OPTIONS /" status: 200 len: 401 microversion: - time: 0.001080
2019-03-25 08:39:06.603 21 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer

And it seems microversion are not set properly:

2019-03-25 08:39:03.849 23 INFO nova.api.openstack.requestlog [req-2094c0ec-9694-4910-9d84-1c954d1af7da 69f4059d6181471aaec5704e323fcf5e 9147a486efe84d2f8e7a89ec0eaaf9da - default default] 192.168.24.1 "OPTIONS /" status: 200 len: 401 microversion: - time: 0.001286

[Martin Kopec (mkopec)]

Just FYI, I compared tempest.confs before and now and there is a difference in network-feature-enabled.api_extensions. The new failing one contains also qos-rules-alias extension and according to this bug [1] the feature adding the extension was merged 5 days ago, maybe this is part of the reason the test is failing?

[1] https://bugs.launchpad.net/neutron/+bug/1777627/comments/23

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/648099

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648099
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=a295c006d8b3a196b6aaca3e313eb00ce51a8d94
Submitter: Zuul
Branch: master

commit a295c006d8b3a196b6aaca3e313eb00ce51a8d94
Author: Arx Cruz <email address hidden>
Date: Wed Mar 27 11:47:45 2019 +0000

    Adding test_server_multi_create_auto_allocate to skip list

    Test is failing probably due microversion settings included in the
    recent change in python-tempestconf. Still being investigated, but for
    now added in skip list.

    Change-Id: Ica29e0468be06de50b5478352895b6b855f5386f
    Related-Bug: 1821690

[Marios Andreou (marios-b)]

k so the skip merged https://review.openstack.org/#/c/648099/ but i think there is still work here? Do we need to reach out to Nova re the microversions

We also haven't yet had a green run last one timed out in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master (but can't see errors in the tempest log - but not sure if it completed OK http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/baec50c/logs/undercloud/home/zuul/tempest.log.txt.gz or if it was caught by the timeout).

[Marios Andreou (marios-b)]

trying to understand if something further is needed here. I think it is not, because as per comments #2 and #4 this test was being skipped before https://review.openstack.org/#/c/641368 as microversion support was missing from tempestconf.

Went looking for the failing test to try understand more, found [1] - docstring there suggests this is microversion related but that's about as much as i can say :D

Going to propose closing out for now, but I will ask other folks to check this preferably compute team (and maybe network after that if necessary).

[1] https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_auto_allocate_network.py#L32

[Arx Cruz (arxcruz)]

I am still woring on this. I haven't the chance to have a environment reproduced for this yet, and with all tempest release I'm with small slot for this. Right now it's not an issue for promotion because the test is skipped.

[Arx Cruz (arxcruz)]

So here's my findings:

With https://github.com/openstack/python-tempestconf/commit/ca51d4d94c7b87444c65eb33734ee76e74834188 we added support to microversions and so, tempest started to have compute.min_microversion and compute.max_microversion option in tempest.conf.
With these two options, the test tempest test_auto_allocate_network started to run. This test tries to create 3 servers, and auto allocate networks for that, failing with this:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_auto_allocate_network.py", line 177, in test_server_multi_create_auto_allocate
    min_count=3)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 198, in create_test_server
    **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 97, in create_server
    resp, body = self.post('servers', post_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 283, in post
    return self.request('POST', url, extra_headers, headers, body, chunked) ...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/650310

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650310
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=0cfee0b75f19b8831d7d0343a976dc7d960f79c1
Submitter: Zuul
Branch: master

commit 0cfee0b75f19b8831d7d0343a976dc7d960f79c1
Author: Arx Cruz <email address hidden>
Date: Fri Apr 5 14:42:42 2019 +0000

    Adding test_server_multi_create_auto_allocate to stein skip list

    This test is failing, and it's not supposed to be running.
    It's related to micro version in nova/neutron api.

    Change-Id: Idae977d8c39921942f00e96cd94e3b9abdaabb70
    Related-Bug: 1821690

[Bernard Cafarelli (bcafarel)]

On the network side, the "get me a network" feature is used to create a private network ready to use (with a subnet, a subnet pool so you don't need to set a prefix/range, a router and connection to external public network).

So here the ext-subnet subnet does not matter for the test (the needed subnetpool is only used for the private subnet), and you can configure the subnetpool with pretty much any parameters - big enough to host all the addresses needed for the test of course.

Apparently you set everything correctly, the feature needs:
* a default external network to connect to
* a subnetpool that will be used for the created subnet in the tests

It will then create a net, a subnet (using the recommended pool), a router linking the net to the default external network.

[Martin Kopec (mkopec)]

So one more time, the test was skipped before and no one complained. After a certain feature was added to python-tempestconf it was unskipped and started failing.
Solution: Skip the test and close the issue.

[Ronelle Landy (rlandy)]

Per Martin's comment - closing this

[chandan kumar (chkumar246)]

Affecting http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/04c1fe5/logs/tempest.html.gz rocky aslo.

[chandan kumar (chkumar246)]

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/c7d1f45/logs/tempest.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/657964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657964
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=cf9ac314b18b283e60326038ed5124c0e2845924
Submitter: Zuul
Branch: master

commit cf9ac314b18b283e60326038ed5124c0e2845924
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu May 9 15:02:10 2019 +0530

    Adding test_server_multi_create_auto_allocate to rocky/queens skip list

    This test is failing, and it's not supposed to be running.
    It's related to micro version in nova/neutron api.

    Related-Bug: 1821690

    https://tree.taiga.io/project/tripleo-ci-board/task/1050

    Change-Id: I134694323b8462e939e8370334e5777c2c743d18
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>