fs020 queens is failing tempest tests - test_floatingip.FloatingIPQosTest and test_server_multi_create_auto_allocate

Bug #1900357 reported by Ronelle Landy on 2020-10-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Soniya Murlidhar Vyas

Bug Description

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens is failing two tempest tests:

neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest
 - test_qos
tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest
 - test_server_multi_create_auto_allocate

Sometimes tempest times out but when it does not, it has been failing these two tests since 10/14:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens%20

Log is included below:

https://logserver.rdoproject.org/69/29969/7/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/c4e4840/logs/undercloud/home/zuul/tempest/tempest.html.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/0eb1267/logs/undercloud/home/zuul/tempest/tempest.html.gz

Ronelle Landy (rlandy) on 2020-10-18
Changed in tripleo:
milestone: none → victoria-rc1
importance: Undecided → Critical
status: New → Triaged
tags: added: promotion-blocker
Slawek Kaplonski (slaweq) wrote :

For the test_server_multi_create_auto_allocate I see in nova logs error like:

2020-10-18 16:18:33.032 18 DEBUG nova.network.neutronv2.api [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] No networks are available for project 660eedbd5fc24b5c92c1e1ebc41b84a9; checking to see if we can automatically allocate a network. _ports_needed_per_instance /
usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1746
2020-10-18 16:18:33.053 18 DEBUG neutronclient.v2_0.client [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Error message: {"NeutronError": {"message": "Deployment error: No default router:external network.", "type": "AutoAllocationFailure", "detail": ""}} _handle_faul
t_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:259
2020-10-18 16:18:33.054 18 DEBUG nova.network.neutronv2.api [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Unable to auto-allocate networks. Deployment error: No default router:external network.
Neutron server returns request_ids: ['req-4d99a477-09fb-4c8d-a2c2-a634afe28c85'] _can_auto_allocate_network /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1683
2020-10-18 16:18:33.054 18 INFO nova.api.openstack.wsgi [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] HTTP exception thrown: Unable to automatically allocate a network for project 660eedbd5fc24b5c92c1e1ebc41b84a9
2020-10-18 16:18:33.054 18 DEBUG nova.api.openstack.wsgi [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Returning 400 to user: Unable to automatically allocate a network for project 660eedbd5fc24b5c92c1e1ebc41b84a9 call /usr/lib/python2.7/site-packages/nova/api/o
penstack/wsgi.py:1064

So it seems for me that it is also related to the job's config as there should be default external network configured in this job.

Ronelle Landy (rlandy) on 2020-10-21
Changed in tripleo:
assignee: nobody → Soniya Murlidhar Vyas (svyas)
Slawek Kaplonski (slaweq) wrote :

Regarding issue with tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_multi_create_auto_allocate test it seems that fix for that may require many changes becuase:
- our jobs are using os_tempest role which uses openstack-ansible-os_tempest
- openstack-ansible-os_tempest is using openstack.cloud.os_network ansible module to create public network
- openstack.cloud.os_network module don't supports setting "is_default" flag for network, also this flag is added only by auto-allocate service plugin so if that plugin is not enabled, there is no "is_default" flag in network resource in Neutron.

Based on all of that I think that skipping this test (as we are doing now) is correct approach.

Changed in tripleo:
milestone: victoria-rc1 → wallaby-1
Sandeep Yadav (sandeepyadav93) wrote :
Download full text (4.9 KiB)

Hello Slawek,

Thank you so much for your help with this bug.

1) neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest

We discussed last week that this test is currently only triggering on master and not on other branches.

I investigated this with our tempest squad internally, This is because as per[1] this test requires qos-fip extension, if qos-fip extension is not present then test will be skipped.

[1] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/scenario/test_floatingip.py#L339

and from logs it appears that qos-fip is only enabled in master.

Master logs:-

On Master we have qos-fip in network-feature-enabled.api_extensions.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/07ac197/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz
~~~
2020-10-30 11:16:49.700 235685 DEBUG tempest [-] network-feature-enabled.api_extensions = ['address-group', 'address-scope', 'agent', 'agent-resources-synced', 'allowed-address-pairs', 'auto-allocated-topology', 'availability_zone', 'availability_zone_filter', 'default-subnetpools', 'dhcp_agent_scheduler', 'dns-integration', 'empty-string-filtering', 'external-net', 'extra_dhcp_opt', 'extraroute', 'fip-port-details', 'flavors', 'ip_allocation', 'ip-substring-filtering', 'l2_adjacency', 'router', 'ext-gw-mode', 'multi-provider', 'net-mtu', 'net-mtu-writable', 'network_availability_zone', 'network-ip-availability', 'pagination', 'port-mac-address-regenerate', 'port-resource-request', 'binding', 'binding-extended', 'port-security', 'project-id', 'provider', 'qos', 'qos-bw-limit-direction', 'qos-bw-minimum-ingress', 'qos-default', 'qos-fip', 'qos-port-network-policy', 'qos-rule-type-details', 'qos-rules-alias', 'quotas', 'quota_details', 'rbac-policies', 'rbac-address-scope', 'rbac-security-groups', 'rbac-subnetpool', 'revision-if-match', 'standard-attr-revisions', 'router_availability_zone', 'port-security-groups-filtering', 'security-group', 'segment', 'segments-peer-subnet-host-routes', 'service-type', 'sorting', 'standard-attr-segment', 'standard-attr-description', 'stateful-security-group', 'subnet_onboard', 'subnet-segmentid-writable', 'subnet-service-types', 'subnet_allocation', 'subnetpool-prefix-ops', 'standard-attr-tag', 'standard-attr-timestamp', 'trunk', 'trunk-details'] log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2593
~~~

Ussuri:-

This test is skipped because qos-fip is not enabled

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/9b5908c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

~~~
{1} setUpClass (neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest) ... SKIPPED: Skipped because network extension: qos-fip is not enabled
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/9b5908c/logs/undercloud/home/zuul/te...

Read more...

Sandeep Yadav (sandeepyadav93) wrote :

Hi,

I checked with slaweq about latest failure in master branch[1]

default neutron backend is ovn and it don't supports floating IP qos yet (it's almost there but still some patches aren't merged).

CI patches - https://review.opendev.org/#/c/759829/ & https://review.opendev.org/#/c/759832/ will not help with the issue.

We agreed to skip this test(FloatingipQOS) in that job for now.

[1] neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos is failing on master

~~~
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/9942a40/logs/undercloud/var/log/tempest/stestr_results.html.gz
~~~

Rafael Folco (rafaelfolco) wrote :

Is anyone actively looking at this issue? I observed this test is in skip list for rocky, and now I just submitted for queens:

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/763831

Skip FloatingIPQosTest on queens [WIP] [NEW]

Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers