fs020 queens is failing tempest tests - test_floatingip.FloatingIPQosTest and test_server_multi_create_auto_allocate

[Ronelle Landy]

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens is failing two tempest tests:

neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest
 - test_qos
tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest
 - test_server_multi_create_auto_allocate

Sometimes tempest times out but when it does not, it has been failing these two tests since 10/14:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens%20

Log is included below:

https://logserver.rdoproject.org/69/29969/7/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/c4e4840/logs/undercloud/home/zuul/tempest/tempest.html.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/0eb1267/logs/undercloud/home/zuul/tempest/tempest.html.gz

[Slawek Kaplonski (slaweq)]

For the test_server_multi_create_auto_allocate I see in nova logs error like:

2020-10-18 16:18:33.032 18 DEBUG nova.network.neutronv2.api [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] No networks are available for project 660eedbd5fc24b5c92c1e1ebc41b84a9; checking to see if we can automatically allocate a network. _ports_needed_per_instance /
usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1746
2020-10-18 16:18:33.053 18 DEBUG neutronclient.v2_0.client [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Error message: {"NeutronError": {"message": "Deployment error: No default router:external network.", "type": "AutoAllocationFailure", "detail": ""}} _handle_faul
t_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:259
2020-10-18 16:18:33.054 18 DEBUG nova.network.neutronv2.api [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Unable to auto-allocate networks. Deployment error: No default router:external network.
Neutron server returns request_ids: ['req-4d99a477-09fb-4c8d-a2c2-a634afe28c85'] _can_auto_allocate_network /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1683
2020-10-18 16:18:33.054 18 INFO nova.api.openstack.wsgi [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] HTTP exception thrown: Unable to automatically allocate a network for project 660eedbd5fc24b5c92c1e1ebc41b84a9
2020-10-18 16:18:33.054 18 DEBUG nova.api.openstack.wsgi [req-b7d7ab33-ce6d-42b9-9140-0cb3f769b9ca - - - - -] Returning 400 to user: Unable to automatically allocate a network for project 660eedbd5fc24b5c92c1e1ebc41b84a9 call /usr/lib/python2.7/site-packages/nova/api/o
penstack/wsgi.py:1064

So it seems for me that it is also related to the job's config as there should be default external network configured in this job.

[Slawek Kaplonski (slaweq)]

I checked logs from that failed job and it seems for me that we are missing fip_qos extension to be loaded in the L3 agent: https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/0eb1267/logs/overcloud-controller-0/var/log/containers/neutron/l3-agent.log.txt.gz (look for agent.extensions config option).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/759832

[Slawek Kaplonski (slaweq)]

Regarding issue with tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_multi_create_auto_allocate test it seems that fix for that may require many changes becuase:
- our jobs are using os_tempest role which uses openstack-ansible-os_tempest
- openstack-ansible-os_tempest is using openstack.cloud.os_network ansible module to create public network
- openstack.cloud.os_network module don't supports setting "is_default" flag for network, also this flag is added only by auto-allocate service plugin so if that plugin is not enabled, there is no "is_default" flag in network resource in Neutron.

Based on all of that I think that skipping this test (as we are doing now) is correct approach.

[Sandeep Yadav (sandeepyadav93)]

Hello Slawek,

Thank you so much for your help with this bug.

1) neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest

We discussed last week that this test is currently only triggering on master and not on other branches.

I investigated this with our tempest squad internally, This is because as per[1] this test requires qos-fip extension, if qos-fip extension is not present then test will be skipped.

[1] https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/scenario/test_floatingip.py#L339

and from logs it appears that qos-fip is only enabled in master.

Master logs:-

On Master we have qos-fip in network-feature-enabled.api_extensions.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/07ac197/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz
~~~
2020-10-30 11:16:49.700 235685 DEBUG tempest [-] network-feature-enabled.api_extensions = ['address-group', 'address-scope', 'agent', 'agent-resources-synced', 'allowed-address-pairs', 'auto-allocated-topology', 'availability_zone', 'availability_zone_filter', 'default-subnetpools', 'dhcp_agent_scheduler', 'dns-integration', 'empty-string-filtering', 'external-net', 'extra_dhcp_opt', 'extraroute', 'fip-port-details', 'flavors', 'ip_allocation', 'ip-substring-filtering', 'l2_adjacency', 'router', 'ext-gw-mode', 'multi-provider', 'net-mtu', 'net-mtu-writable', 'network_availability_zone', 'network-ip-availability', 'pagination', 'port-mac-address-regenerate', 'port-resource-request', 'binding', 'binding-extended', 'port-security', 'project-id', 'provider', 'qos', 'qos-bw-limit-direction', 'qos-bw-minimum-ingress', 'qos-default', 'qos-fip', 'qos-port-network-policy', 'qos-rule-type-details', 'qos-rules-alias', 'quotas', 'quota_details', 'rbac-policies', 'rbac-address-scope', 'rbac-security-groups', 'rbac-subnetpool', 'revision-if-match', 'standard-attr-revisions', 'router_availability_zone', 'port-security-groups-filtering', 'security-group', 'segment', 'segments-peer-subnet-host-routes', 'service-type', 'sorting', 'standard-attr-segment', 'standard-attr-description', 'stateful-security-group', 'subnet_onboard', 'subnet-segmentid-writable', 'subnet-service-types', 'subnet_allocation', 'subnetpool-prefix-ops', 'standard-attr-tag', 'standard-attr-timestamp', 'trunk', 'trunk-details'] log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2593
~~~

Ussuri:-

This test is skipped because qos-fip is not enabled

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/9b5908c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

~~~
{1} setUpClass (neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest) ... SKIPPED: Skipped because network extension: qos-fip is not enabled
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/9b5908c/logs/undercloud/home/zuul/te...

[Sandeep Yadav (sandeepyadav93)]

Hi Slaweq,

In the latest runs FloatingipQOS tempest test failed on master.

1) neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos is failing on master

~~~
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/9942a40/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/003689f/logs/undercloud/var/log/tempest/stestr_results.html.gz
~~~

Could you please help us confirming this is because of the same issue - missing QoS plugin in neutron?

For which you have patches up - https://review.opendev.org/#/c/759829/ & https://review.opendev.org/#/c/759832/

[Sandeep Yadav (sandeepyadav93)]

Hi,

I checked with slaweq about latest failure in master branch[1]

default neutron backend is ovn and it don't supports floating IP qos yet (it's almost there but still some patches aren't merged).

CI patches - https://review.opendev.org/#/c/759829/ & https://review.opendev.org/#/c/759832/ will not help with the issue.

We agreed to skip this test(FloatingipQOS) in that job for now.

[1] neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos is failing on master

~~~
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/9942a40/logs/undercloud/var/log/tempest/stestr_results.html.gz
~~~

[Rafael Folco (rafaelfolco)]

Is anyone actively looking at this issue? I observed this test is in skip list for rocky, and now I just submitted for queens:

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/763831

Skip FloatingIPQosTest on queens [WIP] [NEW]

[Marios Andreou (marios-b)]

from slaweq:

Backports proposed:
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764090
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764091
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764092
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764093
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764094
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764095

[Ronelle Landy (rlandy)]

Closing this bug out - asked Soniya to remove the skiplist entry

[Soniya Murlidhar Vyas (svyas)]

I have added the patch to remove the skiplist entry.

- https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/814903