Tempest test test_create_update_port_with_dns_domain failure KeyError: 'dns_domain'

[Amol Kahat]

Description

Tempest test: test_create_update_port_with_dns_domain failed.

Traceback:
2022-09-21 15:03:32,837 183447 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"port": {"network_id": "78677eaf-101d-4226-9d03-4e814a61ef67", "dns_name": "d1", "dns_domain": "test.org."}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '885', 'x-openstack-request-id': 'req-9abfaa44-e7f1-4c00-8003-a24a988939ee', 'date': 'Wed, 21 Sep 2022 15:03:32 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://192.168.24.3:9696/v2.0/ports'}
        Body: b'{"port":{"id":"1cbb7c1e-e764-43c0-b172-e2a8b89005ee","name":"","network_id":"78677eaf-101d-4226-9d03-4e814a61ef67","tenant_id":"2a13c5ec235e456fa6263d48c97c7eba","mac_address":"fa:16:3e:0e:3a:0c","admin_state_up":true,"status":"DOWN","device_id":"","device_owner":"","fixed_ips":[{"subnet_id":"11b92ce6-1eaa-495d-9899-e06f817e864d","ip_address":"10.100.0.44"}],"project_id":"2a13c5ec235e456fa6263d48c97c7eba","qos_policy_id":null,"port_security_enabled":true,"security_groups":["22659cde-cc48-437d-b829-9b6832992822"],"binding:vnic_type":"normal","allowed_address_pairs":[],"extra_dhcp_opts":[],"description":"","qos_network_policy_id":null,"dns_name":"d1","dns_assignment":[{"ip_address":"10.100.0.44","hostname":"d1","fqdn":"d1.openstackgate.local."}],"ip_allocation":"immediate","tags":[],"created_at":"2022-09-21T15:03:32Z","updated_at":"2022-09-21T15:03:32Z","revision_number":1}}'
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/api/test_ports.py", line 119, in test_create_update_port_with_dns_domain
    self.assertEqual('test.org.', body['dns_domain'])
KeyError: 'dns_domain'

Logs:
- https://logserver.rdoproject.org/65/41465/12/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/eaaf445/logs/undercloud/var/log/tempest/stestr_results.html.gz
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/d2ff917/logs/undercloud/var/log/tempest/stestr_results.html.gz
- https://logserver.rdoproject.org/65/41465/13/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/4bb1839/logs/undercloud/var/log/tempest/stestr_results.html.gz

[yatin (yatinkarel)]

The failure is triggered by https://review.opendev.org/c/openstack/neutron/+/857791 as before that patch the test used to be skipped:- neutron_tempest_plugin.api.test_ports.PortsTestJSON.test_create_update_port_with_dns_domain ... SKIPPED: Skipped because network extension: dns-domain-ports is not enabled

"dns_domain" is not in Body: b'{"port":{"id":"1cbb7c1e-e764-43c0-b172-e2a8b89005ee","name":"","network_id":"78677eaf-101d-4226-9d03-4e814a61ef67","tenant_id":"2a13c5ec235e456fa6263d48c97c7eba","mac_address":"fa:16:3e:0e:3a:0c","admin_state_up":true,"status":"DOWN","device_id":"","device_owner":"","fixed_ips":[{"subnet_id":"11b92ce6-1eaa-495d-9899-e06f817e864d","ip_address":"10.100.0.44"}],"project_id":"2a13c5ec235e456fa6263d48c97c7eba","qos_policy_id":null,"port_security_enabled":true,"security_groups":["22659cde-cc48-437d-b829-9b6832992822"],"binding:vnic_type":"normal","allowed_address_pairs":[],"extra_dhcp_opts":[],"description":"","qos_network_policy_id":null,"dns_name":"d1","dns_assignment":[{"ip_address":"10.100.0.44","hostname":"d1","fqdn":"d1.openstackgate.local."}],"ip_allocation":"immediate","tags":[],"created_at":"2022-09-21T15:03:32Z","updated_at":"2022-09-21T15:03:32Z","revision_number":1}}'

The test not got tested in neutron CI with OVN so likely we missing patch to support that extenstion with OVN or tests might need update. Till it's figured out we can skip the failing test in wallaby jobs(both component/integration ones) as it used to be.

[yatin (yatinkarel)]

Tested OVN api job in neutron and it went green so it's all good from neutron side. https://review.opendev.org/c/openstack/neutron/+/858953

Checked tripleo master and that is also impacted[1] but didn't hit because it was missing coverage, proposed https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859027 to fix the coverage part.

But we need to skip test_create_update_port_with_dns_domain in master component/integration jobs too.

The skips can be reverted once we fix jobs to get test passing, mainly need to fix extension_drivers for OVN.

[yatin (yatinkarel)]

<< Till it's figured out we can skip the failing test in wallaby jobs(both component/integration ones) as it used to be.
Test is skipped for both master and wallaby api jobs with https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859074

<< The skips can be reverted once we fix jobs to get test passing, mainly need to fix extension_drivers for OVN.
Currently we have NeutronPluginExtensions: qos,port_security,dns and for that test to be passing need to include dns_domain_ports or any other driver inheriting from it like subnet_dns_publish_fixed_ip or dns_domain_keywords(it includes all other dns related extension drivers).
So question is we enable it only for CI or the defaults, will check with @slawek if we have some criteria for changing defaults in this case.

[Slawek Kaplonski (slaweq)]

Hi yatin,
I don't think there are any specific criteria for changing those defaults. IMO we can change it as OVN driver really supports that extension even in Wallaby AFAIK. So IMO it's simply misconfiguration what we have currently and changing it shouldn't cause any issues really.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859288
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/7efe0063b97bfdd9dee960d920642f0385095558
Submitter: "Zuul (22348)"
Branch: master

commit 7efe0063b97bfdd9dee960d920642f0385095558
Author: yatinkarel <email address hidden>
Date: Mon Sep 26 17:17:25 2022 +0530

    Enable dns_domain_ports extension by default

    'dns_domain_ports' extension allows dns_domain to be set
    for ports and ML2 OVN supports it since wallaby[1]
    and ML2 OVS long before that.
    This patch replaces 'dns' with 'dns_domain_ports' for
    NeutronPluginExtensions in ml2 plugin heat template and
    all environment files.
    'dns' extension not equired to be enabled explicitly
    with 'dns_domain_ports' as it inherits all the capabilities
    of 'dns' extension.

    [1] https://review.opendev.org/q/Ic269902ef7a16893c4ea624b04347d04db4f52d9

    Related-Bug: #1990480
    Change-Id: I25f75d7bd8864ea132246cea79e073232d664d76

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859834

[Soniya Murlidhar Vyas (svyas)]

This issue is happening in periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby

[1] https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby/3aeaab4/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2]https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby/3632aea/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3]https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby/df87817/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Soniya Murlidhar Vyas (svyas)]

Here is the skiplist patch for c9-full-tempest-api-compute-wallaby - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/860079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859834
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/aaf4435d2c013ab631103ac2a45e73d50dac34cc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit aaf4435d2c013ab631103ac2a45e73d50dac34cc
Author: yatinkarel <email address hidden>
Date: Mon Sep 26 17:17:25 2022 +0530

    Enable dns_domain_ports extension by default

    'dns_domain_ports' extension allows dns_domain to be set
    for ports and ML2 OVN supports it since wallaby[1]
    and ML2 OVS long before that.
    This patch replaces 'dns' with 'dns_domain_ports' for
    NeutronPluginExtensions in ml2 plugin heat template and
    all environment files.
    'dns' extension not equired to be enabled explicitly
    with 'dns_domain_ports' as it inherits all the capabilities
    of 'dns' extension.

    [1] https://review.opendev.org/q/Ic269902ef7a16893c4ea624b04347d04db4f52d9

    Related-Bug: #1990480
    Change-Id: I25f75d7bd8864ea132246cea79e073232d664d76
    (cherry picked from commit 7efe0063b97bfdd9dee960d920642f0385095558)
    Conflicts:
            environments/services/undercloud-neutron-ovn-dhcp.yaml