[check][os_tempest] os_tempest standalone job is broken with etwork_type value 'vxlan' not supported.

[chandan kumar]

Standalone os_tempest non-voting job is broken currently.
http://logs.openstack.org/60/634060/12/check/tripleo-ci-centos-7-standalone-os-tempest/2981018/job-output.txt.gz#_2019-02-19_08_21_01_192646

2019-02-19 08:21:01.192646 | primary | Tuesday 19 February 2019 08:21:01 +0000 (0:00:00.316) 0:43:35.997 ******
2019-02-19 08:21:06.258203 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left).
2019-02-19 08:21:19.555821 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left).
2019-02-19 08:21:32.747744 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left).
2019-02-19 08:21:46.019431 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left).
2019-02-19 08:21:59.441350 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left).
2019-02-19 08:22:12.771957 | primary | fatal: [undercloud -> 127.0.0.2]: FAILED! => {
2019-02-19 08:22:12.772210 | primary | "attempts": 5,
2019-02-19 08:22:12.772288 | primary | "changed": false
2019-02-19 08:22:12.772328 | primary | }
2019-02-19 08:22:12.772364 | primary |
2019-02-19 08:22:12.772407 | primary | MSG:
2019-02-19 08:22:12.772443 | primary |
2019-02-19 08:22:12.772755 | primary | (400) Client Error for url: http://192.168.24.1:9696/v2.0/networks.json Invalid input for operation: network_type value 'vxlan' not supported.

Here is the ara report of that task:
http://logs.openstack.org/60/634060/12/check/tripleo-ci-centos-7-standalone-os-tempest/2981018/logs/ara_oooq/result/11f3eab6-8fdf-4cda-afb6-8de4ff78bafd/

It needs to be investigated.

[chandan kumar (chkumar246)]

From here https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L135
tempest_private_net_provider_type: "vxlan" and in validate-tempest side,

https://github.com/openstack/tripleo-quickstart-extras/blob/cb60c88eca3080be3b1fe91e6ea840d09025ecf2/roles/validate-tempest/templates/cleanup-network.sh.j2#L84

public_network_type= flat used. -> coming from https://github.com/openstack/tripleo-quickstart-extras/blob/2f434a20108fdc7b72df097a08eae62f88de5729/roles/validate-tempest/defaults/main.yml#L12

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/637838

[chandan kumar (chkumar246)]

I think there is something wrong on network ovn side
http://logs.openstack.org/60/634060/12/check/tripleo-ci-centos-7-standalone-os-tempest/2981018/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-19_08_13_14_027
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: KeyError: 'neutron:ovn-metadata-sb-cfg'
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event Traceback (most recent call last):
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event File "/usr/lib/python2.7/site-packages/ovsdbapp/event.py", line 137, in notify_loop
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event match.run(event, row, updates)
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/ovsdb_monitor.py", line 98, in run
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event self._metadata_nb_cfg(row))
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/ovsdb_monitor.py", line 86, in _metadata_nb_cfg
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event return int(row.external_ids[ovn_const.OVN_AGENT_METADATA_SB_CFG_KEY])
2019-02-19 08:13:14.027 ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 30 ERROR ovsdbapp.event KeyError: 'neutron:ovn-metadata-sb-cfg'

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637838
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=97f8b3ef86b10293e2458018ed257f15c1f7c130
Submitter: Zuul
Branch: master

commit 97f8b3ef86b10293e2458018ed257f15c1f7c130
Author: Chandan Kumar <email address hidden>
Date: Tue Feb 19 18:42:18 2019 +0530

    Set tempest_private_net_provider_type to geneve for os_tempest

    In tripleo, due to ovn migration in tripleo, vxlan does not
    work, we need to switch to geneve for private
    network type provider.

    Change-Id: Iec87c79b0ab503f38cdde744c440a551f3e3ec35
    Closes-Bug: #1816552

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.