OVN: tempest fails on floating-ip

[Emilien Macchi]

When promoting to latest networking-ovn, tempest fails to create a VM and assign a floating-ip:
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/62266a4/subnode-2/var/log/extra/docker/containers/neutron_api/stdout.log.txt.gz#_2017-12-31_16_53_49_827

It might be related to https://review.openstack.org/#/c/527680/.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/530593

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530593
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503
Submitter: Zuul
Branch: master

commit 2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503
Author: Emilien Macchi <email address hidden>
Date: Sun Dec 31 09:42:14 2017 -0800

    fs030: disable tempest

    Promotion jobs fail to run Tempest on fs030, OVN fails to create a
    floating-IP.

    Change-Id: I3b48cae7b5e199a93d31372e893fadcbfc4ab18e
    Related-Bug: #1740698

[Eran Kuris (ekuris)]

can you specify which test fails?

[Alex Schultz (alex-schultz)]

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/62266a4/tempest.html.gz

[Numan Siddique (numansiddique)]

Looking into why its failing.

[Numan Siddique (numansiddique)]

The reason for the failure is because of this patch - https://review.openstack.org/#/c/527680/9 which Emilien pointed out in the bug report.

There is a corresponding OVS patch here [1] which is needed.
Three solutions
- Fix code in networking-ovn
- Apply the patch [1] in rdo OVS 2.8.1
- Request for another OVS 2.8 release upstream and then RDO picking from there.

I prefer first or the third. I will check with the team and get back on this.

[1] - https://github.com/openvswitch/ovs/commit/7d0959a6225d32580d234ce2a456bbab7fa3f1e0#diff-90d020781f83e2f707edb26de0b5c552

[Assaf Muller (amuller)]

You could argue that networking-ovn shouldn't have such strict OVS version requirements. We might be imposing unfriendly requirements from distros and operators, not to mention that the required version is not discoverable / documented.

[Daniel Alvarez (dalvarezs)]

Fix released: https://review.openstack.org/#/c/530721/
I sent the patch before I saw this bug so missed the "Closes-Bug" tag.

[Alan Pevec (apevec)]

For openvswitch specifically, min. required version dep could be documented in
https://github.com/openstack/networking-ovn/blob/master/requirements.txt#L8

For general deps without Python lib part, we could come up with something using bindep.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/530890

[Alan Pevec (apevec)]

Followup fix https://review.openstack.org/531425
Alternatively, we have prepared https://github.com/rdo-common/openvswitch/pull/5 to update OVS 2.8.1 based RPM in RDO deps repo, before upstream 2.8.2 is released.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 4.0.0.0b3 development milestone.

[Alan Pevec (apevec)]

Reintroduced in https://review.openstack.org/#/c/528746/19/networking_ovn/ovsdb/commands.py@1034

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 4.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530890
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=4c9120434458d17298fa38a32dfcc76c72697e56
Submitter: Zuul
Branch: master

commit 4c9120434458d17298fa38a32dfcc76c72697e56
Author: Daniel Alvarez <email address hidden>
Date: Wed Jan 3 11:14:42 2018 +0100

    Revert "fs030: disable tempest"

    This reverts commit 2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503.
    This patch [0] in networking-ovn fixes the issue that prevented
    RDO promotion. As it's already merged we can reenable tempest
    on fs030.

    Related-Bug: #1740698

    [0] https://review.openstack.org/#/c/530721/

    Change-Id: Icb235d311693f6b4e0fd466d63cd7437e988145c

[yatin (yatinkarel)]

After reverting it we are having a new issue both in master and queens:-
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/home/jenkins/tempest/tempest.html.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/a55e13e/undercloud/home/jenkins/tempest/tempest.html.gz

This time it's SSH issue, keys are not injected into instance, so SSH via key fails:-
failed to get http://169.254.169.254/2009-04-04/meta-data/public-keys
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/user-data
warning: no ec2 metadata for user-data

Since it's a new issue new bug can be opened for it.

[yatin (yatinkarel)]

<<< This time it's SSH issue, keys are not injected into instance, so SSH via key fails:-
>>>
Looking further i found that SSH keys are injected but not that which are provided by ovn metadata(or passed during vm creation) but by metadata agent running for rdocloud.

The same hostname of undercloud([1]) is assigned to vm spawned(nova console-log see [2]) by tempest on overcloud:-
upstream-centos-7-2-node-rdo-cloud-tripleo-94021.rdocloud, Also the keys injected is multinode-admins keys

[1] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/etc/hostname.txt.gz
[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/home/jenkins/tempest/tempest.html.gz

I don't know what's causing it but may be some networks are conflicting or ovn metadata.

[yatin (yatinkarel)]

<< Since it's a new issue new bug can be opened for it.
>> https://bugs.launchpad.net/tripleo/+bug/1757134

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 5.0.0.0b1 development milestone.