OVN: tempest fails on floating-ip

Bug #1740698 reported by Emilien Macchi on 2017-12-31
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
High
Numan Siddique
Changed in tripleo:
assignee: nobody → Numan Siddique (numansiddique)

Reviewed: https://review.openstack.org/530593
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503
Submitter: Zuul
Branch: master

commit 2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503
Author: Emilien Macchi <email address hidden>
Date: Sun Dec 31 09:42:14 2017 -0800

    fs030: disable tempest

    Promotion jobs fail to run Tempest on fs030, OVN fails to create a
    floating-IP.

    Change-Id: I3b48cae7b5e199a93d31372e893fadcbfc4ab18e
    Related-Bug: #1740698

Eran Kuris (ekuris) wrote :

can you specify which test fails?

Numan Siddique (numansiddique) wrote :

Looking into why its failing.

Numan Siddique (numansiddique) wrote :

The reason for the failure is because of this patch - https://review.openstack.org/#/c/527680/9 which Emilien pointed out in the bug report.

There is a corresponding OVS patch here [1] which is needed.
Three solutions
- Fix code in networking-ovn
- Apply the patch [1] in rdo OVS 2.8.1
- Request for another OVS 2.8 release upstream and then RDO picking from there.

I prefer first or the third. I will check with the team and get back on this.

[1] - https://github.com/openvswitch/ovs/commit/7d0959a6225d32580d234ce2a456bbab7fa3f1e0#diff-90d020781f83e2f707edb26de0b5c552

Assaf Muller (amuller) wrote :

You could argue that networking-ovn shouldn't have such strict OVS version requirements. We might be imposing unfriendly requirements from distros and operators, not to mention that the required version is not discoverable / documented.

Daniel Alvarez (dalvarezs) wrote :

Fix released: https://review.openstack.org/#/c/530721/
I sent the patch before I saw this bug so missed the "Closes-Bug" tag.

Changed in tripleo:
status: Triaged → Fix Released
Alan Pevec (apevec) wrote :

For openvswitch specifically, min. required version dep could be documented in
https://github.com/openstack/networking-ovn/blob/master/requirements.txt#L8

For general deps without Python lib part, we could come up with something using bindep.txt

Alan Pevec (apevec) on 2018-01-05
Changed in tripleo:
status: Fix Released → In Progress
Changed in tripleo:
importance: Critical → High
tags: added: tech-debt
removed: alert ci promotion-blocker
Alan Pevec (apevec) wrote :

Followup fix https://review.openstack.org/531425
Alternatively, we have prepared https://github.com/rdo-common/openvswitch/pull/5 to update OVS 2.8.1 based RPM in RDO deps repo, before upstream 2.8.2 is released.

Alan Pevec (apevec) on 2018-01-08
Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/networking-ovn 4.0.0.0b3 development milestone.

Alan Pevec (apevec) wrote :
Changed in tripleo:
status: Fix Released → Confirmed
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
status: Confirmed → Triaged

This issue was fixed in the openstack/networking-ovn 4.0.0.0rc1 release candidate.

Changed in tripleo:
milestone: queens-rc1 → rocky-1

Reviewed: https://review.openstack.org/530890
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=4c9120434458d17298fa38a32dfcc76c72697e56
Submitter: Zuul
Branch: master

commit 4c9120434458d17298fa38a32dfcc76c72697e56
Author: Daniel Alvarez <email address hidden>
Date: Wed Jan 3 11:14:42 2018 +0100

    Revert "fs030: disable tempest"

    This reverts commit 2f14e6d3ad15c71f6919bfdebfdd1208ab8b2503.
    This patch [0] in networking-ovn fixes the issue that prevented
    RDO promotion. As it's already merged we can reenable tempest
    on fs030.

    Related-Bug: #1740698

    [0] https://review.openstack.org/#/c/530721/

    Change-Id: Icb235d311693f6b4e0fd466d63cd7437e988145c

yatin (yatinkarel) wrote :

After reverting it we are having a new issue both in master and queens:-
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/home/jenkins/tempest/tempest.html.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/a55e13e/undercloud/home/jenkins/tempest/tempest.html.gz

This time it's SSH issue, keys are not injected into instance, so SSH via key fails:-
failed to get http://169.254.169.254/2009-04-04/meta-data/public-keys
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/user-data
warning: no ec2 metadata for user-data

Since it's a new issue new bug can be opened for it.

yatin (yatinkarel) wrote :

<<< This time it's SSH issue, keys are not injected into instance, so SSH via key fails:-
>>>
Looking further i found that SSH keys are injected but not that which are provided by ovn metadata(or passed during vm creation) but by metadata agent running for rdocloud.

The same hostname of undercloud([1]) is assigned to vm spawned(nova console-log see [2]) by tempest on overcloud:-
upstream-centos-7-2-node-rdo-cloud-tripleo-94021.rdocloud, Also the keys injected is multinode-admins keys

[1] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/etc/hostname.txt.gz
[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/36ddc06/undercloud/home/jenkins/tempest/tempest.html.gz

I don't know what's causing it but may be some networks are conflicting or ovn metadata.

yatin (yatinkarel) wrote :

<< Since it's a new issue new bug can be opened for it.
>> https://bugs.launchpad.net/tripleo/+bug/1757134

This issue was fixed in the openstack/networking-ovn 5.0.0.0b1 development milestone.

Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers