multi-node-bridge role in zuul-jobs uses a released OVS package

Bug #1780780 reported by Ronelle Landy on 2018-07-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

Tempest test test_network_basic_ops running on master with scenario007 and scenario008 jobs fails with networking/connection issues:

http://logs.openstack.org/53/579653/3/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/5e299dc/logs/undercloud/home/zuul/tempest/tempest.html.gz

The problem stems from that fact that we now set up br-ex using pre.yaml from zuul-jobs which relies on a released version of openvswitch:

https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/vars/RedHat.yaml#L4

https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L21

The role defaulted to the ocata version - hence the incompatibility with master deployments.
I was able to get the scenario007 to pass in my local reproducer environment when using the released queens version of openvswitch.

The methodology here is problematic. If we plan to test TripleO integration with OVS, and we only bring in the change with the run playbooks, the pre playbooks will set up with the released version and we will never test what is being changed - or even what is in the master repos.

When you used a role in tripleo-quickstart-extras to create br-ex, an updated OVS version was already available to be used in the test.

Ronelle Landy (rlandy) on 2018-07-09
tags: added: ci
Changed in tripleo:
milestone: none → rocky-3
importance: Undecided → High
status: New → Triaged
description: updated
wes hayutin (weshayutin) on 2018-07-09
tags: added: alert promotion-blocker
Alan Pevec (apevec) wrote :

Can we define the RCA here?
My take, copying from comment in https://github.com/rdo-infra/rdo-release/issues/10
OVS installed by openstack-infra could be and afaik is updated by the tripleoci, problem might be 2.6 -> 2.9 upgrade steps are missing i.e. restart ovs service since this is not done anymore by the RPM %post to avoid uncontrolled data plane outages during updates.

wes hayutin (weshayutin) wrote :

RCA,

We have ovs packages centos that are old.
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-ocata/

openvswitch-2.6.1-4.1.git20161206.el7.x86_64.rpm

This should be updated to openvswitch 2.9 from newton --> rocky

Ronelle Landy (rlandy) wrote :

https://review.openstack.org/#/c/580518/ Switch to centos-release-openstack-queens for ovs
and
https://review.openstack.org/#/c/581116/ Reparent scenario007 and 008 to tripleo-ci-base

should allow the move to zuulv3 to continue.

wrt what steps we should take in CI to update OVS, that discussion is still on going.

wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.