multi-node-bridge role in zuul-jobs uses a released OVS package

[Ronelle Landy]

Tempest test test_network_basic_ops running on master with scenario007 and scenario008 jobs fails with networking/connection issues:

http://logs.openstack.org/53/579653/3/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/5e299dc/logs/undercloud/home/zuul/tempest/tempest.html.gz

The problem stems from that fact that we now set up br-ex using pre.yaml from zuul-jobs which relies on a released version of openvswitch:

https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/vars/RedHat.yaml#L4

https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L21

The role defaulted to the ocata version - hence the incompatibility with master deployments.
I was able to get the scenario007 to pass in my local reproducer environment when using the released queens version of openvswitch.

The methodology here is problematic. If we plan to test TripleO integration with OVS, and we only bring in the change with the run playbooks, the pre playbooks will set up with the released version and we will never test what is being changed - or even what is in the master repos.

When you used a role in tripleo-quickstart-extras to create br-ex, an updated OVS version was already available to be used in the test.

[Alan Pevec (apevec)]

Can we define the RCA here?
My take, copying from comment in https://github.com/rdo-infra/rdo-release/issues/10
OVS installed by openstack-infra could be and afaik is updated by the tripleoci, problem might be 2.6 -> 2.9 upgrade steps are missing i.e. restart ovs service since this is not done anymore by the RPM %post to avoid uncontrolled data plane outages during updates.

[wes hayutin (weshayutin)]

RCA,

We have ovs packages centos that are old.
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-ocata/

openvswitch-2.6.1-4.1.git20161206.el7.x86_64.rpm

This should be updated to openvswitch 2.9 from newton --> rocky

[Ronelle Landy (rlandy)]

https://review.openstack.org/#/c/580518/ Switch to centos-release-openstack-queens for ovs
and
https://review.openstack.org/#/c/581116/ Reparent scenario007 and 008 to tripleo-ci-base

should allow the move to zuulv3 to continue.

wrt what steps we should take in CI to update OVS, that discussion is still on going.

[wes hayutin (weshayutin)]

https://review.openstack.org/#/c/580518/4/roles/multi-node-bridge/tasks/common.yaml