Tempest tests are failing in fs30(both master/queens) promotion job while doing SSH to VM

[yatin]

featureset030 job failing in promotion jobs(both master/queens) after enabling tempest in [1].

Tempest was disabled/enabled with reference to bug:- https://bugs.launchpad.net/tripleo/+bug/1740698 which had different issue. So details about the issue and fix can be found there.

Now we have different issue so good to have a new bug:-

Logs:-
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/b374b34/undercloud/home/jenkins/tempest/tempest.html.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/4c52bff/undercloud/home/jenkins/tempest/tempest.html.gz

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/b374b34/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-03-19_11_49_34_072

These ^^ Errors might be because of ovsdbapp(0.10.0) not updated in master yet.

Copying comment from bug to have some details here:-

Looking further i found that SSH keys are injected but not that which are provided by ovn metadata(or passed during vm creation) but by metadata agent running for rdocloud.

The same hostname of undercloud([1]) is assigned to vm spawned(nova console-log see [2]) by tempest on overcloud:-
upstream-centos-7-2-node-rdo-cloud-tripleo-102250.rdocloud, Also the keys injected is multinode-admins keys

[1] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/4c52bff/undercloud/etc/hostname.txt.gz
[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/4c52bff/undercloud/home/jenkins/tempest/tempest.html.gz

I don't know what's causing it but may be some networks are conflicting or ovn metadata.

[Lucas Alvares Gomes (lucasagomes)]

Hi Yatin,

The error related to the "ipv6_ra_configs" column not existing [0] doesn't seem related to ovsdbapp. The "ipv6_ra_configs" column only exist in the versions >= 2.9 of OVN [1], apparently this job is using the 2.8.2 version [2].

Maybe the reason why we haven't caught that in our gate is because the minimum version being tested by our CI upstream is the 2.9 [3].

Would it be possible to run this job again as a test with the OVS version 2.9 to see if it would fix it ?

[0] KeyError: 'ipv6_ra_configs': https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/b374b34/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-03-19_11_49_34_072

[1] https://github.com/openvswitch/ovs/commit/4446661aca36968205387574ce333ebc45f1b554

[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/b374b34/subnode-2/var/log/containers/openvswitch/ovsdb-server-nb.log.txt.gz

[3] https://review.openstack.org/#/c/540646/

[Miguel Angel Ajo (mangelajo)]

We depend on OVS 2.9 for Queens

[Lucas Alvares Gomes (lucasagomes)]

Just for the record, probably not a valid because I didn't have a TripleO setup handy but it might give people a hint about this failure.

I tried to run the "test_network_basic_ops" tempest test in a local setup with DevStack and it was giving me a similar error as in this ticket:

$ tox -eall-plugin tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops

...

AssertionError: False is not true : Timed out waiting for 172.24.4.3 to become reachable

( See full logs of the failure here: http://paste.openstack.org/show/707558/ )

I then noticed that br-ex wasn't configured, so I ran:

$ sudo ip link set br-ex up
$ sudo ip route add 172.24.4.0/24 dev br-ex
$ sudo ip addr add 172.24.4.1/24 dev br-ex

Tried again and this time it worked: http://paste.openstack.org/show/707569/

[Lucas Alvares Gomes (lucasagomes)]

Here's the fix/workaround for the ipv6_ra_column problem: https://review.openstack.org/554930

Once merged I will backport it to Queens.

[yatin (yatinkarel)]

I checked again and found that ovn_metadata_enabled=True (default is False) is not set in [ovn] section, not only this other config([1]) for ovn ml2 are not set: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/41d08fc/subnode-2/var/log/config-data/neutron/etc/neutron/plugins/ml2/ml2_conf.ini.txt.gz

[1] https://github.com/openstack/tripleo-heat-templates/blob/df31016a9af5003533f80989bcb8d3da42099953/puppet/services/neutron-plugin-ml2-ovn.yaml#L96-L103

And reason i found is the part ^^ which set's ovn metadata config for ml2 is not included in scenario007. I tried this locally and it worked. Will push a patch for it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/556350

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556350
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e534a7a213294bdf8908addaa3b3da4e070a78dc
Submitter: Zuul
Branch: master

commit e534a7a213294bdf8908addaa3b3da4e070a78dc
Author: yatin <email address hidden>
Date: Mon Mar 26 11:28:58 2018 +0530

    Correctly set NeutronMl2PluginBase for OVN scenario

    OVN configuration was not done when deployed with
    scenario007 as default for NeutronMl2PluginBase was
    used which is neutron-plugin-ml2.yaml. This patch
    fixes this to use neutron-plugin-ml2-ovn.yaml which
    correctly configures neutron for ovn metadata.

    Change-Id: I7cadd0567951b85c1ba69d4b4843ee29b67e7a11
    Closes-Bug: #1757134

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/556823

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556823
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8c22a649cb3cc7170744ab55371a6a882a674207
Submitter: Zuul
Branch: stable/queens

commit 8c22a649cb3cc7170744ab55371a6a882a674207
Author: yatin <email address hidden>
Date: Mon Mar 26 11:28:58 2018 +0530

    Correctly set NeutronMl2PluginBase for OVN scenario

    OVN configuration was not done when deployed with
    scenario007 as default for NeutronMl2PluginBase was
    used which is neutron-plugin-ml2.yaml. This patch
    fixes this to use neutron-plugin-ml2-ovn.yaml which
    correctly configures neutron for ovn metadata.

    Change-Id: I7cadd0567951b85c1ba69d4b4843ee29b67e7a11
    Closes-Bug: #1757134
    (cherry picked from commit e534a7a213294bdf8908addaa3b3da4e070a78dc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.2 release.