Functional job times out on xenial

[Jakub Libosvar]

logstash-querry: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22gate-neutron-dsvm-functional-ubuntu-xenial%5C%22%20AND%20build_status%3A%5C%22FAILURE%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20message%3A%5C%22Killed%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20timeout%20-s%209%20%5C%22

9 hits in 2 days

[Jakub Libosvar (libosvar)]

Example of failed job: http://logs.openstack.org/52/400852/3/check/gate-neutron-dsvm-functional-ubuntu-xenial/4ecb4db/

[Jakub Libosvar (libosvar)]

It looks like ovsdb communication is malfunctioned: http://logs.openstack.org/52/400852/3/check/gate-neutron-dsvm-functional-ubuntu-xenial/4ecb4db/logs/dsvm-functional-logs/neutron.tests.functional.agent.linux.test_ovsdb_monitor.TestSimpleInterfaceMonitor.test_get_events_includes_ofport_native_.txt.gz

The error starts there when commands start returning TRY_AGAIN

[Jakub Libosvar (libosvar)]

This bug blocks transition to Xenial which is due on 6th Dec

[Jakub Libosvar (libosvar)]

Related patch pushed with wrong commit message https://review.openstack.org/#/c/404795/

[IWAMOTO Toshihiro (iwamoto)]

From the logstash-query, there were 19 build failures and 80+ successful builds in last 7 days, meaning ~20% failures may be accountable to this bug.

It looks like ovsdb related, so added otherwiseguy to the list of subscribers. ;)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/405996

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/405996
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1ee3f58c111c0c7698ca8e46b0868fa085290e02
Submitter: Jenkins
Branch: stable/newton

commit 1ee3f58c111c0c7698ca8e46b0868fa085290e02
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 30 09:33:42 2016 -0500

    Don't compile OVS kernel modules for functional job

    The only reason we need newer openvswitch is following fix [1] for
    user-space part of openvswitch. This patch fixes our
    test_install_flood_to_tun tests.

    Current 2.6 branch has issues with ovsdb where connection with native
    interface is getting lost and functional job gets stuck. To mitigate
    that, this patch switches back to OVS 2.5.1 version for functional jobs,
    compiling only user-space part as OVS 2.5.1 kernel modules are not
    supported by Xenial.

    Fullstack job remains running with 2.6 because the job needs OVS kernel patch
    for local VXLAN tunneling.

    [1] https://github.com/openvswitch/ovs/commit/56de2148f626d559a199028952fdd13f301ca80a

    Related-bug: 1646107
    Change-Id: Ic5419afe7170e759749afd7055441c82c317efe0
    (cherry picked from commit b0ea7c95a3e08199a5d72a4dbd6ebf8a54296bdf)

[Jakub Libosvar (libosvar)]

I lowered importance as it currently doesn't fail on the gate.

[Ihar Hrachyshka (ihar-hrachyshka)]

I think this is not happening anymore. I will close the bug.