functional test failures caused by failure to setup OVS bridge

[Kevin Benton]

Right now we get random functional test failures and it seems to be related to the fact that we only log and error and continue when a flow fails to be inserted into a bridge: http://logs.openstack.org/90/283790/10/check/gate-neutron-dsvm-functional/ab54e0a/logs/neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_ipv6_native_.log.txt.gz#_2016-03-01_01_47_17_903

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286428

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286428
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a6c231cdd731dae900c372799a890407bbc24a2c
Submitter: Jenkins
Branch: master

commit a6c231cdd731dae900c372799a890407bbc24a2c
Author: Kevin Benton <email address hidden>
Date: Mon Feb 29 21:28:48 2016 -0800

    Make run_ofctl check for socket error

    When the OVS bridge is still being initialized we get
    a "failed to connect to socket" error when running ovs-ofctl.
    This shows up quite frequently in our functional tests and
    may be the source of their high failure rate.

    Ultimately we need to change the behavior of run_ofctl to not
    ignore errors by default, but this will require a lot of effort
    because there are many places that likely expect this behavior.

    As a workaround, this patch checks for the specific socket failure
    and attempts the command again up to 10 times, sleeping for 1
    second between each attempt to wait for the bridge to be ready.

    Closes-Bug: #1550623
    Closes-Bug: #1551593
    Change-Id: I663a54608ed96133014104fe033ecea0a867ac4c

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/297034

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/297035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297035
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5390cc8af2dba69f3823fe8fdaffb579d528fe62
Submitter: Jenkins
Branch: stable/kilo

commit 5390cc8af2dba69f3823fe8fdaffb579d528fe62
Author: Kevin Benton <email address hidden>
Date: Mon Feb 29 21:28:48 2016 -0800

    Make run_ofctl check for socket error

    When the OVS bridge is still being initialized we get
    a "failed to connect to socket" error when running ovs-ofctl.
    This shows up quite frequently in our functional tests and
    may be the source of their high failure rate.

    Ultimately we need to change the behavior of run_ofctl to not
    ignore errors by default, but this will require a lot of effort
    because there are many places that likely expect this behavior.

    As a workaround, this patch checks for the specific socket failure
    and attempts the command again up to 10 times, sleeping for 1
    second between each attempt to wait for the bridge to be ready.

    Conflicts:
     neutron/agent/common/ovs_lib.py

    Closes-Bug: #1550623
    Closes-Bug: #1551593
    Change-Id: I663a54608ed96133014104fe033ecea0a867ac4c
    (cherry picked from commit a6c231cdd731dae900c372799a890407bbc24a2c)
    (cherry picked from commit eec85f361ecd8af96c73d0c8ff1c9fb1d347220a)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 2015.1.4 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297034
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=eec85f361ecd8af96c73d0c8ff1c9fb1d347220a
Submitter: Jenkins
Branch: stable/liberty

commit eec85f361ecd8af96c73d0c8ff1c9fb1d347220a
Author: Kevin Benton <email address hidden>
Date: Mon Feb 29 21:28:48 2016 -0800

    Make run_ofctl check for socket error

    When the OVS bridge is still being initialized we get
    a "failed to connect to socket" error when running ovs-ofctl.
    This shows up quite frequently in our functional tests and
    may be the source of their high failure rate.

    Ultimately we need to change the behavior of run_ofctl to not
    ignore errors by default, but this will require a lot of effort
    because there are many places that likely expect this behavior.

    As a workaround, this patch checks for the specific socket failure
    and attempts the command again up to 10 times, sleeping for 1
    second between each attempt to wait for the bridge to be ready.

    Conflicts:
     neutron/agent/common/ovs_lib.py

    Closes-Bug: #1550623
    Closes-Bug: #1551593
    Change-Id: I663a54608ed96133014104fe033ecea0a867ac4c
    (cherry picked from commit a6c231cdd731dae900c372799a890407bbc24a2c)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 7.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 2015.1.4 release.