Functional tests failing due to failure with getting datapath ID from ovs

[Slawek Kaplonski]

In logs of failed test there is something like:

2020-01-19 09:29:00.139 5422 DEBUG OFPHandler [-] switch features ev version=0x4,msg_type=0x6,msg_len=0x20,xid=0x73ce681f,OFPSwitchFeatures(auxiliary_id=0,capabilities=79,datapath_id=143601332877903,n_buffers=0,n_tables=254) switch_features_handler /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/controller/ofp_handler.py:199
2020-01-19 09:29:00.139 5422 DEBUG OfctlService [-] add dpid 143601332877903 datapath <os_ken.controller.controller.Datapath object at 0x7f7c7f324400> new_info <os_ken.app.ofctl.service._SwitchInfo object at 0x7f7c7f299dd8> old_info None _switch_features_handler /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/app/ofctl/service.py:80
2020-01-19 09:29:00.148 5422 DEBUG OFPHandler [-] move onto main mode multipart_reply_handler /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/controller/ofp_handler.py:231
2020-01-19 09:29:00.888 5422 DEBUG os_ken.base.app_manager [-] EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/base/app_manager.py:314
2020-01-19 09:29:01.889 5422 DEBUG os_ken.base.app_manager [-] EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/base/app_manager.py:314
2020-01-19 09:29:02.890 5422 DEBUG os_ken.base.app_manager [-] EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/base/app_manager.py:314
2020-01-19 09:29:03.892 5422 DEBUG os_ken.base.app_manager [-] EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/base/app_manager.py:314
2020-01-19 09:29:04.893 5422 DEBUG os_ken.base.app_manager [-] EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest send_event /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/os_ken/base/app_manager.py:314

Example of such failure: https://f26da45659020db1220c-76fc92e5e7c4e5a091c792a95503ad1d.ssl.cf5.rackcdn.com/701565/5/check/neutron-functional/4db8d3f/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_policy_rule_delete_egress_.txt

It may be issue in os-ken but for now I open it for Neutron to track it here. If this will be os-ken bug, we can open new issue in storyboard.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/705069

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704397
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=740741864adc142b9bb037e92fd563cae124046a
Submitter: Zuul
Branch: master

commit 740741864adc142b9bb037e92fd563cae124046a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 27 19:15:19 2020 +0000

    Remove sleep command when retrieving OVS dp

    OVS agent is a single thread module executed on a os-ken AppManager
    context. os-ken uses, by default (and no other implementation is
    available today [1]), "eventlet" threads. Those threads are scheduled
    manually by the code itself; the context switch is done through
    yielding. The easiest way to do this is by executing:
      eventlet.sleep()

    If the assigned thread is not ready to take the GIL and do not yield
    back the executor, other threads will starve and eventually will
    timeout.

    This patch removes the "sleep" command during the DP retrieval. This
    will keep the executor on the current thread and will prevent the
    execution timeouts, as seen in the bug related.

    [1]https://opendev.org/openstack/os-ken/src/commit/1f751b2d7d385a75d4255934694177d6fd6f012f/os_ken/lib/hub.py

    Closes-Bug: #1861269

    Change-Id: I19e1af1bda788ed970d30ab251e895f7daa11e39

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/705398

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/705399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/705400

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/705401

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705399
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=63ff0486a2c3f92d055350bcc06a41219031fcd9
Submitter: Zuul
Branch: stable/stein

commit 63ff0486a2c3f92d055350bcc06a41219031fcd9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 27 19:15:19 2020 +0000

    Remove sleep command when retrieving OVS dp

    OVS agent is a single thread module executed on a os-ken AppManager
    context. os-ken uses, by default (and no other implementation is
    available today [1]), "eventlet" threads. Those threads are scheduled
    manually by the code itself; the context switch is done through
    yielding. The easiest way to do this is by executing:
      eventlet.sleep()

    If the assigned thread is not ready to take the GIL and do not yield
    back the executor, other threads will starve and eventually will
    timeout.

    This patch removes the "sleep" command during the DP retrieval. This
    will keep the executor on the current thread and will prevent the
    execution timeouts, as seen in the bug related.

    [1]https://opendev.org/openstack/os-ken/src/commit/1f751b2d7d385a75d4255934694177d6fd6f012f/os_ken/lib/hub.py

    Closes-Bug: #1861269

    Change-Id: I19e1af1bda788ed970d30ab251e895f7daa11e39
    (cherry picked from commit 740741864adc142b9bb037e92fd563cae124046a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705398
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e59a9f6dbc9072126f640a9134bd76cfedbf42a
Submitter: Zuul
Branch: stable/train

commit 1e59a9f6dbc9072126f640a9134bd76cfedbf42a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 27 19:15:19 2020 +0000

    Remove sleep command when retrieving OVS dp

    OVS agent is a single thread module executed on a os-ken AppManager
    context. os-ken uses, by default (and no other implementation is
    available today [1]), "eventlet" threads. Those threads are scheduled
    manually by the code itself; the context switch is done through
    yielding. The easiest way to do this is by executing:
      eventlet.sleep()

    If the assigned thread is not ready to take the GIL and do not yield
    back the executor, other threads will starve and eventually will
    timeout.

    This patch removes the "sleep" command during the DP retrieval. This
    will keep the executor on the current thread and will prevent the
    execution timeouts, as seen in the bug related.

    [1]https://opendev.org/openstack/os-ken/src/commit/1f751b2d7d385a75d4255934694177d6fd6f012f/os_ken/lib/hub.py

    Closes-Bug: #1861269

    Change-Id: I19e1af1bda788ed970d30ab251e895f7daa11e39
    (cherry picked from commit 740741864adc142b9bb037e92fd563cae124046a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705401
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=19e6f610989ef5e649b00405804c9c5c58278662
Submitter: Zuul
Branch: stable/queens

commit 19e6f610989ef5e649b00405804c9c5c58278662
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 27 19:15:19 2020 +0000

    Remove sleep command when retrieving OVS dp

    OVS agent is a single thread module executed on a os-ken AppManager
    context. os-ken uses, by default (and no other implementation is
    available today [1]), "eventlet" threads. Those threads are scheduled
    manually by the code itself; the context switch is done through
    yielding. The easiest way to do this is by executing:
      eventlet.sleep()

    If the assigned thread is not ready to take the GIL and do not yield
    back the executor, other threads will starve and eventually will
    timeout.

    This patch removes the "sleep" command during the DP retrieval. This
    will keep the executor on the current thread and will prevent the
    execution timeouts, as seen in the bug related.

    [1]https://opendev.org/openstack/os-ken/src/commit/1f751b2d7d385a75d4255934694177d6fd6f012f/os_ken/lib/hub.py

    Closes-Bug: #1861269

    Change-Id: I19e1af1bda788ed970d30ab251e895f7daa11e39
    (cherry picked from commit 740741864adc142b9bb037e92fd563cae124046a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=54b00629639faa27611ab5d5ee6c788c217d964b
Submitter: Zuul
Branch: stable/rocky

commit 54b00629639faa27611ab5d5ee6c788c217d964b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 27 19:15:19 2020 +0000

    Remove sleep command when retrieving OVS dp

    OVS agent is a single thread module executed on a os-ken AppManager
    context. os-ken uses, by default (and no other implementation is
    available today [1]), "eventlet" threads. Those threads are scheduled
    manually by the code itself; the context switch is done through
    yielding. The easiest way to do this is by executing:
      eventlet.sleep()

    If the assigned thread is not ready to take the GIL and do not yield
    back the executor, other threads will starve and eventually will
    timeout.

    This patch removes the "sleep" command during the DP retrieval. This
    will keep the executor on the current thread and will prevent the
    execution timeouts, as seen in the bug related.

    [1]https://opendev.org/openstack/os-ken/src/commit/1f751b2d7d385a75d4255934694177d6fd6f012f/os_ken/lib/hub.py

    Closes-Bug: #1861269

    Change-Id: I19e1af1bda788ed970d30ab251e895f7daa11e39
    (cherry picked from commit 740741864adc142b9bb037e92fd563cae124046a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.7 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/705069

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.