fullstack: test_connectivity fails due to dhclient crash

[Jakub Libosvar]

http://logs.openstack.org/66/515566/7/check/legacy-neutron-dsvm-fullstack/92fcb8e/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_GRE-l2pop-arp_responder,openflow-native_ovsdb-native_.txt.gz#_2017-10-31_06_52_37_750

The test fails because ping is not successful probably because instances don't get ips:

http://logs.openstack.org/66/515566/7/check/legacy-neutron-dsvm-fullstack/92fcb8e/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_GRE-l2pop-arp_responder,openflow-native_ovsdb-native_.txt.gz#_2017-10-31_06_52_18_204

2017-10-31 06:52:18.204 25430 DEBUG neutron.agent.linux.async_process [-] Halting async process [ip netns exec test-42effae2-ba0c-45be-95e3-85e7a8b6315f dhclient -sf /opt/stack/new/neutron/.tox/dsvm-fullstack/bin/fullstack-dhclient-script --no-pid -d port6b08d0] in response to an error. _handle_process_error neutron/agent/linux/async_process.py:196

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/523518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523518
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4f00cebdc75b5bbded664e8a3d09fc08d2936344
Submitter: Zuul
Branch: master

commit 4f00cebdc75b5bbded664e8a3d09fc08d2936344
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Nov 28 12:01:28 2017 -0800

    fullstack: disable all test_connectivity test cases

    Those are known to be unstable. Disable them while we figure out why
    they fail.

    Change-Id: Iae72fd9ae208ee9d8821376cb5f841a8fc65fbc7
    Related-Bug: #1728948

[Slawek Kaplonski (slaweq)]

It affects many different fullstack tests.
For example one of tests in http://logs.openstack.org/88/545688/1/check/neutron-fullstack/535a24c/logs/testr_results.html.gz failed due to this reason.
Logs from this example fail (with additional output also): http://logs.openstack.org/88/545688/1/check/neutron-fullstack/535a24c/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetworkOnOvsBridgeControllerStop.test_controller_timeout_does_not_break_connectivity_sigkill_VXLAN,openflow-native_.txt.gz#_2018-02-18_20_35_59_818

[Slawek Kaplonski (slaweq)]

I tried to add "respawn_timeout" to dhclient process and it looks that this helps for this issue.
For example in http://logs.openstack.org/78/545778/1/check/neutron-fullstack/050ba41/logs/dsvm-fullstack-logs/TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart_LB,Flat-network_.txt.gz#_2018-02-19_09_59_44_177 - dhclient is stopped due to error, then it is restarted 10 seconds later and test is passed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545820

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/545964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545820
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92959238a3e408e810ccd4f3d3d453a35afb5bba
Submitter: Zuul
Branch: master

commit 92959238a3e408e810ccd4f3d3d453a35afb5bba
Author: Sławek Kapłoński <email address hidden>
Date: Mon Feb 19 13:28:57 2018 +0100

    [Fullstack] Respawn dhclient process in case of error

    When dhclient process is started as async process by fake machine resource,
    it might happen that "None" will be returned as first line of output
    from this process. This is treated as an error and dhclient is halted
    immediately.
    Because of that fake machine don't have configured IP address and
    test fails.

    This patch adds "respawn_timeout" value set to 5 seconds for dhclient
    async process. When dhclient process is restarted it should works fine
    and IP address should be then configured properly.

    Change-Id: Ie056578abbe6e18c8415c6e61d755f2248a70541
    Closes-Bug: #1728948

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/547386

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/547387

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547386
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=24fb0be51ca7ba8c220662dd1b2803ef3de26845
Submitter: Zuul
Branch: stable/queens

commit 24fb0be51ca7ba8c220662dd1b2803ef3de26845
Author: Sławek Kapłoński <email address hidden>
Date: Mon Feb 19 13:28:57 2018 +0100

    [Fullstack] Respawn dhclient process in case of error

    When dhclient process is started as async process by fake machine resource,
    it might happen that "None" will be returned as first line of output
    from this process. This is treated as an error and dhclient is halted
    immediately.
    Because of that fake machine don't have configured IP address and
    test fails.

    This patch adds "respawn_timeout" value set to 5 seconds for dhclient
    async process. When dhclient process is restarted it should works fine
    and IP address should be then configured properly.

    Change-Id: Ie056578abbe6e18c8415c6e61d755f2248a70541
    Closes-Bug: #1728948
    (cherry picked from commit 92959238a3e408e810ccd4f3d3d453a35afb5bba)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/545964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547387
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=82167ddca384e54b743446f02392b25c1d1084b0
Submitter: Zuul
Branch: stable/pike

commit 82167ddca384e54b743446f02392b25c1d1084b0
Author: Sławek Kapłoński <email address hidden>
Date: Mon Feb 19 13:28:57 2018 +0100

    [Fullstack] Respawn dhclient process in case of error

    When dhclient process is started as async process by fake machine resource,
    it might happen that "None" will be returned as first line of output
    from this process. This is treated as an error and dhclient is halted
    immediately.
    Because of that fake machine don't have configured IP address and
    test fails.

    This patch adds "respawn_timeout" value set to 5 seconds for dhclient
    async process. When dhclient process is restarted it should works fine
    and IP address should be then configured properly.

    Change-Id: Ie056578abbe6e18c8415c6e61d755f2248a70541
    Closes-Bug: #1728948
    (cherry picked from commit 92959238a3e408e810ccd4f3d3d453a35afb5bba)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.4 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/575675

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575675
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fd572b1e4e3ba5aa0e9c4cdc2d874fde8eaff9c7
Submitter: Zuul
Branch: stable/ocata

commit fd572b1e4e3ba5aa0e9c4cdc2d874fde8eaff9c7
Author: Sławek Kapłoński <email address hidden>
Date: Mon Feb 19 13:28:57 2018 +0100

    [Fullstack] Respawn dhclient process in case of error

    When dhclient process is started as async process by fake machine resource,
    it might happen that "None" will be returned as first line of output
    from this process. This is treated as an error and dhclient is halted
    immediately.
    Because of that fake machine don't have configured IP address and
    test fails.

    This patch adds "respawn_timeout" value set to 5 seconds for dhclient
    async process. When dhclient process is restarted it should works fine
    and IP address should be then configured properly.

    Change-Id: Ie056578abbe6e18c8415c6e61d755f2248a70541
    Closes-Bug: #1728948
    (cherry picked from commit 92959238a3e408e810ccd4f3d3d453a35afb5bba)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.