[Stable/Queens] Functional tests neutron.tests.functional.agent.l3.test_ha_router failing 100% times

[Slawek Kaplonski]

Tests from module neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase are failing 100% times since few days in stable/queens branch.
Example of failure: http://logs.openstack.org/78/593078/1/check/neutron-functional/28fe681/logs/testr_results.html.gz

[Slawek Kaplonski (slaweq)]

Same failures happens also on stable/pike branch, see e.g. http://logs.openstack.org/12/577012/3/check/neutron-functional/c7f762d/logs/testr_results.html.gz

[Slawek Kaplonski (slaweq)]

And ocata also, see: http://logs.openstack.org/26/593026/1/check/neutron-functional/2d2ff19/logs/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/596488

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/596500

[Slawek Kaplonski (slaweq)]

After some checks it looks for me that reason of issue is in "broken" keepalived package. In master branch version "1:1.3.9-1ubuntu0.18.04.1~cloud0" is used and this one works fine. In Queens, Pike and Ocata patches it uses version "1:1.2.24-1ubuntu0.16.04.1" which seems to be broken and which removes all IP addresses from qr-XXX interface during spawning process.
I checked locally that versions older and newer than 1:1.2.24-1ubuntu0.16.04.1 works fine and on this one version there is this issue every time.

[Slawek Kaplonski (slaweq)]

I reported bug against keepalived package now: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1789045

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/596556

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/596559

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/596560

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/593078
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=159490502e206f474c6828090e8b86c613f9c8db
Submitter: Zuul
Branch: stable/queens

commit 159490502e206f474c6828090e8b86c613f9c8db
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 17 17:14:21 2018 +0200

    cap bandit in test-requirements.txt

    bandit is a linter and is listed in the "blacklist" from the
    requirements repo, so it does not appear in the constraints lists.
    Project teams are expected to manage the verions(s) allowed on their
    own, to allow different teams to roll ahead to new versions as they can
    rather than having the entire community do it in lock-step. This change
    caps the version of bandit to the one available during the rocky
    development cycle to avoid introducing the new rules from newer releases
    into a stable branch.

    This patch also changes to use older keepalived version in functional
    tests.
    This issue is reported in bug 1788185.

    It looks that current keepalived version which is available in
    Ubuntu Xenial repositories (1:1.2.24-1ubuntu0.16.04.1) is broken
    and cause failure of some functional tests in Neutron.
    Details are in [1].
    Older version works fine so as temporary solution we can use
    this version in functional tests.

    This issue don't happens on master and stable/rocky branch, as there
    newer cloud-archive repo is used and it has newer version of keepalived
    which works fine.

    [1] https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1789045

    Change-Id: Ia59de069b29f584cce21163a77812ec0ed243e65
    Closes-Bug: #1788185

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/596556
Reason: it's already merged together with "cap bandit version" patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/596488

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/596500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596560
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=01d82626646963d3f614d4586b1285d5dad58a87
Submitter: Zuul
Branch: stable/ocata

commit 01d82626646963d3f614d4586b1285d5dad58a87
Author: Slawek Kaplonski <email address hidden>
Date: Sat Aug 25 22:15:22 2018 +0200

    Use older keepalived version in functional tests

    It looks that current keepalived version which is available in
    Ubuntu Xenial repositories (1:1.2.24-1ubuntu0.16.04.1) is broken
    and cause failure of some functional tests in Neutron.
    Details are in [1].
    Older version works fine so as temporary solution we can use
    this version in functional tests.

    This issue don't happens on master and stable/rocky branch, as there
    newer cloud-archive repo is used and it has newer version of keepalived
    which works fine.

    [1] https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1789045

    Change-Id: I418a967cd503991736e72134d4a105b6e97021e8
    Closes-Bug: #1788185

[Dr. Jens Harbott (j-harbott)]

I tested a bit with 1.2.24 and it seems that it is doing the correct thing as opposed to 1.2.19 in that it removes the virtual IP from the qr- interface while it is in backup state. When I add a sleep(10) to neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ha_router_process_ipv6_subnets_to_existing_port, the test passes in that scenario[0]. So I think the correct fix for this particular test would be to have it wait for router.ha_state == 'master' like it is done for some other tests in the same file.

Also, test_ha_router_process_ipv6_subnets_to_existing_port is the only test that seems to be failing on master with that keepalived version. For stable branches there are some additional failures, but they look to be happening more like 50% of the time only instead of consistently.

I'm still not sure why the issue isn't seen with the version 1.3.9 from Queens UCA.

[0] http://paste.openstack.org/show/729042/

[Dr. Jens Harbott (j-harbott)]

Here are more results from my testing: http://paste.openstack.org/show/729057/

After adding timestamps, it looks like there is some flapping happening with 1.2.24 while it transitions to master state. Adding the sleep makes the test happen after the flapping is over, so it passes, but this still would be a regression in keepalived, then. Sadly no success still with reproducing with standalone keepalived outside of neutron and no time currently for me to continue on this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596559
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=27519e8ff5e11e0fbb69bb2211bae13a015d1654
Submitter: Zuul
Branch: stable/pike

commit 27519e8ff5e11e0fbb69bb2211bae13a015d1654
Author: Slawek Kaplonski <email address hidden>
Date: Sat Aug 25 22:15:22 2018 +0200

    Use older keepalived version in functional tests

    It looks that current keepalived version which is available in
    Ubuntu Xenial repositories (1:1.2.24-1ubuntu0.16.04.1) is broken
    and cause failure of some functional tests in Neutron.
    Details are in [1].
    Older version works fine so as temporary solution we can use
    this version in functional tests.

    This issue don't happens on master and stable/rocky branch, as there
    newer cloud-archive repo is used and it has newer version of keepalived
    which works fine.

    [1] https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1789045

    Change-Id: I418a967cd503991736e72134d4a105b6e97021e8
    Closes-Bug: #1788185

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.4 release.

[Edward Hope-Morley (hopem)]

Looks like this is the real cause of your problems - https://bugs.launchpad.net/neutron/+bug/1793102

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/663363

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663363
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4bdd17a74359eaf42c0653e73e84183dba12491a
Submitter: Zuul
Branch: stable/pike

commit 4bdd17a74359eaf42c0653e73e84183dba12491a
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 17 17:14:21 2018 +0200

    cap bandit in test-requirements.txt

    bandit is a linter and is listed in the "blacklist" from the
    requirements repo, so it does not appear in the constraints lists.
    Project teams are expected to manage the verions(s) allowed on their
    own, to allow different teams to roll ahead to new versions as they can
    rather than having the entire community do it in lock-step. This change
    caps the version of bandit to the one available during the rocky
    development cycle to avoid introducing the new rules from newer releases
    into a stable branch.

    This patch also changes to use older keepalived version in functional
    tests.
    This issue is reported in bug 1788185.

    It looks that current keepalived version which is available in
    Ubuntu Xenial repositories (1:1.2.24-1ubuntu0.16.04.1) is broken
    and cause failure of some functional tests in Neutron.
    Details are in [1].
    Older version works fine so as temporary solution we can use
    this version in functional tests.

    This issue don't happens on master and stable/rocky branch, as there
    newer cloud-archive repo is used and it has newer version of keepalived
    which works fine.

    [1] https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1789045

    Change-Id: Ia59de069b29f584cce21163a77812ec0ed243e65
    Closes-Bug: #1788185
    (cherry picked from commit 159490502e206f474c6828090e8b86c613f9c8db)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron pike-eol release.