vpnaas functional test continues to fail with "neutron-keepalived-state-change" failed: No such file or directory

[Akihiro Motoki]

gate-neutron-vpnaas-dsvm-functional neutron_vpnaas.tests.functional.common.test_scenario.TestIPSecScenario. test_ipsec_site_connections_with_l3ha_routers
RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory

For example, http://logs.openstack.org/11/261111/1/check/gate-neutron-vpnaas-dsvm-functional/06c2f50/

It succeeded at Dec 23 03:04UTC and failed Dec 23 20:55UTC
from the results of the recent neutron-vpnaas jenkins runs.

[Akihiro Motoki (amotoki)]

strongswan functional tests are also failing.
http://logs.openstack.org/75/260375/1/gate/gate-neutron-vpnaas-dsvm-functional-sswan/06d1dfc/testr_results.html.gz

[Elena Ezhova (eezhova)]

In the logs of failed jobs I can also see the same errors for neutron-ns-metadata-proxy.

2015-12-23 21:51:30.999 | 2015-12-23 21:51:31.022 | 2015-12-23 21:50:43.699 6948 ERROR neutron.callbacks.manager RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-ns-metadata-proxy" failed: No such file or directory

As well as other "No such file or directory" errors in strongswan functional tests logs.

Meanwhile all functional tests pass on devstack, perhaps there is some misconfiguration in jenkins jobs.

[Akihiro Motoki (amotoki)]

Thanks for taking care of this.
I use https://review.openstack.org/#/c/261224/ for testing now. If you need to change the environment for debugging, feel free to use it.

I saw similar "No such file or directory" errors in the logs.
All expected binaries are installed appropriately.
http://logs.openstack.org/24/261224/1/check/gate-neutron-vpnaas-dsvm-functional/cb862ca/console.html#_2015-12-24_08_59_46_616

I think exit code 1 comes from sudo.
If rootwrap fails to find a program, it returns a bigger code than 127(?).

[Akihiro Motoki (amotoki)]

Looking at http://logs.openstack.org/11/261111/1/check/gate-neutron-vpnaas-dsvm-functional/06c2f50/logs/sudoers.d/60-neutron-func-test-rootwrap.txt.gz
the path of tox env has "-constraints" like /opt/stack/new/neutron-vpnaas/.tox/dsvm-functional-constraints/etc/neutron/rootwrap.conf
In neutron-vpnaas functional test, we use "dsvm-functional" and "-constraints" should not be included now.

neutron commit fce50f3beb19dfa747ca12d11c8a0f70f7ea16a8 added "-constraints" and it was merged at Dec 23 16:35:13 2015 +0000, so the timing where the failure starts matches this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/261534

[Elena Ezhova (eezhova)]

Thank you for your research, Akihiro! Your change fixes the tests.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261534
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=477b390b328e4e66a2242d346a08b81a29408908
Submitter: Jenkins
Branch: master

commit 477b390b328e4e66a2242d346a08b81a29408908
Author: Akihiro Motoki <email address hidden>
Date: Fri Dec 25 23:27:03 2015 +0900

    Allow to control to use constraint env for functional jobs

    VPNaaS functional tests now use dsvm-functional instead of
    *-constraints. Always assuming *-constraints breaks the functional
    jobs which is not run in a constrainted environment yet.

    Paritial-Bug: #1529027
    Change-Id: I3a4f1d61d13f1e444f7624d68d121d04d81eb724

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261224
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=a1cabc194320072a84576e2fa5654fc90c377471
Submitter: Jenkins
Branch: master

commit a1cabc194320072a84576e2fa5654fc90c377471
Author: Akihiro Motoki <email address hidden>
Date: Thu Dec 24 17:49:59 2015 +0900

    Don't use constrained environment for functional tests

    Closes-Bug: #1529027
    Depends-On: I3a4f1d61d13f1e444f7624d68d121d04d81eb724
    Change-Id: I8291c7cded5fad3227cda0d301dae24e110b6258

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron-vpnaas 8.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/321541

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321541
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bff727bc199eea8af10a495550207b128092199f
Submitter: Jenkins
Branch: stable/liberty

commit bff727bc199eea8af10a495550207b128092199f
Author: Akihiro Motoki <email address hidden>
Date: Fri Dec 25 23:27:03 2015 +0900

    Allow to control to use constraint env for functional jobs

    VPNaaS functional tests now use dsvm-functional instead of
    *-constraints. Always assuming *-constraints breaks the functional
    jobs which is not run in a constrainted environment yet.

    Paritial-Bug: #1529027
    Change-Id: I3a4f1d61d13f1e444f7624d68d121d04d81eb724
    (cherry picked from commit 477b390b328e4e66a2242d346a08b81a29408908)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283693
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=865b5f7032c9f68fb011b108d32aba1c7cbd5120
Submitter: Jenkins
Branch: stable/liberty

commit 865b5f7032c9f68fb011b108d32aba1c7cbd5120
Author: Paul Michali <email address hidden>
Date: Tue Feb 16 15:45:54 2016 -0500

    Use constraints on all targets.

    The neutron-vpnaas repo already has converted to using constraints for
    targets. However, the TC recommendation is to use constraints on the
    original job names (pep8, py27,...), instead of using the constrained
    names (e.g. pep8-constraints).

    A separate commit (I34f0b355703fe0133f160afd4e32f2272d41ecaa) restored
    the use of the non-constraint NAMED jobs for Liberty and newer.

    This commit modifies those non-constraints targets so that they use
    constraints. A manual check was made that each target used the same
    package versions for both constraints and non-constraints targets.

    Then, after both constraints and non-constraints jobs passed in Jenkins
    (before and after the project-config commit was upstreamed), the constraints
    based targets were removed from tox.ini.

    Note: Found out that infra doesn't support constraints based cover and
    releasenotes. Added TODO to these targets so that they can be updated,
    if and when constraints are supported.

    Note: In the gate hook, config_for_vpn_func_testing.sh calls Neutron's
    config_for_func_testing.sh, which will setup files in the venv. To prevent
    that script from modifying the VENV variable to have -constraints suffix
    and store files in the wrong venv, we turn off USE_CONSTRAINT_ENV, prior
    to calling the Neutron script.

    Once this is upstreamed, we can backport this fix to Liberty to complete
    the effort for this repo.

    Conflicts:
     neutron_vpnaas/tests/contrib/gate_hook.sh
     tox.ini

    Depends-On: I34f0b355703fe0133f160afd4e32f2272d41ecaa
    Partial-Bug: #1522503
    (cherry picked from commit 73a6ad9a13d4b6c2d9efe7d9ec142514fc6904f2)

    Also squashed the following follow up fixes:

    ===

    VPNaaS: Cleanup constraints in tox.ini

    Missed changing the envlist to not specify constraints based target
    names for master (where constraints are used by default). This
    simple commit cleans that up.

    Conflicts:
     tox.ini

    Change-Id: I9d11cfd47adae3f49d33e5db8c916ebd983a8f96
    Related-Bug: #1522503
    (cherry picked from commit ad122cf23c296c9dc657a2d92abcb0ff316e8d99)

    ===

    Fix tox.ini constraints for post jobs

    Post jobs are currently failing since constraints are not available but
    required by this tox.ini file. Change venv environment to work without
    constraints.

    Change-Id: I736be6a49f919302c31fc7ebd8251cb0a0d73853
    (cherry picked from commit cf9e80a16fbfcadd39322760969b342a8191cc96)

    ===

    Don't use constrained environment for functional tests

    Closes-Bug: #1529027
    Depends-On: I3a4f1d61d13f1e444f7624d68d121d04d81eb724
    Change-Id: I8291c7cded5fad3227cda0d301dae24e110b6258
    (cherry ...

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/neutron-vpnaas 7.1.1 release.