Neutron functional job is broken

[Kashyap Chamarthy]

I encountered this as part of "check-neutron-dsvm-functional" test job failure from Jenkins.

Contextual log snippet (from console.html)[1]:
----------------------------------------------------
.
.
.
2014-07-01 03:31:05.323 | 2014-07-01 03:31:05.304 | Command: ['sudo', 'ovs-vsctl', '--timeout=10', 'get', 'Interface', 'peertest1-e9c183', 'ofport']
2014-07-01 03:31:05.325 | 2014-07-01 03:31:05.306 | Exit code: 1
2014-07-01 03:31:05.327 | 2014-07-01 03:31:05.308 | Stdout: ''
2014-07-01 03:31:05.329 | 2014-07-01 03:31:05.310 | Stderr: 'sudo: no tty present and no askpass program specified\n'
2014-07-01 03:31:05.331 | 2014-07-01 03:31:05.312 | 2014-07-01 03:31:05,077 ERROR [neutron.agent.linux.utils]
2014-07-01 03:31:05.333 | 2014-07-01 03:31:05.314 | Command: ['sudo', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-br', 'patchtest-e9c183']
2014-07-01 03:31:05.336 | 2014-07-01 03:31:05.316 | Exit code: 1
2014-07-01 03:31:05.338 | 2014-07-01 03:31:05.319 | Stdout: ''
2014-07-01 03:31:05.340 | 2014-07-01 03:31:05.321 | Stderr: 'sudo: no tty present and no askpass program specified\n'
2014-07-01 03:31:05.342 | 2014-07-01 03:31:05.323 | 2014-07-01 03:31:05,078 ERROR [neutron.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-br', 'patchtest-e9c183']. Exception:
2014-07-01 03:31:05.345 | 2014-07-01 03:31:05.325 | Command: ['sudo', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-br', 'patchtest-e9c183']
2014-07-01 03:31:05.347 | 2014-07-01 03:31:05.328 | Exit code: 1
2014-07-01 03:31:05.349 | 2014-07-01 03:31:05.330 | Stdout: ''
2014-07-01 03:31:05.351 | 2014-07-01 03:31:05.332 | Stderr: 'sudo: no tty present and no askpass program specified\n'
2014-07-01 03:31:05.353 | 2014-07-01 03:31:05.334 | }}}
2014-07-01 03:31:05.355 | 2014-07-01 03:31:05.336 |
2014-07-01 03:31:05.357 | 2014-07-01 03:31:05.338 | Traceback (most recent call last):
2014-07-01 03:31:05.359 | 2014-07-01 03:31:05.340 | File "neutron/tests/functional/sanity/test_sanity.py", line 58, in test_ovs_patch_support_runs
2014-07-01 03:31:05.361 | 2014-07-01 03:31:05.342 | checks.patch_supported(self.root_helper)
2014-07-01 03:31:05.363 | 2014-07-01 03:31:05.343 | File "neutron/cmd/sanity/checks.py", line 35, in patch_supported
2014-07-01 03:31:05.365 | 2014-07-01 03:31:05.346 | port = br.add_patch_port(patch_name, peer_name)
2014-07-01 03:31:05.367 | 2014-07-01 03:31:05.348 | File "neutron/agent/linux/ovs_lib.py", line 275, in add_patch_port
2014-07-01 03:31:05.369 | 2014-07-01 03:31:05.350 | return self.get_port_ofport(local_name)
2014-07-01 03:31:05.371 | 2014-07-01 03:31:05.352 | File "neutron/agent/linux/ovs_lib.py", line 183, in get_port_ofport
2014-07-01 03:31:05.373 | 2014-07-01 03:31:05.354 | int(ofport)
2014-07-01 03:31:05.375 | 2014-07-01 03:31:05.356 | TypeError: int() argument must be a string or a number, not 'NoneType'
2014-07-01 03:31:05.377 | 2014-07-01 03:31:05.358 | ======================================================================
2014-07-01 03:31:05.379 | 2014-07-01 03:31:05.360 | FAIL: process-returncode
2014-07-01 03:31:05.381 | 2014-07-01 03:31:05.362 | tags: worker-2
2014-07-01 03:31:05.383 | 2014-07-01 03:31:05.363 | ----------------------------------------------------------------------
2014-07-01 03:31:05.385 | 2014-07-01 03:31:05.366 | Binary content:
2014-07-01 03:31:05.387 | 2014-07-01 03:31:05.368 | traceback (test/plain; charset="utf8")
2014-07-01 03:31:05.389 | 2014-07-01 03:31:05.369 | ======================================================================
2014-07-01 03:31:05.391 | 2014-07-01 03:31:05.371 | FAIL: process-returncode
2014-07-01 03:31:05.393 | 2014-07-01 03:31:05.374 | tags: worker-2
2014-07-01 03:31:05.395 | 2014-07-01 03:31:05.376 | ----------------------------------------------------------------------
2014-07-01 03:31:05.397 | 2014-07-01 03:31:05.378 | Binary content:
2014-07-01 03:31:05.399 | 2014-07-01 03:31:05.380 | traceback (test/plain; charset="utf8")
2014-07-01 03:31:05.401 | 2014-07-01 03:31:05.382 | Ran 9 tests in 0.367s
2014-07-01 03:31:05.403 | 2014-07-01 03:31:05.384 | FAILED (id=0, failures=4, skips=2)
2014-07-01 03:31:05.405 | 2014-07-01 03:31:05.386 | error: testr failed (1)
2014-07-01 03:31:05.408 | 2014-07-01 03:31:05.388 | ERROR: InvocationError: '/opt/stack/new/neutron/.tox/functional/bin/python setup.py testr --slowest --testr-args='
2014-07-01 03:31:05.409 | 2014-07-01 03:31:05.390 | ___________________________________ summary ____________________________________
2014-07-01 03:31:05.411 | 2014-07-01 03:31:05.392 | ERROR: functional: commands failed
.
.
.
----------------------------------------------------

Here's[2] the logstash query for "int() argument must be a string or a number, not 'NoneType'" -- looking at, 'build_name' in the logs, most of them are 'check-neutron-dsvm-functional'

  [1] http://logs.openstack.org/90/102490/4/check/check-neutron-dsvm-functional/9db189a/console.html#_2014-07-01_03_31_05_126
  [2] http://logstash.openstack.org/#eyJzZWFyY2giOiJcInN1ZG86IG5vIHR0eSBwcmVzZW50IGFuZCBubyBhc2twYXNzIHByb2dyYW0gc3BlY2lmaWVkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDQxOTcwNzI1MDh9

[Kashyap Chamarthy (kashyapc)]

Trying to understand the Traceback:

'get_port_ofport' -- the function where it's 'ofport' is returning None type.
(from ./neutron/agent/linux/ovs_lib.py)
---------------
    178 def get_port_ofport(self, port_name):
    179 ofport = self.db_get_val("Interface", port_name, "ofport")
    180 # This can return a non-integer string, like '[]' so ensure a
    181 # common failure case
    182 try:
    183 int(ofport)
    184 return ofport
    185 except ValueError:
    186 return constants.INVALID_OFPORT
--------------

So, 'ofport' value to be obtained from 'db_get_val' function (refer
below).

'get_port_ofport' is called from 'add_patch_port' (from
./neutron/agent/linux/ovs_lib.py)
--------------
    271 def add_patch_port(self, local_name, remote_name):
    272 self.run_vsctl(["add-port", self.br_name, local_name,
    273 "--", "set", "Interface", local_name,
    274 "type=patch", "options:peer=%s" % remote_name])
    275 return self.get_port_ofport(local_name)
---------------

'add_patch_port' is called from 'patch_supported (from
./neutron/cmd/sanity/checks.py)
--------------
     29 def patch_supported(root_helper):
     30 seed = utils.get_random_string(6)
     31 name = "patchtest-" + seed
     32 peer_name = "peertest0-" + seed
     33 patch_name = "peertest1-" + seed
     34 with ovs_lib.OVSBridge(name, root_helper) as br:
     35 port = br.add_patch_port(patch_name, peer_name)
     36 return port != ovs_const.INVALID_OFPORT
--------------

'patch_supported' is called from 'test_ovs_patch_support_runs' (from
./neutron/tests/functional/sanity/test_sanity.py)
--------------
     57 def test_ovs_patch_support_runs(self):
     58 checks.patch_supported(self.root_helper)
--------------

db_get_val:
--------------
    284 def db_get_val(self, table, record, column, check_error=False):
    285 output = self.run_vsctl(["get", table, record, column], check_error)
    286 if output:
    287 return output.rstrip("\n\r")
--------------

[Paul Michali (pcm)]

I see this on two separate reviews. Wondering if change made to testbed. See https://bugs.launchpad.net/openstack-ci/+bug/1290669

[Kashyap Chamarthy (kashyapc)]

Seems like this should fix it:

  https://review.openstack.org/#/c/103909/ -- Revert "Proper passing of SUDO flag for neutron functional tests"

[Maru Newby (maru)]

This bug covers 2 failures caused by a recent patch that enabled execution of tests in the functional job that require sudo privileges:

 - OVSBridge.db_get_val() can return None if no output is returned from ovs-vsctl, and get_port_ofport() was calling int() on the return value to cause the reported failure. get_port_ofport() needs to be updated to handle the possibility of a None return value from db_get_val().

 - sudo invocations are requiring a password. the functional tests need to run with an account that allows password-less sudo.

[Maru Newby (maru)]

The fix for the second issue will be replacing the use of sudo with rootwrap, which can be invoked password-less by default.

[Maru Newby (maru)]

There's another failure, periodic in nature, that will also be addressed as part of this bug. neutron.tests.functional.agent.linux.test_ovsdb_monitor.TestOvsdbMonitor.test_killed_monitor_respawns() will sometimes fail due to whitespace differences in the comparison strings. gessau has agreed to submit a fix for this problem that ignores the whitespace differences.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104028

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104029

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104042

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/104043

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104005
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=17686c1f24ddf1977bbd930ce94b3e3d76962c5b
Submitter: Jenkins
Branch: master

commit 17686c1f24ddf1977bbd930ce94b3e3d76962c5b
Author: Henry Gessau <email address hidden>
Date: Tue Jul 1 17:56:49 2014 -0400

    Ignore variable column widths in ovsdb functional tests

    The ovsdb tests check the output of "ovsdb-client monitor" before and after
    some operations. The first line of output is a header row with spaces for
    column separation. The column widths can vary depending on the data in the
    columns, so compress multiple spaces to one for testing.

    Change-Id: I6b4690ea9ef1c1b215411ca2c39b1e65fec2c6c7
    Partial-bug: #1336172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104028
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fc91e158f202112ae5cc576ad3e3fa0f89deff1e
Submitter: Jenkins
Branch: master

commit fc91e158f202112ae5cc576ad3e3fa0f89deff1e
Author: Maru Newby <email address hidden>
Date: Tue Jul 1 23:51:15 2014 +0000

    Fix OVSBridge.get_port_ofport to handle empty output

    Previously ovs_lib.OVSBridge.get_port_ofport() was not handling the
    case where db_get_val() was returning None when no output was recieved
    from ovs-vsctl. This patch ensures that the TypeError that results
    from this condition is appropriately handled.

    Change-Id: I168eb0f2c6f0cb98a3eb3e118972bbaf0383ff89
    Partial-bug: #1336172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104029
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=798824cab0322d9de82594dbbdb6c4ead0e1cb2f
Submitter: Jenkins
Branch: master

commit 798824cab0322d9de82594dbbdb6c4ead0e1cb2f
Author: Maru Newby <email address hidden>
Date: Wed Jul 2 00:02:08 2014 +0000

    Allow setting a rootwrap cmd for functional tests

    Previously, sudo-requiring functional tests hardcoded the use of
    'sudo' as the root helper. Devstack gate jobs do not allow
    password-less invocation of 'sudo', though, so such tests were unable
    to run in the gate. This patch adds the ability to configure the
    rootwrap command installed by devstack by setting the OS_ROOTWRAP_CMD
    environment variable in the test execution environment, allowing
    sudo-requiring tests to run.

    Change-Id: I3b8f6b4f14ac1743e08b9401f73951885165350a
    Partial-bug: #1336172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104042
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=37086a8f175ed47776cfc7160aeb29b10060ec8b
Submitter: Jenkins
Branch: master

commit 37086a8f175ed47776cfc7160aeb29b10060ec8b
Author: Maru Newby <email address hidden>
Date: Wed Jul 2 00:35:12 2014 +0000

    Add a gate-specific tox env for functional tests

    The functional gate jobs (*-neutron-dsvm-functional) need special
    configuration (e.g. OS_SUDO_TESTING, OS_ROOTWRAP_CMD) to run tests.
    This patch adds a new gate-specific tox env (tox -e dsvm-functional)
    that sets the appropriate values and allows easy duplication of the
    results of the upstream job on a host that has successfully run
    devstack with neutron enabled.

    A new configuration option, OS_FAIL_ON_MISSING_DEPS, is also set by
    the dsvm-functional tox env to ensure that dependency failures will
    result in failures rather than skips when run in the gate.

    Change-Id: Idbfef1ba72ed129a3021509c7969e2685b8a6d09
    Partial-bug: #1336172

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maru Newby (<email address hidden>) on branch: master
Review: https://review.openstack.org/104043
Reason: Now that the functional job is working this change is no longer necessary.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/110787

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111035
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1d4a3e33c169ea3072f205cb81386a6624cf37fe
Submitter: Jenkins
Branch: stable/icehouse

commit 1d4a3e33c169ea3072f205cb81386a6624cf37fe
Author: Maru Newby <email address hidden>
Date: Thu Jul 31 18:44:57 2014 +0000

    Add dsvm-functional tox env to fix functional job

    Fixes for the bug in question landed in master to get the functional
    job working again, and the first step in backporting those fixes is
    ensuring that the {gate|check}-neutron-dsvm-functional jobs can invoke
    the dsvm-functional tox env. There is no good backport for this
    addition because the patches in master are highly coupled so as to
    make cherry-picking basically impossible without this patch.

    Change-Id: Ide15f6925d516c18cae1afa98f5661f0609bb50e
    Partial-bug: #1336172

[Armando Migliaccio (armando-migliaccio)]

I observed a failure of this job here:

http://logs.openstack.org/13/111813/2/gate/gate-neutron-dsvm-functional/0dcf762/console.html

This doesn't seem to be consistent though.

[Armando Migliaccio (armando-migliaccio)]

Not sure if this is the same failure mode though, Test:

neutron.tests.functional.agent.linux.test_ovsdb_monitor.TestOvsdbMonitor.test_killed_monitor_respawns

seems to have failed for a different reason.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110787
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b1e5eec12afcfd5400c8a82698cf453c205e0cd2
Submitter: Jenkins
Branch: stable/icehouse

commit b1e5eec12afcfd5400c8a82698cf453c205e0cd2
Author: Henry Gessau <email address hidden>
Date: Tue Jul 1 17:56:49 2014 -0400

    Ignore variable column widths in ovsdb functional tests

    The ovsdb tests check the output of "ovsdb-client monitor" before and after
    some operations. The first line of output is a header row with spaces for
    column separation. The column widths can vary depending on the data in the
    columns, so compress multiple spaces to one for testing.

    Change-Id: I6b4690ea9ef1c1b215411ca2c39b1e65fec2c6c7
    Partial-bug: #1336172
    (cherry picked from commit 17686c1f24ddf1977bbd930ce94b3e3d76962c5b)