test_get_root_helper_child_pid_returns_first_child gate failure

Bug #1659965 reported by Kevin Benton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Jakub Libosvar
psmisc
New
Undecided
Unassigned

Bug Description

http://logs.openstack.org/01/410501/5/check/gate-neutron-dsvm-functional-ubuntu-xenial/8a82bb0/testr_results.html.gz

ft1.1: neutron.tests.functional.agent.linux.test_utils.TestGetRootHelperChildPid.test_get_root_helper_child_pid_returns_first_child_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
DEBUG [stevedore.extension] found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
   DEBUG [stevedore.extension] found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
   DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
   DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
}}}

Traceback (most recent call last):
  File "neutron/tests/base.py", line 116, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/agent/linux/test_utils.py", line 81, in test_get_root_helper_child_pid_returns_first_child
    sleep=0.1)
  File "neutron/common/utils.py", line 736, in wait_until_true
    while not predicate():
  File "neutron/tests/functional/agent/linux/test_utils.py", line 67, in wait_for_sleep_is_spawned
    return 'sleep' == processes[-1]
IndexError: list index out of range

tags: added: gate-failure
Revision history for this message
Victor Morales (electrocucaracha) wrote :

Maybe the way that is collected the child processes is not the best one[1]. I'm using pstree (PSmisc) 22.20 and it seems to use a single hyphen for creating the tree.

[1] https://github.com/openstack/neutron/blob/master/neutron/tests/functional/agent/linux/test_utils.py#L65

Revision history for this message
Victor Morales (electrocucaracha) wrote :

Never mind, the format seems fine[1] on different versions, therefore the problem could be caused by the chosen parent_pid

[1] https://sourceforge.net/p/psmisc/code/ci/master/tree/src/pstree.c#l117

Revision history for this message
Jakub Libosvar (libosvar) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/427141

Changed in neutron:
assignee: nobody → Jakub Libosvar (libosvar)
status: New → In Progress
tags: added: functional-tests
Changed in neutron:
importance: Undecided → High
milestone: none → ocata-rc1
tags: added: auto-allocated-topology
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/427141
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e5320e764625bfc5b6723c52ec202ebdb926d32f
Submitter: Jenkins
Branch: master

commit e5320e764625bfc5b6723c52ec202ebdb926d32f
Author: Jakub Libosvar <email address hidden>
Date: Tue Jan 31 07:32:21 2017 -0500

    functional: Check for processes only if there are any

    In one of tests, pstree received SIGSEGV signal and was terminated. This
    means the output is empty and we check on empty string. This patch makes
    the test less prone to the errors coming from executed binaries.

    Change-Id: I22a7f2fea56c9d97a1d765f0f25e9f28c7942b55
    Closes-bug: 1659965

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0rc1

This issue was fixed in the openstack/neutron 10.0.0.0rc1 release candidate.

Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote :

Link on #3 is dead now.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22pstree%5C%22%20AND%20message%3A%5C%22segfault%5C%22

shows several messages such as:

  pstree[8267]: segfault at 0 ip 00007f93355b4264 sp 00007ffceaaf6370 error 4 in libc-2.23.so[7f9335547000+1bf000]

The OS is ubuntu xenial and it might be possible to find the offending libc function from ip.
IMO, pstree segfaulting is no good.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.