fullstack tests leave neutron-server running

Bug #1494363 reported by Ihar Hrachyshka
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
IWAMOTO Toshihiro

Bug Description

For each fullstack test I execute, I get a new neutron-server instance hanging there:

 6717 pts/1 S 0:00 /opt/stack/neutron/.tox/dsvm-fullstack/bin/python /opt/stack/neutron/.tox/dsvm-fullstack/bin/neutron-server --log-dir /tmp/dsvm-fullstack-logs/TestConnectivitySameNetwork.test_connectivity --log-file neutron-server--2015-09-10--17-01-31-376365.log --config-file /tmp/tmppafpKE/tmpgiLm_2/neutron.conf --config-file /tmp/tmppafpKE/tmpgiLm_2/ml2_conf.ini
 6718 pts/1 S 0:01 /opt/stack/neutron/.tox/dsvm-fullstack/bin/python /opt/stack/neutron/.tox/dsvm-fullstack/bin/neutron-server --log-dir /tmp/dsvm-fullstack-logs/TestConnectivitySameNetwork.test_connectivity --log-file neutron-server--2015-09-10--17-01-31-376365.log --config-file /tmp/tmppafpKE/tmpgiLm_2/neutron.conf --config-file /tmp/tmppafpKE/tmpgiLm_2/ml2_conf.ini
 6719 pts/1 S 0:00 /opt/stack/neutron/.tox/dsvm-fullstack/bin/python /opt/stack/neutron/.tox/dsvm-fullstack/bin/neutron-server --log-dir /tmp/dsvm-fullstack-logs/TestConnectivitySameNetwork.test_connectivity --log-file neutron-server--2015-09-10--17-01-31-376365.log --config-file /tmp/tmppafpKE/tmpgiLm_2/neutron.conf --config-file /tmp/tmppafpKE/tmpgiLm_2/ml2_conf.ini

I believe ProcessFixture does not behave with neutron-server, probably because of workers.

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Yes, ps -ef shows that all processes are owned by init process, meaning their parent died.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/222556

Changed in neutron:
assignee: nobody → IWAMOTO Toshihiro (iwamoto)
status: New → In Progress
Revision history for this message
Jakub Libosvar (libosvar) wrote :
Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote :

I've commented on the bug that Jakub mentioned above.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/222556
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=470a7d8a106a274e06fb1311c6738f333a98f59c
Submitter: Jenkins
Branch: master

commit 470a7d8a106a274e06fb1311c6738f333a98f59c
Author: IWAMOTO Toshihiro <email address hidden>
Date: Fri Sep 11 19:01:20 2015 +0900

    AsyncProcess: try to kill tender

    _kill_process kills processes with SIGKILL, which prevents the
    processes' cleanup from running. Issue SIGTERM first and wait a bit.

    Change-Id: Ie7b94011bbd11b1d672c95e3be19bb3c84ef77ec
    Closes-bug: 1494363

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

The patch was reverted.

Changed in neutron:
status: Fix Committed → Confirmed
Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote :

The reverted patch should work fine if the remaining use of _kill_process in the test code is replaced with utils.execute(['kill',...]).

Changed in neutron:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/231351

Changed in neutron:
status: Confirmed → In Progress
Assaf Muller (amuller)
tags: added: fullstack
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/231351
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: In Progress → Incomplete
assignee: IWAMOTO Toshihiro (iwamoto) → nobody
Assaf Muller (amuller)
Changed in neutron:
status: Incomplete → Confirmed
Changed in neutron:
assignee: nobody → IWAMOTO Toshihiro (iwamoto)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Doug Wiegley (<email address hidden>) on branch: master
Review: https://review.openstack.org/231351
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

Shall we get on with this once and for all? The fix has already been abandoned and reinstated twice :)

Changed in neutron:
status: In Progress → Incomplete
assignee: IWAMOTO Toshihiro (iwamoto) → nobody
status: Incomplete → In Progress
assignee: nobody → IWAMOTO Toshihiro (iwamoto)
Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote :

It took some time but it'll be all set as soon as we get ryu>=3.29.1 in global-requirements.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/278501

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/278501
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5a7b892df46f5d48840f600f03e7be2db999a1d
Submitter: Jenkins
Branch: master

commit a5a7b892df46f5d48840f600f03e7be2db999a1d
Author: Jakub Libosvar <email address hidden>
Date: Wed Feb 10 16:35:30 2016 +0000

    fullstack: Gracefully stop neutron-server process

    There is possible scenario that neutron-server loses connection to AMQP
    bus and that can cause timeouts when hanging on rpc calls. On the other
    hand, we should be able to stop service gracefully even if AMQP server
    isn't running.

    During teardown there were still neutron-server orphaned api workers
    processes running that had open connection to database which caused
    unexpected failures.

    Note that this change applies only for neutron-server because previous
    investigations showed agents using oslo service can hang on rpc causing
    other failures. Next step should be making sure rabbitmq is stopped as
    last or decrease rpc timeouts in agents.

    Change-Id: I3d793ba924dc3fd229d8588f4be1e943614f22bb
    Related-bug: 1494363
    Closes-bug: 1541742

Revision history for this message
Assaf Muller (amuller) wrote :

@Jakub: With https://review.openstack.org/278501 merged, are there any more actions to be taken to close this bug?

Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote :

I think the above commit should have had "closes-bug: 1494363" instead.

Fullstack tests no longer leave neutron-server running, so this bug can be closed, IMO.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by IWAMOTO Toshihiro (<email address hidden>) on branch: master
Review: https://review.openstack.org/231351
Reason: The mentioned bug has been fixed in a different way.

Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.