neutron-ns-metadata-proxy cleanup does not occur

Bug #1648095 reported by Barak
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
networking-odl
Fix Released
High
Isaku Yamahata
neutron
Fix Released
High
Isaku Yamahata

Bug Description

Devstack + newton + networking-odl / v2.
To reproduce:
1. Create a new vxlan network.
2. Add 1 or more subnets
3. Delete the network.
I notice this occur during tempest.api.network.test_networks.NetworksIpV6Test.test_create_list_subnet_with_no_gw64_one_network

Finally linux is Out of memory and terminating processes based on score.

Many like this;
stack 2538 1 0 11:19 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-ns-metadata-proxy --pid_file=/opt/stack/data/neutron/external/pids/b408d77c-ca69-4b2c-b2f1-5a889f149f11.pid --metadata_proxy_socket=/opt/stack/data/neutron/metadata_proxy --network_id=b408d77c-ca69-4b2c-b2f1-5a889f149f11 --state_path=/opt/stack/data/neutron --metadata_port=80 --metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug
stack 2981 1 0 11:20 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-ns-metadata-proxy --pid_file=/opt/stack/data/neutron/external/pids/ddfc2980-2c16-46f4-8eab-e91a047e796c.pid --metadata_proxy_socket=/opt/stack/data/neutron/metadata_proxy --network_id=ddfc2980-2c16-46f4-8eab-e91a047e796c --state_path=/opt/stack/data/neutron --metadata_port=80 --metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug
stack 4977 1 0 11:28 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-ns-metadata-proxy --pid_file=/opt/stack/data/neutron/external/pids/2b6d0fe6-f13a-4be4-9ea6-f6bc12ed3d69.pid --metadata_proxy_socket=/opt/stack/data/neutron/metadata_proxy --network_id=2b6d0fe6-f13a-4be4-9ea6-f6bc12ed3d69 --state_path=/opt/stack/data/neutron --metadata_port=80 --metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug
stack 6199 12085 0 11:34 pts/6 00:00:00 grep neutron-ns-metadata-proxy
stack 29462 1 0 11:11 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-ns-metadata-proxy --pid_file=/opt/stack/data/neutron/external/pids/00192768-29d1-46d2-9092-7a7bbd7b55ef.pid --metadata_proxy_socket=/opt/stack/data/neutron/metadata_proxy --network_id=00192768-29d1-46d2-9092-7a7bbd7b55ef --state_path=/opt/stack/data/neutron --metadata_port=80 --metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug
stack 32627 1 0 11:17 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-ns-metadata-proxy --pid_file=/opt/stack/data/neutron/external/pids/b2aa9b47-b55e-4b3f-bf2c-31bc91d86b33.pid --metadata_proxy_socket=/opt/stack/data/neutron/metadata_proxy --network_id=b2aa9b47-b55e-4b3f-bf2c-31bc91d86b33 --state_path=/opt/stack/data/neutron --metadata_port=80 --metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug

Barak (barakd)
description: updated
Barak (barakd)
summary: - neutron-ns-metadata-proxy cleanup does occur
+ neutron-ns-metadata-proxy cleanup does not occur
Barak (barakd)
description: updated
Revision history for this message
Isaku Yamahata (yamahata) wrote :

Do you know if this occurs recently or this exists before?

Changed in networking-odl:
importance: Undecided → High
assignee: nobody → Isaku Yamahata (yamahata)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/411566

Changed in neutron:
assignee: nobody → Isaku Yamahata (yamahata)
status: New → In Progress
Changed in networking-odl:
status: New → In Progress
tags: added: ocata-rc-potential
Changed in neutron:
importance: Undecided → High
milestone: none → ocata-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/411566
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a502c96b8efa62627e245ffa69a23f4b6d3f90a5
Submitter: Jenkins
Branch: master

commit a502c96b8efa62627e245ffa69a23f4b6d3f90a5
Author: Isaku Yamahata <email address hidden>
Date: Thu Dec 15 17:05:26 2016 -0800

    Kill the metadata proxy process unconditionally

    When force_metadata=True and enable_isolated_metadata=False,
    the namespace metadata proxy process might not be terminated
    when the network is deleted because the subnets and ports
    will have already been deleted, so we could incorrectly
    determine it was started. Calling destroy_monitored_metadata_proxy() is
    a noop when there is no process running.

    Change-Id: I77ff545ce02f2dca4c38e587b37ea809ad6f072c
    Closes-Bug: #1648095

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/428652

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-odl (master)

Fix proposed to branch: master
Review: https://review.openstack.org/428809

tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0rc1

This issue was fixed in the openstack/neutron 10.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-odl (master)

Reviewed: https://review.openstack.org/428809
Committed: https://git.openstack.org/cgit/openstack/networking-odl/commit/?id=9a358fcfec20c5d69fac04f597776b52264c8b29
Submitter: Jenkins
Branch: master

commit 9a358fcfec20c5d69fac04f597776b52264c8b29
Author: Isaku Yamahata <email address hidden>
Date: Fri Feb 3 08:38:06 2017 -0800

    Revert "devstack: disable metadata for rally"

    This reverts commit 9b371fe685032c9ab08fd1e99cf8583a6fbf11b5.

    Change-Id: I56faa7d5cf9f5789da6a548f05bd37b2e8fc4da3
    Closes-bug: #1648095

Changed in networking-odl:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/428652
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e7072b05e39d19034106cef61dce28415b01428b
Submitter: Jenkins
Branch: stable/newton

commit e7072b05e39d19034106cef61dce28415b01428b
Author: Isaku Yamahata <email address hidden>
Date: Thu Dec 15 17:05:26 2016 -0800

    Kill the metadata proxy process unconditionally

    When force_metadata=True and enable_isolated_metadata=False,
    the namespace metadata proxy process might not be terminated
    when the network is deleted because the subnets and ports
    will have already been deleted, so we could incorrectly
    determine it was started. Calling destroy_monitored_metadata_proxy() is
    a noop when there is no process running.

    Change-Id: I77ff545ce02f2dca4c38e587b37ea809ad6f072c
    Closes-Bug: #1648095
    (cherry picked from commit a502c96b8efa62627e245ffa69a23f4b6d3f90a5)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-odl 4.0.0

This issue was fixed in the openstack/networking-odl 4.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/439551

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/mitaka)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/439551

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.3.0

This issue was fixed in the openstack/neutron 9.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.