neutron-agents container not control previous namespaces after restart

Bug #1511216 reported by Vu Nguyen Duy (CBR09)
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Sam Yaple
Liberty
Fix Released
Critical
Sam Yaple

Bug Description

After neutron_agents restart, all previous namespaces (dhcp namespace, router namespace) not access, exec command inside neutron_agents container.
/# ip netns ls
qdhcp-39fb8fd7-b000-4a99-9a94-19b7be8bb46f
qdhcp-105e3939-bf2e-4baf-897c-06a5a80bd60f
qdhcp-eca827e4-ec4b-4e26-92cf-911b5be826fa
qrouter-45a706b8-ef65-4f08-a3b0-2661e7bca0de
qrouter-f1873ff1-5843-4b47-ab75-505a21701226
qdhcp-c1232fda-0a91-45c7-8b4f-af2d8f592dd0
qdhcp-27547045-11fa-4d21-a218-24780a39c313

/# ip netns exec qdhcp-39fb8fd7-b000-4a99-9a94-19b7be8bb46f ip a
seting the network namespace "qdhcp-39fb8fd7-b000-4a99-9a94-19b7be8bb46f" failed: Invalid argument

And I've posted issue in docker-dev:
https://groups.google.com/forum/?fromgroups#!topic/docker-dev/akMNjpf51Vw

Maybe docker not support this, and currently I don't see any solution for this.

Changed in kolla:
milestone: none → mitaka-1
description: updated
summary: - neutron-agents not maintain namespace state after restart
+ neutron-agents container not control previous namespaces after restart
Revision history for this message
Sam Yaple (s8m) wrote :

This is a regression in the Kolla code.

Its part of a more complicated issue that we will be able to solve with Docker 1.10

In the short term I will fix the regression.

Changed in kolla:
assignee: nobody → Sam Yaple (s8m)
Revision history for this message
Vu Nguyen Duy (CBR09) (nguyenduyvu099) wrote :

Is there any workaround for this issue before you fix the regression?. I need it as soon as possible.
Thanks

Revision history for this message
Hui Kang (huikang27) wrote :

I do not think container will support this. Every network namespace needs to be bounded to /var/run/netns. Since a container will have a different mount namespace than the host, so the host will be not able to see the network namespace created inside the container. You can verify this by checking the permission of the network namespace under /var/run/netns/<network namespace name> or /run/netns/<network namespace name>

you can also compare the output of the output of "mount" between container and host: network namespace is not visible to host if it is created in the container. - Hui

Revision history for this message
Sam Yaple (s8m) wrote :

we need to be cleaning up the namespaces before starting the container. this is a regressions.

I will submit a patch right now.

Hui: With docker 1.10 this will work. its magic!

Revision history for this message
Hui Kang (huikang27) wrote :

Sam, this sounds a very nice feature. Do you mind pointing out the changelog of docker 1.10? Thanks. - Hui

Changed in kolla:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/244768
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=baf9d93358f8fc79df9bb71f24dbc50b85860ea1
Submitter: Jenkins
Branch: master

commit baf9d93358f8fc79df9bb71f24dbc50b85860ea1
Author: Sam Yaple <email address hidden>
Date: Thu Nov 12 17:05:31 2015 +0000

    Fix namespace regression for neutron

    Due the to way network namespaces work, they will not survive a
    container restart. This is not an issue related to Kolla or Neutron
    but rather an issue with how network namespaces work.

    This is putting back into place a workaround that was in correctly
    removed in the past. We simply cleanup the namespace cruft when the
    neutron-agents container restarts thus avoiding this issue.

    Change-Id: Ic16f2f766b4fa46347ea14a1ece5a4653c69d779
    Closes-Bug: #1511216

Changed in kolla:
status: In Progress → Fix Committed
Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/248261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/liberty)

Reviewed: https://review.openstack.org/248261
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=8f9c42758a3df46b788cbfcfa8cc40faeb7d43e3
Submitter: Jenkins
Branch: stable/liberty

commit 8f9c42758a3df46b788cbfcfa8cc40faeb7d43e3
Author: Sam Yaple <email address hidden>
Date: Thu Nov 12 17:05:31 2015 +0000

    Fix namespace regression for neutron

    Due the to way network namespaces work, they will not survive a
    container restart. This is not an issue related to Kolla or Neutron
    but rather an issue with how network namespaces work.

    This is putting back into place a workaround that was in correctly
    removed in the past. We simply cleanup the namespace cruft when the
    neutron-agents container restarts thus avoiding this issue.

    Change-Id: Ic16f2f766b4fa46347ea14a1ece5a4653c69d779
    Closes-Bug: #1511216
    (cherry picked from commit baf9d93358f8fc79df9bb71f24dbc50b85860ea1)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.