some platforms do not support namespaces

[dan wendlandt]

this means two things:

1) we need to document what versions of ubuntu/red hat support namespaces
2) ideally, we need a way of using quantum (without overlapping IP support) that does not require namespaces.

[dan wendlandt (danwent)]

we believe ubuntu 12.04 and above supports namespaces.

issue is assigned to garyk to explore fedora/red hat.

[Aaron Rosen (arosen)]

Ubuntu 12.04 - Yes
Ubuntu 11.10 - No
Fedora16 - No

[dan wendlandt (danwent)]

to be clear, aaron's comments about are whether the 'ip' utility suports the 'netns' argument.

[Aaron Rosen (arosen)]

This looks like it could be another possible option to use for namespaces. I believe it just relies on a userspace program vnoded (something we could package with quantum). It looks like it uses veths (and vnoded) in order to emulate namespaces.

http://code.google.com/p/coreemu/wiki/Namespaces
http://code.google.com/p/coreemu/wiki/NamespaceKernels

[Aaron Rosen (arosen)]

I'm trying to get this to work with fedora16 right now and it seems that the kernel .config it has: CONFIG_NET_NS=y so it should have netns support.

I believe the issue might be an outdated route2 package (iproute-2.6.39-6.fc16.x86_64) . I installed the latest iproute2 package and this is what I'm getting now.

2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils] Running command: sudo ip link set tap09158e3a-1c address fa:16:3e:b9:5b:c2
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'link', 'set', 'tap09158e3a-1c', 'address', 'fa:16:3e:b9:5b:c2']
Exit code: 0
Stdout: ''
Stderr: ''
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils] Running command: sudo ip -o netns list
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', '-o', 'netns', 'list']
Exit code: 0
Stdout: ''
Stderr: ''
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns add 6e57a8cd-103c-42a3-be52-6a152d504994
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'add', '6e57a8cd-103c-42a3-be52-6a152d504994']
Exit code: 0
Stdout: ''
Stderr: ''
2012-08-11 15:39:14 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec 6e57a8cd-103c-42a3-be52-6a152d504994 ip link set lo up
2012-08-11 15:39:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'exec', '6e57a8cd-103c-42a3-be52-6a152d504994', 'ip', 'link', 'set', 'lo', 'up']
Exit code: 0
Stdout: ''
Stderr: ''
2012-08-11 15:39:15 DEBUG [quantum.agent.linux.utils] Running command: sudo ip link set tap09158e3a-1c netns 6e57a8cd-103c-42a3-be52-6a152d504994
2012-08-11 15:39:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'link', 'set', 'tap09158e3a-1c', 'netns', '6e57a8cd-103c-42a3-be52-6a152d504994']
Exit code: 0
Stdout: ''
Stderr: ''
2012-08-11 15:39:15 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec 6e57a8cd-103c-42a3-be52-6a152d504994 ip link set tap09158e3a-1c up
2012-08-11 15:39:15 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'exec', '6e57a8cd-103c-42a3-be52-6a152d504994', 'ip', 'link', 'set', 'tap09158e3a-1c', 'up']
Exit code: 1
Stdout: ''
Stderr: 'mount of /sys failed: Device or resource busy\n'
2012-08-11 15:39:15 WARNING [quantum.agent.dhcp_agent] Unable to enable dhcp. Exception:
Command: ['sudo', 'ip', 'netns', 'exec', '6e57a8cd-103c-42a3-be52-6a152d504994', 'ip', 'link', 'set', 'tap09158e3a-1c', 'up']
Exit code: 1
Stdout: ''
Stderr: 'mount of /sys failed: Device or resource busy\n'
2012-08-11 15:39:18 DEBUG [quantum.agent.linux.dhcp] Unable to access /opt/stack/data/dhcp/6e57a8cd-103c-42a3-be52-6a152d504994/pid
2012-08-11 15:39:18 DEBUG [quantum.agent.linux.utils] Running command: sudo cat /proc/None/cmdline
2012-08-11 15:39:18 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'cat', '/proc/None/cmdline']
Exit code: 1
Stdout: ''
Stderr: 'cat: /proc/None/cmdline: No such file or directory\n'
2012-08-11 15:39:18 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec 6e57a8cd-103c-42a3-be52-6a152d504994 ip -o link show tap09158e3a-1c
2012-08-11 15:39:18 ...

[Aaron Rosen (arosen)]

Installing the latest iproute2 allows overlapping ip in 11.10.

Aaron

[Gary Kotton (garyk)]

Is there a place to list limitations? Should this just be listed in the "Admin" guide?
I think that Aarons fix for disabling the namespaces is a good and decent work around if the linux version does not support namespaces.
Thanks
Gary

[Yang Zhenyu (yangzhenyu)]

Does the CentOS6.3 support namespace?
My kernel is 2.6.32-279.5.1.el6.x86_64.
And it is not command ip netns.

Thanks.

[Gary Kotton (garyk)]

Can you please check which ip version you are using. ip -V.
This may not be the version that has support for the netns command.
Thanks
Gary

[Yang Zhenyu (yangzhenyu)]

Now it has the command ip netns when I install the latest iproute2.
But the dhcp can not work to assign a ip to the instance.
The quantum-dhcp log :

2012-08-31 02:16:31 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec 3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9 ip link set tap4cffe1f8-07 up
2012-08-31 02:16:32 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'exec', '3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9', 'ip', 'link', 'set', 'tap4cffe1f8-07', 'up']
Exit code: 1
Stdout: ''
Stderr: 'seting the network namespace failed: Function not implemented\n'
2012-08-31 02:16:32 WARNING [quantum.agent.dhcp_agent] Unable to enable dhcp. Exception:
Command: ['sudo', 'ip', 'netns', 'exec', '3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9', 'ip', 'link', 'set', 'tap4cffe1f8-07', 'up']
Exit code: 1
Stdout: ''
Stderr: 'seting the network namespace failed: Function not implemented\n'
2012-08-31 02:17:46 DEBUG [quantum.agent.rpc] Unknown event_type: port.create.start.
2012-08-31 02:17:46 DEBUG [quantum.agent.rpc] Unknown event_type: port.create.start.
2012-08-31 02:17:46 WARNING [quantum.agent.dhcp_agent] Unable to reload_allocations dhcp. Exception: [Errno 2] No such file or directory: '/opt/stack/data/dhcp/3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9/tmpoFHWiu'
2012-08-31 02:17:46 WARNING [quantum.agent.dhcp_agent] Unable to reload_allocations dhcp. Exception: [Errno 2] No such file or directory: '/opt/stack/data/dhcp/3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9/tmpVeqFXG'

Thanks!

[Yang Zhenyu (yangzhenyu)]

And the ovs-vsctl show:
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap09e60884-7e"
            tag: 1
            Interface "tap09e60884-7e"
        Port "tap4cffe1f8-07"
            tag: 1
            Interface "tap4cffe1f8-07"
                type: internal

[dan wendlandt (danwent)]

it looks like your system still is having an error when trying to use namespaces:

Command: ['sudo', 'ip', 'netns', 'exec', '3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9', 'ip', 'link', 'set', 'tap4cffe1f8-07', 'up']
Exit code: 1
Stdout: ''
Stderr: 'seting the network namespace failed: Function not implemented\n'

Are you even able to run a basic command in that namespace, like

sudo ip netns exec 3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9 ip addr list

?

[Yang Zhenyu (yangzhenyu)]

When I exec this command:
# sudo ip netns exec 3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9 ip addr list
seting the network namespace failed: Function not implemented

Is it a problem with iproute2?

# ip -V
ip utility, iproute2-ss120319

I have update the linux kernel from 2.6.32-279.5.1.el6.x86_64, Now is:
# uname -a
Linux openstack-f3 3.2.28 #1 SMP Wed Aug 29 23:56:50 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks

[Aaron Rosen (arosen)]

Doesn't look like it since it didn't complain about the netns param. You're kernel seems like it wasn't built with CONFIG_NET_NS=y. You can confirm by looking at the .config for the kernel you are running.

[dan wendlandt (danwent)]

are you able to run the command I asked about?

sudo ip netns exec 3f68d8dc-b761-4cb9-a3bf-7e8c4a227da9 ip addr list

[Yang Zhenyu (yangzhenyu)]

#vi /usr/src/linux-3.2.28/.config
I can find this:
CONFIG_NET_NS=y

Thanks

[Yang Zhenyu (yangzhenyu)]

Hi dan

I run this command you asked, but it has a error: seting the network namespace failed: Function not implemented

Thanks

[Aaron Rosen (arosen)]

Hi Yang,

I just looked at the ip2route code and it does:

 if (setns(netns, CLONE_NEWNET) < 0) {
      fprintf(stderr, "seting the network namespace failed: %s\n",
         strerror(errno));
      return -1;
   }

static int setns(int fd, int nstype)
{
#ifdef __NR_setns
   return syscall(__NR_setns, fd, nstype);
#else
   errno = ENOSYS;
   return -1;
#endif
}

So it seem on the ip binary when it was built __NR_setns was defined so errno ENOSYS aka Function not implemented. Perhaps rebuilding iproute2 will help you assuming this is defined.

[Yang Zhenyu (yangzhenyu)]

Hi Aaron
I add a define in the /iproute2/ip/ipnetns.c
#define __NR_setns 268

Then rebuilding iproute2, the problem has been solved.
Thank you

But, there is other error of dhcp When I create a network. And now the instance cannot obtain a IP address. The log:

 DEBUG [quantum.agent.linux.dhcp] Unable to access /opt/stack/data/dhcp/218d8883-7c1c-4332-8811-518d130fa419/pid

Stderr: 'cat: /proc/None/cmdline: No such file or directory\n'

Command: ['sudo', 'ip', 'netns', 'exec', '218d8883-7c1c-4332-8811-518d130fa419', 'ip', '-o', 'link', 'show', 'tap2afc920f-0f']
Exit code: 1
Stdout: ''
Stderr: 'seting the network namespace failed: Bad address\n'

Command: ['sudo', 'ip', 'link', 'set', 'tap2afc920f-0f', 'address', 'fa:16:3e:89:b1:61']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "tap2afc920f-0f"\n'

#ovs-vsctl show

4d4b586f-1206-4eea-bd18-57011d490b16
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        Port "tap2afc920f-0f"
            tag: 1
            Interface "tap2afc920f-0f"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}

I just hope the instance obtain a IP address from dhcp. Does it must use the network namespace?
Thanks.

[Yang Zhenyu (yangzhenyu)]

I have solved the problem.
Thanks for dan and Aaron.

[Kalle Happonen (kalle-happonen)]

Just to comment on this issue, since I'm encountering this too. I'm testing devstack Folsom with CentOS 6.3. The RHEL 6 based distros don't seem to have namespace support at all. First I encountered a problem with the ip command
Object "netns" is unknown, try "ip help".

After compiling a new iproute2 package this problem went away, but I noticed that
/proc/self/ns
is missing completely, which I assume is a sign that the kernel has no namespace support.

[Jay Buffington (jaybuff)]

Hi Kalle,

I ran into this issue with 6.3 as well. It seems that you need iproute2 3.1.0, glibc 2.14 and 3.0.0 of the kernel, none of which RHEL 6.3 provides.

This bug in the redhat queue discusses resolving the issue, and it looks like a fix may be in 6.5 when that is released:
https://bugzilla.redhat.com/show_bug.cgi?id=869004

[Qiu Yu (unicell)]

The solution discussed in https://bugzilla.redhat.com/show_bug.cgi?id=869004 only fixes the iproute2 part (adding netns support).

However, in order to make ip netns stuff work, kernel also need to support file descriptor of /proc/[pid]/ns/net . And this feature is merged to Linux upstream since 3.0 Not so sure whether it will be backported to RHEL 6.5 or not.

[Qiu Yu (unicell)]

For those who working with RHEL/CentOS 6.3, following are a kernel version with necessary patches backported. For testing purpose only.

Patches
https://github.com/unicell/redpatch/commits/rhel-2.6.32-358.6.2.ns.el6

Prebuilt binaries
http://trilocell.info/rpms/

[Sushma Korati (sushma-korati)]

Hi Yang,

Even I am facing the same issue.

$ip netns exec test1 ifconfig
output: seting the network namespace failed: Bad address

Can you please tell me how to solve it?

Thanks.

[Tomasz Paszkowski (tpaszkowski)]

For SUSE Linux Enterpise, SLE11SP3 is having support for net ns. SLE11SP2 also have support for netns enabled in kernel, but iproute2 utility is missing netns command.

[Kyle Mestery (mestery)]

Is this still an issue at this point? This bug was filed in 2012, and the last update was last summer. I am advocating closing this bug for now, and if a platform which supports OpenStack Neutron doesn't have namespace support, we can file a separate bug.

[Patrick (5-launchpad-y)]

Still an issue under centos 6.5. Looking at this thread, it's not clear to me if a work-around was ever figured out. Is it possible to just disable openstack's use of namespaces?

==> /var/log/neutron/dhcp-agent.log <==
2014-05-28 16:18:54.351 2601 INFO neutron.agent.dhcp_agent [-] Synchronizing state
2014-05-28 16:18:54.516 2601 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for bd09371a-e8d1-46cf-89bb-3d64984696b5.
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/dhcp_agent.py", line 127, in call_driver
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent getattr(driver, action)(**action_kwargs)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/dhcp.py", line 166, in enable
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent reuse_existing=True)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/dhcp.py", line 835, in setup
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent namespace=network.namespace)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/interface.py", line 195, in plug
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent namespace_obj = ip.ensure_namespace(namespace)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 135, in ensure_namespace
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent if not self.netns.exists(name):
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 469, in exists
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent output = self._parent._execute('o', 'netns', ['list'])
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent root_helper=root_helper)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent raise RuntimeError(m)
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent Command: ['ip', '-o', 'netns', 'list']
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent Exit code: 255
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent Stderr: 'Object "netns" is unknown, try "ip help".\n'
2014-05-28 16:18:54.516 2601 TRACE neutron.agent.dhcp_agent
2014-05-28 16:18:54.517 2601 INFO neutron.agent.dhcp_agent [-] Synchronizing state complete

[Patrick (5-launchpad-y)]

FWIW this is the version of iproute I'm getting from the icehouse repo:

[patrick@openstack1 ~]$ yum info iproute
Installed Packages
Name : iproute
Arch : x86_64
Version : 2.6.32
Release : 130.el6ost.netns.2
Size : 933 k
Repo : installed
From repo : openstack-icehouse
Summary : Advanced IP routing and network device configuration tools
URL : http://linux-net.osdl.org/index.php/Iproute2
License : GPLv2+ and Public Domain
Description : The iproute package contains networking utilities (ip and rtmon, for
            : example) which are designed to use the advanced networking
            : capabilities of the Linux 2.4.x and 2.6.x kernel.

.... and ip:

[patrick@openstack1 ~]$ ip -V
ip utility, iproute2-ss091226

[Tomoe Sugihara (tomoe)]

Hi Patrick,

I have exactly the same version of iproute on centos6.5, and I can do this:

[root@vagrant-centos65 vagrant]# ip -V
ip utility, iproute2-ss091226
[root@vagrant-centos65 vagrant]# ip netns add foo
[root@vagrant-centos65 vagrant]# ip netns list
foo
[root@vagrant-centos65 vagrant]# ip netns del foo
[root@vagrant-centos65 vagrant]# ip netns list
[root@vagrant-centos65 vagrant]# uname -a
Linux vagrant-centos65.vagrantup.com 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Are you sure that the node in which dhcp agent is running has the same version of iproute?
Another possibility would be dhcp agent is executing a different binary (if multiple versions are installed on different paths)?

[Eugene Nikanorov (enikanorov)]

I think this bug loses relevance as time goes on and more versions get namespace support.
I'm marking it as incomplete to let submitter provide the input or close it as invalid.

[Cedric Brandily (cbrandily)]

The bug also seems inrelevant because namespace use can be disabled in dhcp/l3 agents

[Ihar Hrachyshka (ihar-hrachyshka)]

I guess this bug should be solved by adding a new sanity check for namespaces. I guess the importance of the bug can be lowered to Low.