cannot setup network on nova Docker virt driver with neutron

[Tomokazu Hirai]

I setup OpenStack havana with Docker.

my environments are ...

* all in one
* using ubuntu cloud package ( version : 1:2013.2-0ubuntu1~cloud0 )
* using docker 0.6.1

I solved that bug with this patch ( https://github.com/openstack/nova/commit/c0e546ace478277cc2911b1c5bef1e082b76b546).

  bug : https://bugs.launchpad.net/nova/+bug/1244829

but now I found new bug. error messages are ...

2013-11-07 12:07:37.256 14098 ERROR nova.compute.manager [req-b3c9ea61-b343-4f0f-847a-e4d6cc61b6d2 a9c9757ed0384309acfc1a01f95798e1 f0b6f917678b4e8c89667366568f7b6d] [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Error: Cannot setup network: Unexpected error while runni
ng command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ln -sf /proc/17653/ns/net /var/run/netns/469e8ed190faef784a364bba593cfb2cec6e43e7cdf813a578a05c92c7ed6cc2
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/nova-rootwrap: Unauthorized command: ln -sf /proc/17653/ns/net /var/run/netns/469e8ed190faef784a364bba593cfb2cec6e43e7cdf813a578a05c92c7ed6cc2 (no filter matched)\n'
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Traceback (most recent call last):
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1037, in _build_instance
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] set_access_ip=set_access_ip)
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1410, in _spawn
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] LOG.exception(_('Instance failed to spawn'), instance=instance)
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1407, in _spawn
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] block_device_info)
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] File "/usr/lib/python2.7/dist-packages/nova/virt/docker/driver.py", line 314, in spawn
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] instance_id=instance['name'])
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] InstanceDeployFailure: Cannot setup network: Unexpected error while running command.
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ln -sf /proc/17653/ns/net /var/run/netns/469e8ed190faef784a364bba593cfb2cec6e43e7cdf813a578a05c92c7ed6cc2
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Exit code: 99
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Stdout: ''
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3] Stderr: '/usr/bin/nova-rootwrap: Unauthorized command: ln -sf /proc/17653/ns/net /var/run/netns/469e8ed190faef784a364bba593cfb2cec6e43e7cdf813a578a05c92c7ed6cc2 (no filter matched)\n'
2013-11-07 12:07:37.256 14098 TRACE nova.compute.manager [instance: e9b78f8d-3046-4a41-8b87-c28348ae87e3]

at result , I could not boot instance.

% nova list
+--------------------------------------+-----------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------+
| e9b78f8d-3046-4a41-8b87-c28348ae87e3 | testcon01 | ERROR | None | NOSTATE | |
+--------------------------------------+-----------+--------+------------+-------------+----------+

Any Idea do you have ?

Thanks.

-- Tomokazu HIRAI @jedipunkz

[melanie witt (melwitt)]

This error is rootwrap saying the command docker is using to set up the network is unauthorized.

https://wiki.openstack.org/wiki/Rootwrap

Can you check in your deployment if you have this file in place, that authorizes the command?

https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/docker.filters

[Tomokazu Hirai (tomokazu-hirai)]

Thanks for your reply, Melanie.

now I deployed docker.filter to /etc/nova/rootwrap.d/ . and I restart nova-compute

again, I booted docker instance. I found these messages.

2013-11-07 12:43:53.348 25669 ERROR nova.compute.manager [req-7e883911-7923-49df-88da-9e67e2b244d2 a9c9757ed0384309acfc1a01f95798e1 f0b6f917678b4e8c89667366568f7b6d] [instance: 73958771-5b10-422a-b4f9-781d2301b471] Error: Cannot setup network: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif br-int pvnetl61464
Exit code: 1
Stdout: ''
Stderr: "can't add pvnetl61464 to bridge br-int: Operation not supported\n"
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] Traceback (most recent call last):
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1037, in _build_instance
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] set_access_ip=set_access_ip)
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1410, in _spawn
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] LOG.exception(_('Instance failed to spawn'), instance=instance)
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1407, in _spawn
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] block_device_info)
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] File "/usr/lib/python2.7/dist-packages/nova/virt/docker/driver.py", line 314, in spawn
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] instance_id=instance['name'])
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] InstanceDeployFailure: Cannot setup network: Unexpected error while running command.
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addif br-int pvnetl61464
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] Exit code: 1
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] Stdout: ''
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471] Stderr: "can't add pvnetl61464 to bridge br-int: Operation not supported\n"
2013-11-07 12:43:53.348 25669 TRACE nova.compute.manager [instance: 73958771-5b10-422a-b4f9-781d2301b471]

These error messages are like that bug.

https://bugs.launchpad.net/nova/+bug/1247295

duplicated ?

Thanks !