Devstack fails to load NovaDocker driver during stack.sh run on Ubuntu 14.04.2

[Boris Derzhavets]

The most recent versions ( approximately since 02/22/2015) of .../stack/nova/openstack-common.conf doesn't contain entry
"module=log"

What causes stack.sh to crash with message

2015-02-24 10:07:05.567 | + service=n-cpu
2015-02-24 10:07:05.568 | + echo 'Error: Service n-cpu is not running'
2015-02-24 10:07:05.568 | Error: Service n-cpu is not running
2015-02-24 10:07:05.569 | + '[' -n /home/ubuntu/stack/status/stack/n-cpu.failure ']'

n-cpu screen log reports :-

ubuntu@ubuntu-Standard:~/devstack$ cd /home/ubuntu/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf & echo $! >/home/ubuntu/stack/status/stack/n-cpu.pid; fg || echo "n-cpu failed to start" | tee "/home/ubuntu/stack/status/stack/n-cpu.failure" 2> error
[1] 14444

cd /home/ubuntu/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf
2015-02-24 06:04:01.623 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db from (pid=14445) __new__ /home/ubuntu/stack/nova/nova/servicegroup/api.py:62
2015-02-24 06:04:01.731 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-02-24 06:04:01.766 INFO nova.virt.driver [-] Loading compute driver 'novadocker.virt.docker.DockerDriver'
2015-02-24 06:04:01.780 ERROR nova.virt.driver [-] Unable to load the virtualization driver
2015-02-24 06:04:01.780 TRACE nova.virt.driver Traceback (most recent call last):
2015-02-24 06:04:01.780 TRACE nova.virt.driver File "/home/ubuntu/stack/nova/nova/virt/driver.py", line 1391, in load_compute_driver
2015-02-24 06:04:01.780 TRACE nova.virt.driver virtapi)
2015-02-24 06:04:01.780 TRACE nova.virt.driver File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 52, in import_object_ns
2015-02-24 06:04:01.780 TRACE nova.virt.driver return import_class(import_str)(*args, **kwargs)
2015-02-24 06:04:01.780 TRACE nova.virt.driver File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
2015-02-24 06:04:01.780 TRACE nova.virt.driver __import__(mod_str)
2015-02-24 06:04:01.780 TRACE nova.virt.driver File "/usr/local/lib/python2.7/dist-packages/novadocker/virt/docker/__init__.py", line 20, in <module>
2015-02-24 06:04:01.780 TRACE nova.virt.driver from novadocker.virt.docker import driver
2015-02-24 06:04:01.780 TRACE nova.virt.driver File "/usr/local/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 41, in <module>
2015-02-24 06:04:01.780 TRACE nova.virt.driver from nova.openstack.common import log
2015-02-24 06:04:01.780 TRACE nova.virt.driver ImportError: cannot import name log
2015-02-24 06:04:01.780 TRACE nova.virt.driver

n-cpu failed to start

NovaDocker Driver previously installed as follows :-

First installed docker 1.5

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

Second NovaDocker driver itself

$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ sudo pip install .

****************
My local.conf
****************
[[local|localrc]]
HOST_IP=192.168.1.57
ADMIN_PASSWORD=secret
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
SERVICE_PASSWORD=secret
FLOATING_RANGE=192.168.10.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.10.150,end=192.168.10.254
PUBLIC_NETWORK_GATEWAY=192.168.10.15
SERVICE_TOKEN=super-secret-admin-token
VIRT_DRIVER=novadocker.virt.docker.DockerDriver

DEST=$HOME/stack
SERVICE_DIR=$DEST/status
DATA_DIR=$DEST/data
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs

FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker

[[post-config|$NOVA_CONF]]
[DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver
.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1426293/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Boris Derzhavets (bderzhavets)]

Due to commit "Sync oslo imports to be in line with Nova trunk" 333656f4ad0e2c06f11eb8244bd6796110b9931d. (https://bugs.launchpad.net/nova-docker/+bug/1425111)

In meantime Nova launch instance and starts container. Attempting reproduce

http://blog.oddbit.com/2015/02/11/installing-novadocker-with-devstack/

Perform statements :-
$ nova floating-ip-create
+------------+-----------+----------+--------+
| Ip | Server Id | Fixed Ip | Pool |
+------------+-----------+----------+--------+
| 172.24.4.3 | - | - | public |
+------------+-----------+----------+--------+
$ nova floating-ip-associate test0 172.24.4.3
$ curl http://172.24.4.3
curl: (7) Failed to connect to 172.24.4.3 port 80: No route to host

Then scan via hping3 port 80 at 172.24.4.3

ubuntu@ubuntu-VM:~$ sudo hping3 -c 1 -V -p 80 -s 5050 -M 0 -UPF 172.24.4.3
using br-ex, addr: 172.24.4.1, MTU: 1500
HPING 172.24.4.3 (br-ex 172.24.4.3): FPU set, 40 headers + 0 data bytes
len=40 ip=172.24.4.3 ttl=64 DF id=62681 tos=0 iplen=40
sport=80 flags=RA seq=0 win=0 rtt=3.9 ms
seq=0 ack=1 sum=913b urp=0

--- 172.24.4.3 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.9/3.9/3.9 ms

ubuntu@ubuntu-VM:~$ sudo hping3 -c 1 -V -p 80 -s 5050 -M 0 -Y 172.24.4.3
using br-ex, addr: 172.24.4.1, MTU: 1500
HPING 172.24.4.3 (br-ex 172.24.4.3): Y set, 40 headers + 0 data bytes
len=40 ip=172.24.4.3 ttl=64 DF id=64274 tos=0 iplen=40
sport=80 flags=RA seq=0 win=0 rtt=3.4 ms
seq=0 ack=0 sum=923b urp=0

--- 172.24.4.3 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.4/3.4/3.4 ms

hping3 return packets , it means port 80 is closed.

UbuntuDocker is container launched via docker image Ubuntu Rastasheep (sshd), which worked fine on 02/15/2015 and still working fine on host running stack.sh instance, recoverable between reboots.
In meantime I get :-

ubuntu@ubuntu-VM:~/devstack$ nova list
+--------------------------------------+--------------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+--------------------------------+
| f7d3731c-d2bf-44bf-8517-8d351f6af502 | UbuntuDocker | ACTIVE | - | Running | private=10.254.1.4, 172.24.4.4 |
| 8cb55a85-ce20-4f61-bf34-f4075e6480a1 | test0 | ACTIVE | - | Running | private=10.254.1.2, 172.24.4.3 |
+--------------------------------------+--------------+--------+------------+-------------+--------------------------------+
ubuntu@ubuntu-VM:~/devstack$ sudo hping3 -c 3 -p 22 -s 5050 -FPU 172.24.4.4
HPING 172.24.4.4 (br-ex 172.24.4.4): FPU set, 40 headers + 0 data bytes
len=40 ip=172.24.4.4 ttl=64 DF id=41373 sport=22 flags=RA seq=0 win=0 rtt=3.8 ms
len=40 ip=172.24.4.4 ttl=64 DF id=41432 sport=22 flags=RA seq=1 win=0 rtt=3.3 ms
len=40 ip=172.24.4.4 ttl=64 DF id=41498 sport=22 flags=RA seq=2 win=0 rtt=3.3 ms

--- 172.24.4.4 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.3/3.5/3.8 ms

ubuntu@ubuntu-VM:~/devstack$ nova secgroup-list-rules default
+-...

[Boris Derzhavets (bderzhavets)]

In meantime problem may be fixed as follows :-

Install working NovaDocker driver (with network alive, e.g. floating IPs) on Ubuntu 14.04.2
via ./stack.sh

$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ git revert 9d06520645f28d96ef905a709f8ff0c27842b58b
$ sudo pip install .

[Boris Derzhavets (bderzhavets)]

I would guess that in commit 9d06520645f28d96ef905a709f8ff0c27842b58b :-

- utils.execute('ip', 'netns', 'exec', container_id, 'ifconfig',
- if_remote_name, ip, run_as_root=True)
+ utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'addr',
+ 'add', ip, 'dev', if_remote_name, run_as_root=True)

The last statement should be followed by a kind of :-

+ utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'link',
  'set', if_remote_name,'up',run_as_root=True)

[Boris Derzhavets (bderzhavets)]

Just finished testing my patch , it works fine. Am I supposed to submit patch to upstream ?

[Boris Derzhavets (bderzhavets)]

--- a/nova-docker/novadocker/virt/docker/vifs.py 2015-03-07 17:32:24.118070981 +0300
+++ b/nova-docker/novadocker/virt/docker/vifs.py 2015-03-07 17:35:36.698098732 +0300
@@ -248,6 +248,8 @@
                           run_as_root=True)
             utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'addr',
                           'add', ip, 'dev', if_remote_name, run_as_root=True)
+ utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'link',
+ 'set', if_remote_name,'up',run_as_root=True)
             if gateway is not None:
                 utils.execute('ip', 'netns', 'exec', container_id,
                               'ip', 'route', 'replace', 'default', 'via',

[Boris Derzhavets (bderzhavets)]

This patch allows to build Nova Docker driver with network enabled ( floating IPs are working)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "dfw.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Boris Derzhavets (bderzhavets)]

Patch in git format

[Boris Derzhavets (bderzhavets)]

Patch merged upstream https://review.openstack.org/#/c/163022/