Instance creation fails using Docker /w libcontainer execution backend

[Pavel Sviderski]

When docker version 0.9.0 is used the nova docker driver cannot create an instance.

2014-03-14 14:50:23.293 ERROR nova.compute.manager [req-f553b192-ff5b-4d7e-ac9d-f7cca3804f74 admin demo] [instance: af7ffeee-da4d-4120-a412-61bad3991ced] Error: Cannot setup network: Cannot find any PID under container "612ada12e14f52771d32e3ae4c410ec31838c28aec8b53357531422d7c02b7d8"
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] Traceback (most recent call last):
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1070, in _build_instance
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] set_access_ip=set_access_ip)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] return function(self, context, *args, **kwargs)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1478, in _spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] six.reraise(self.type_, self.value, self.tb)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/compute/manager.py", line 1475, in _spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] block_device_info)
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] File "/opt/stack/nova/nova/virt/docker/driver.py", line 313, in spawn
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] instance_id=instance['name'])
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced] InstanceDeployFailure: Cannot setup network: Cannot find any PID under container "612ada12e14f52771d32e3ae4c410ec31838c28aec8b53357531422d7c02b7d8"
2014-03-14 14:50:23.293 TRACE nova.compute.manager [instance: af7ffeee-da4d-4120-a412-61bad3991ced]

The driver fails in the '_find_container_pid' method. The reason is that in 0.9 there is a new built-in execution driver used by default (libcontainer). Instead of cgroup path /sys/fs/cgroup/devices/lxc/ it uses /sys/fs/cgroup/devices/docker/.

Could it be possible to rewrite _find_container_pid using the docker remote api and do not access cgroup filesystem directly?

[Pavel Sviderski (spyyy)]

possible solution is in the attachment

[Erica Windisch (ewindisch)]

It should be noted that while libcontainer is now default, it remains possible to run docker with '-e lxc'.

[Erica Windisch (ewindisch)]

Pavel, thank you for the patch. We're moving the docker driver into Stackforge and will be able to apply it once we complete the migration.

[Erica Windisch (ewindisch)]

@Pavel - would you mind submitting this using gerrit / git-review?

The repository is https://github.com/stackforge/nova-docker, we use gerrit and git-review like other projects.

Thanks!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84830

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84830
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=ab6d09cacce89ec87049d96ec89e8630dd2e146a
Submitter: Jenkins
Branch: master

commit ab6d09cacce89ec87049d96ec89e8630dd2e146a
Author: Aaron Rosen <email address hidden>
Date: Wed Apr 2 11:16:43 2014 -0700

    Use docker api in order to find container pid

    Previously, the docker driver would try to obtain a containers pid by
    looking it up in /sys/fs/cgroup/drivces/ though this path is not always
    correct depending on the version of docker one is using. This patch
    corrects this by getting the pid by querying docker instead.

    Change-Id: If3cc342dc781a360f2414a1291efda57d102a85b
    Closes-bug: 1292479
    Co-Authored-By: Pavel Sviderski <email address hidden>