Docker: Nova boot errors out in setting network

[Vinod Nair]

After launching a Few VM's, not able to launch any more VM's.

root@cmbu-gravity-06:~# nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 7b3c24de-5d99-43f1-8a61-8e9f5b76c82d | VIN1 | ERROR | - | NOSTATE | |
| b89b4496-00c3-4c3b-a071-9beb714974b8 | VIN2 | ERROR | - | NOSTATE | |
+--------------------------------------+------+--------+------------+-------------+----------+

b4496-00c3-4c3b-a071-9beb714974b8] Claim successful
2015-04-21 17:30:15.364 3131 ERROR nova.compute.manager [req-df881b8b-0bd2-418f-baae-0798c47cf080 4ecd16d672174a0bb443f983a881efa7 c8d565a446f348aba91879902a08ca96] [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Instance failed to spawn
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Traceback (most recent call last):
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1740, in _spawn
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] block_device_info)
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 303, in spawn
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] self._start_container(instance, network_info)
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] File "/usr/lib/python2.7/dist-packages/novadocker/virt/docker/driver.py", line 274, in _start_container
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] instance_id=instance['name'])
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] InstanceDeployFailure: Cannot setup network: Unexpected error while running command.
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip netns exec a9179a67f774872bd61e4d9f78eec8f3fd08279ae9820c1a38a4b740cc93a21c dhclient ns22c3e21f
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Exit code: 1
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Stdout: u''
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8] Stderr: u''
2015-04-21 17:30:15.364 3131 TRACE nova.compute.manager [instance: b89b4496-00c3-4c3b-a071-9beb714974b8]
@

Version : 2.20-2 Icehouse 14.04

[Ignatious Johnson Christopher (ijohnson-x)]

Analysis:
-------------
During opencontrail attach vifs the containers use dhclient to lease an ip address.
When bulk container create request is sent, the /var/lib/dhcp/dhclient.leases file is overwritten. Due to the corrupted file Successive docker containers are not getting IP lease as the dhclient command is failling .
So the Instance launch is failing.

Fix:
--------
Creating separate lease files for each docker container.

Workaround:
--------------------
Remove/correct the corrupted lease entry in the /var/lib/dhcp/dhclient.leases file and then launch Instances.

[Ignatious Johnson Christopher (ijohnson-x)]

Review in progress for https://github.com/Juniper/nova-docker/pull/6
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/177972

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ignatious Johnson Christopher (<email address hidden>) on branch: master
Review: https://review.openstack.org/177972

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9732
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9770
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9732
Committed: http://github.org/Juniper/contrail-packaging/commit/c40ccffd3b7886c6d30830e8ff21d9fd045b83f2
Submitter: Zuul
Branch: master

commit c40ccffd3b7886c6d30830e8ff21d9fd045b83f2
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Apr 29 22:29:36 2015 -0700

Updating nova-docker a dependent package version to the version
with fix for bug 1446929, dowloaded from PPA.

Change-Id: Ied4c7e3d127aadaed3d7400593208c9f1013009e
Closes-Bug: 1446929

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177972
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Submitter: Jenkins
Branch: master

commit 12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Apr 27 14:12:12 2015 -0700

    During opencontrail attach vifs the containers use dhclient to lease an ip address.
    When bulk container create request is sent, the /var/lib/dhcp/dhclient.leases file is overwritten.
    Due to the corrupted file Successive docker containers are not getting IP lease
    as the dhclient command is failling .

    Fixing it by creating separate lease files for each docker container.
    Closes-Bug: #1446929

    Removing veth interfaces if the contrainer launch failed
    Closes-Bug: #1452151

    Using inet6 argument with ip command to configure ipv6 address to container
    Closes-Bug: #1455616

    Configure network using statically instead of using dhclient to avoid
    it writting to the compute hosts resolv.conf
    Closes-Bug: #1455168

    Periodically check the vrouter agent conncetion and publish the port informaton to
    the vrouter agent.
    Closes-Bug: #1454655

    Change-Id: I345dddefa3036b29a2ee987216e1094ba9bd23e8