[R2.20-Build 17]:Docker container instances informations could not be recovered after agent crash

[alok kumar]

if agent crashed or restarted because of any reason, then once agent is up and running metadata IP of docker containers are not reachable.

For qemu based computes, metadata IP is reachable once agent is UP.

[alok kumar (kalok)]

once agent is up, it doesn't have any instance information.
looks like instances info could not be recovered after agent crash/restart.

after this instances are not able to send traffic to each other, verified it through docker cli.

from analytics:
curl http://nodec64.englab.juniper.net:8081/analytics/uves/virtual-machines
[]

root@nodec62:~# nova list
+--------------------------------------+-------+--------+------------+-------------+-------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+-------------+
| 17dda6d0-89e1-4c72-845c-bdf124564a2d | vm10 | ACTIVE | - | Running | vn4=4.4.4.6 |
| b0716346-ef97-489e-b36e-3c3d9a92c4d6 | vm4-1 | ACTIVE | - | Running | vn4=4.4.4.3 |
| 556f84ce-2240-4fec-b0ec-69d17d5ba137 | vm4-2 | ACTIVE | - | Running | vn4=4.4.4.4 |
| 05c436ed-baca-4c10-95ca-5be0fff8630f | vm9 | ACTIVE | - | Running | vn4=4.4.4.5 |
+--------------------------------------+-------+--------+------------+-------------+-------------+

root@nodec64:/opt/contrail/utils# curl -u admin:contrail123 http://127.0.0.1:8095/virtual-machines | python -mjson.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 778 100 778 0 0 176k 0 --:--:-- --:--:-- --:--:-- 189k
{
    "virtual-machines": [
        {
            "fq_name": [
                "05c436ed-baca-4c10-95ca-5be0fff8630f"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/05c436ed-baca-4c10-95ca-5be0fff8630f",
            "uuid": "05c436ed-baca-4c10-95ca-5be0fff8630f"
        },
        {
            "fq_name": [
                "17dda6d0-89e1-4c72-845c-bdf124564a2d"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/17dda6d0-89e1-4c72-845c-bdf124564a2d",
            "uuid": "17dda6d0-89e1-4c72-845c-bdf124564a2d"
        },
        {
            "fq_name": [
                "556f84ce-2240-4fec-b0ec-69d17d5ba137"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/556f84ce-2240-4fec-b0ec-69d17d5ba137",
            "uuid": "556f84ce-2240-4fec-b0ec-69d17d5ba137"
        },
        {
            "fq_name": [
                "b0716346-ef97-489e-b36e-3c3d9a92c4d6"
            ],
            "href": "http://127.0.0.1:8095/virtual-machine/b0716346-ef97-489e-b36e-3c3d9a92c4d6",
            "uuid": "b0716346-ef97-489e-b36e-3c3d9a92c4d6"
        }
    ]
}

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

[Ignatious Johnson Christopher (ijohnson-x)]

Reviewed: https://review.opencontrail.org/10730
Committed: http://github.org/Juniper/contrail-packaging/commit/70bbb08f57773280a40c6f1565f853918c0bb120
Submitter: Zuul
Branch: master

commit 70bbb08f57773280a40c6f1565f853918c0bb120
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:09:24 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I79360edd27ca59743f5ee6afee928f77c6a31435

[Ignatious Johnson Christopher (ijohnson-x)]

Reviewed: https://review.opencontrail.org/10731
Committed: http://github.org/Juniper/contrail-packaging/commit/22ac9793701d0844dea10bbcffdd3eabe4a2f821
Submitter: Zuul
Branch: R2.20

commit 22ac9793701d0844dea10bbcffdd3eabe4a2f821
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 22 16:33:26 2015 -0700

Removing veth interfaces if the contrainer launch failed
Closes-Bug: 1452151
Closes-Bug: 1454976

Using inet6 argument with ip command to configure ipv6 address to container
Closes-Bug: 1455616

Configure network using statically instead of using dhclient to avoid
it writting to the compute hosts resolv.conf
Closes-Bug: 1455168

Periodically check the vrouter agent conncetion and publish the port informaton to
the vrouter agent.
Closes-Bug: 1454655

Change-Id: I3f59055da0748df2a8dddfd82cf2eb062377dfab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177972
Committed: https://git.openstack.org/cgit/stackforge/nova-docker/commit/?id=12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Submitter: Jenkins
Branch: master

commit 12b11b31b6f4274efa88f3011d2e4b884ed0e30c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Apr 27 14:12:12 2015 -0700

    During opencontrail attach vifs the containers use dhclient to lease an ip address.
    When bulk container create request is sent, the /var/lib/dhcp/dhclient.leases file is overwritten.
    Due to the corrupted file Successive docker containers are not getting IP lease
    as the dhclient command is failling .

    Fixing it by creating separate lease files for each docker container.
    Closes-Bug: #1446929

    Removing veth interfaces if the contrainer launch failed
    Closes-Bug: #1452151

    Using inet6 argument with ip command to configure ipv6 address to container
    Closes-Bug: #1455616

    Configure network using statically instead of using dhclient to avoid
    it writting to the compute hosts resolv.conf
    Closes-Bug: #1455168

    Periodically check the vrouter agent conncetion and publish the port informaton to
    the vrouter agent.
    Closes-Bug: #1454655

    Change-Id: I345dddefa3036b29a2ee987216e1094ba9bd23e8