On doing nova-compute restart on a docker node, interfaces are removed from agent

[Vedamurthy Joshi]

R2.22 Build 116 Ubuntu 14.04 Juno multinode

On a docker hypervisor, there were about 30+ docker instances
On nova-compute restart, we see multiple httpconnection pool errors are the tap interfaces are removed from agent

"2015-12-28 12:13:50.299 20819 WARNING urllib3.connectionpool [-] HttpConnectionPool is full, discarding connection: localhost"

[Ignatious Johnson Christopher (ijohnson-x)]

Hi Vedu,

I tried launching a docker container in nodei27, it got the IP.

Regarding nova-compute restart,
Seems to be a day one issue with the opencontrail docker driver, which during restart of nova-compute doesn’t publish the port information to vrouter-agent.
I have fixed it in the nodei27 (/usr/lib/python2.7/dist-packages/novadocker/virt/docker/opencontrail.py), I have also backed up the old driver(.old).

With this fix I was able to successfully publish(Addport) of all the existing containers to the vrouter-agent.
Request you to share the bug id, I will use it for commiting the fix.

Hi Hari,
Can you point me to the http based api for adding port, so that I will make use of it along with this fix.

Thanks,
Ignatious

[Ignatious Johnson Christopher (ijohnson-x)]

Hi Ignatious,

We need to use vrouter-port-control script to send add/delete port notifications to agent. This script sends http messages for add/delete to agent. It also ensures that port info is persisted in file (Agent reads those files and re-adds port during restarts).

You can look at /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py to see how vrouter-port-control script is being used.

The vrouter-port-control script is present in controller repo under src/vnsw/agent/port_ipc/

Regards,
Ashok

[Ignatious Johnson Christopher (ijohnson-x)]

Fixed in openstack Juno with the commit https://github.com/Juniper/nova-docker/commit/7d8eafce9738df830500e1f550857da6d83a0681

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16322
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/268284

[Ignatious Johnson Christopher (ijohnson-x)]

Fixed the Makefile/changelog to build new nova-docker PPA package in the following commit,
https://github.com/Juniper/contrail-third-party-packages/commit/70f8fb0ffaff40ac8e27f303052495b1314e3119

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16322
Committed: http://github.org/Juniper/contrail-packaging/commit/26b022376ecb9fa40b93af37bcd5a36060963d12
Submitter: Zuul
Branch: master

commit 26b022376ecb9fa40b93af37bcd5a36060963d12
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Jan 15 10:04:19 2016 -0800

Packaging nova-docker package with changes to use http based
vrouter-port-control script to plug/unplug vif'sin the vrouter.
This script ensures that port info is persisted in file.
Agent reads those files and re-adds port during restarts.

Change-Id: Ie650d0b7a5fb6591fd89c0f7c18241f223f09dbc
Closes-Bug: 1533037

[Ignatious Johnson Christopher (ijohnson-x)]

Fixed in stable/icehouse branch of nova-docker Juniper fork with the commit
https://github.com/Juniper/nova-docker/commit/10bf49721937b1bd21f58d74a6ee61f9fd67109b.

Need to build PPA, upload to the build cache before raising review request in contrail-packaging to package the new PPA

[Ignatious Johnson Christopher (ijohnson-x)]

Built PPA packages with these changes https://github.com/Juniper/contrail-third-party-packages/commit/7015defe260313a83d0608a3c96f60b7072be11c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20216
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20217
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20218
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20216
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20216
Committed: http://github.org/Juniper/contrail-packaging/commit/7c0b75ddaa7efc6a48b416ea3364d721dfcf7d01
Submitter: Zuul
Branch: R2.22.x

commit 7c0b75ddaa7efc6a48b416ea3364d721dfcf7d01
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 13 15:45:52 2016 -0700

Packaging nova-docker package with changes to use http based
vrouter-port-control script to plug/unplug vif's in the vrouter.
This script ensures that port info is persisted in file.
Agent reads those files and re-adds port during restarts.

Upgrading lxc-docker package to latest.

Change-Id: I72c77a267662d7d100c09e968d79a8c069c5962f
Closes-Bug: 1533037
Closes-Bug: 1514780

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20217
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20218
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20217
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20218
Committed: http://github.org/Juniper/contrail-packaging/commit/78643e0aaa504284d15e0b8352bbc5723de970d4
Submitter: Zuul
Branch: R2.20

commit 78643e0aaa504284d15e0b8352bbc5723de970d4
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 13 15:45:52 2016 -0700

Packaging nova-docker package with changes to use http based
vrouter-port-control script to plug/unplug vif's in the vrouter.
This script ensures that port info is persisted in file.
Agent reads those files and re-adds port during restarts.

Change-Id: I72c77a267662d7d100c09e968d79a8c069c5962f
Closes-Bug: 1533037

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20217
Committed: http://github.org/Juniper/contrail-packaging/commit/fea940090c8e9fc32f6f13ee1e4cadd1c7ce281a
Submitter: Zuul
Branch: R2.21.x

commit fea940090c8e9fc32f6f13ee1e4cadd1c7ce281a
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 13 15:45:52 2016 -0700

xPackaging nova-docker package with changes to use http based
vrouter-port-control script to plug/unplug vif's in the vrouter.
This script ensures that port info is persisted in file.
Agent reads those files and re-adds port during restarts.

Change-Id: I72c77a267662d7d100c09e968d79a8c069c5962f
Closes-Bug: 1533037

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/268284
Reason: Retiring nova-docker