Upgrades, sometimes docker allocates the same IPs for different containers

[Tatyanka]

Solution: it should be fixed by upgrading of patched docker

========

steps to reproduce:
1. Deploy 5.0.26 master node
2. Run upgrade
3. after upgrade run command fuel nodes

Actual:
http://paste.openstack.org/show/95533/

for any command in cli

keyston error:
http://paste.openstack.org/show/95524/

[Tatyanka (tatyana-leontovich)]

logs will be attached soon

[Dima Shulyak (dshulyak)]

We finally found a reason why upgrades fails sometime!!!

For some reason docker assigns same ipaddress for couple of containers, in this case this is astute and keystone.

[root@nailgun ~]# dockerctl shell astute
[root@442ed8aa3d43 ~]# ip a
155: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether ea:36:26:15:35:fb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.8/16 scope global eth0

[root@nailgun ~]# dockerctl shell keystone
[root@00577252066e ~]# ip a
149: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 5a:47:c0:d0:bb:8b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.8/16 scope global eth0

But you can see that cached arp address is astute one

[root@nailgun ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
10.108.20.1 ether 52:54:00:16:ed:ef C eth0
172.17.0.7 ether d6:fc:16:57:b4:4b C docker0
172.17.0.3 ether ae:1c:84:0a:da:3e C docker0
172.17.0.9 ether 8e:5b:68:a4:1e:ac C docker0
172.17.0.8 ether ea:36:26:15:35:fb C docker0

So this is why it failed

[Matthew Mosesohn (raytrac3r)]

This is a bug in Docker's IP address allocation. It recycles IPs it thinks are free, sometimes leading to duplicate IP addresses. This patch tries to be conservative and not reuse IP addresses until its pool runs out.
upstream docker bug: https://github.com/docker/docker/issues/5737
upstream patch that addresses this: https://github.com/docker/docker/pull/5783/

Reassigning to OSCI team so they can start working on this.

[Matthew Mosesohn (raytrac3r)]

We will not fix this for 5.0.1, but I am adding to release notes.

[OSCI Robot (oscirobot)]

Package docker-io has been built from changeset: http://gerrit.mirantis.com/20633
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20633/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20633/centos"

[OSCI Robot (oscirobot)]

Package docker-io has been built from changeset: http://gerrit.mirantis.com/20633
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos"

[Vitaly Parakhin (vparakhin)]

fix is on mirrors

[Evgeniy L (rustyrobot)]

Moving the bug to 6.0 after the discussion [1] in mailing list

[1] https://lists.launchpad.net/fuel-dev/msg01449.html

[Evgeniy L (rustyrobot)]

This problem appeared after we merged docker update

https://bugs.launchpad.net/fuel/+bug/1360161
https://bugs.launchpad.net/fuel/+bug/1360146

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/118387

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118387
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3ebbe106f1953602d2022297e7249e8a33da2871
Submitter: Jenkins
Branch: master

commit 3ebbe106f1953602d2022297e7249e8a33da2871
Author: Evgeniy L <email address hidden>
Date: Tue Sep 2 16:46:17 2014 +0400

    Upgrades, don't kill new containers during the upgrade

    Stopping of new containers during the upgrade
    leads to a lot of errors and raise conditions.
    At the same time containers should be run under
    supervisor because we need to store logs.

    Rewrote upgrade flow:

    * stop old containers
    * upload the images
    * generate supervisor config with autostart False,
      it allows to prevent supervisor to run containers
    * run containers in method `create_and_start_new_containers`
      one by one in right order
    * regenerate configs for supervisor with autostart
      True, to start all of the containers after supervisor
      restart
    * verify containers

    How it helps:

    * there was race condition when we were running
      services via supervisor and iptables cleaning
      at the same time, supervisor not always was
      able to start all containers, as result we
      could get nat rules with the same port but
      different ip addresses
    * containers stopping could interrupt non-atomic
      actions, like db migration in keystone container
    * since we run container one by one, we will not
      be able to get problem with ip duplication,
      during the upgrade

    Related-bug: #1357357
    Related-bug: #1364087
    Closes-bug: #1364054
    Change-Id: I86accb8b2c2fc5a15425e32838a58c9b45022d8d

[Ihor Kalnytskyi (ikalnytskyi)]

The issue was closed in 5.1 within the Fuel Upgrade script.

[Andrey Sledzinskiy (asledzinskiy)]

verified on fuel-5.1-upgrade-11-2014-09-17_21-40-34.tar.lrz