[upgrade] OSTF isn't available after upgrade due to wrong port forwarding

[Artem Panchenko]

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-08-29_00-01-17
build_number: '486'
feature_groups:
- mirantis
fuellib_sha: 639ac9e633b13e9cbdb93abee0423a881d70b105
fuelmain_sha: c450b341ea416813e8358026754d85228f4513eb
nailgun_sha: a762c6029ba852e73ad6aef89fbcd0a8afb79d87
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

Steps to reproduce:

1. Deploy Fuel 5.0 release iso, create and deploy new OS environment
2. Download and unpack upgrade tarball & run upgrade to 5.1
3. Upgrade is successful. Go to Fuel UI and try to run Health Checks for cluster

Expected result:

- health checks can be performed for the environment

Actual:

- OSTF server is not available

I checked firewall settings and found incorrect DNAT rule for OSTF:

http://paste.openstack.org/show/104174/

As you can see traffic targeted to the OSTF server was redirected to the rabbitmq container. I thought that old rule for 5.0 containers wasn't removed, but according to upgrade logs it didn't exist before upgrade:

http://paste.openstack.org/show/104205/

In my opinion it was created by docker itself or dockerctl hooks after new containers launch. Probably this issue can be workarounded by adding dockerctl post_start_hook for ostf with ports remangle function.

Also, I couldn't reproduce this problem using the same tarball (tried 3 times), so it's floating issue.

Diagnostic snapshot: https://yadi.sk/d/NE5SOfWzaowtg

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/118387

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118387
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3ebbe106f1953602d2022297e7249e8a33da2871
Submitter: Jenkins
Branch: master

commit 3ebbe106f1953602d2022297e7249e8a33da2871
Author: Evgeniy L <email address hidden>
Date: Tue Sep 2 16:46:17 2014 +0400

    Upgrades, don't kill new containers during the upgrade

    Stopping of new containers during the upgrade
    leads to a lot of errors and raise conditions.
    At the same time containers should be run under
    supervisor because we need to store logs.

    Rewrote upgrade flow:

    * stop old containers
    * upload the images
    * generate supervisor config with autostart False,
      it allows to prevent supervisor to run containers
    * run containers in method `create_and_start_new_containers`
      one by one in right order
    * regenerate configs for supervisor with autostart
      True, to start all of the containers after supervisor
      restart
    * verify containers

    How it helps:

    * there was race condition when we were running
      services via supervisor and iptables cleaning
      at the same time, supervisor not always was
      able to start all containers, as result we
      could get nat rules with the same port but
      different ip addresses
    * containers stopping could interrupt non-atomic
      actions, like db migration in keystone container
    * since we run container one by one, we will not
      be able to get problem with ip duplication,
      during the upgrade

    Related-bug: #1357357
    Related-bug: #1364087
    Closes-bug: #1364054
    Change-Id: I86accb8b2c2fc5a15425e32838a58c9b45022d8d

[Andrey Sledzinskiy (asledzinskiy)]

verified on fuel-5.1-upgrade-11-2014-09-17_21-40-34.tar.lrz