[upgrade] Old keystone container is up after upgrade

[Alexander Kurenyshev]

Steps to reproduce:

1. Create cluster
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
4. Deploy the cluster
5. Run upgrade on master
3. Check that upgrade was successful:
    docker ps | tail -n +2 | awk '{ print $NF;}'

Expected behaviour:
All containers have 6.1 version

Actual behaviour:
Keystone container has 6.0 version

[Matthew Mosesohn (raytrac3r)]

This is an issue in fuel_upgrade. Here is the behavior I observed in fuel_upgrade.log:
2015-04-08 09:26:07 INFO 27920 (supervisor_client) Stop all services
2015-04-08 09:31:44 DEBUG 27920 (docker_engine) Stop container: {u'Status': u'Up 8 seconds', u'Created': 1428480541, u'Image': u'fuel/keystone_6.0:latest', u'Ports': [{u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 35357, u'PrivatePort': 35357}, {u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 5000, u'PrivatePort': 5000}], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-6.0-keystone'], u'Id': u'85cd258d521121c74cec0533cc54d8d64a5e0bd5a4a4eb1f964467ee3b998ed5'}

But supervisor is still stopping containers at 09:31:44. In supervisord.log, we see:
2015-04-08 09:32:29,479 DEBG fd 8 closed, stopped monitoring <POutputDispatcher at 30665904 for <Subprocess at 30227920 with name docker-keystone in state RUNNING> (stdout)>
2015-04-08 09:32:29,480 INFO exited: docker-keystone (exit status 0; expected)
2015-04-08 09:32:29,480 DEBG received SIGCLD indicating a child quit
2015-04-08 09:32:30,482 INFO spawned: 'docker-keystone' with pid 8032
2015-04-08 09:32:31,097 DEBG 'docker-keystone' stdout output:
fuel-core-6.0-keystone

It seems supervisord restarted docker-keystone because it didn't start handling the stop command yet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/173389

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/173389
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=740aaea86b9889daa9295ec34ad7c2ee9dba82ca
Submitter: Jenkins
Branch: master

commit 740aaea86b9889daa9295ec34ad7c2ee9dba82ca
Author: Igor Kalnitsky <email address hidden>
Date: Tue Apr 14 16:43:58 2015 +0300

    Change docker upgrade steps

    We used to have an 'error-prone' approach of making Docker upgrades,
    and in really rare cases we had an old container in 'running' state
    instead of new one.

    Here's the steps when issue occurs:

    * One container isn't running due to some reason (for instance, it
      failed). The supervisor scheduled to start it again soon.
    * The fuel_upgrade script stops supervisor's processes. They will be
      stopped and won't be started again by supervisor (even if they have
      autostart=true setting).
    * The container, which wasn't run by the time we was stopping them,
      is untouched. It's still scheduled to be running soon.
    * The fuel_upgrade script stops Docker containers, but one of them
      will be starting again and again because of that working supervisor
      process.

    In order to avoid this situation, we have to do the following things:

    * Point supervisor to new config folder (which is empty by now).
    * Restart supervisor, so it forget old configs and will use no of them.
    * Upload images and create new containers.
    * Generate new supervisor configs and restart supervisor in order to
      apply them (supervisor's processes will be attached automatically
      to running docker containers).

    Signed-off-by: Igor Kalnitsky <email address hidden>

    Change-Id: Ibc133b4471878efeffeb232192b4540d26401fae
    Closes-Bug: #1441693

[Maksym Strukov (unbelll)]

Verified as fixed in 6.1-360

{"build_id": "2015-04-28_22-54-38", "build_number": "360", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}, "2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-28_22-54-38", "build_number": "360", "api": "1.0", "fuel-library_sha": "0e5b82d24853304befb22145ac4aaf3545d295e1", "nailgun_sha": "5753e0ce613473381a63c63d4309777fce4bea70", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "8cd6cf575d3c101dee1032abb6877dfa8487e077", "astute_sha": "04ebab96d57b0e8acbf2d7f3ba05e4fbf31b741e", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "ee112acfdd0f9017ef40be53e8e51bb5c429e97c"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "0e5b82d24853304befb22145ac4aaf3545d295e1", "nailgun_sha": "5753e0ce613473381a63c63d4309777fce4bea70", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "8cd6cf575d3c101dee1032abb6877dfa8487e077", "astute_sha": "04ebab96d57b0e8acbf2d7f3ba05e4fbf31b741e", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "ee112acfdd0f9017ef40be53e8e51bb5c429e97c"}