[upgrade] CentOS HA deployment failed after upgrade with Error: Cannot retrieve repository metadata (repomd.xml) for repository: Auxiliary

[Andrey Sledzinskiy]

{"build_id": "2015-04-17_15-24-00",
"ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847",
"build_number": "321",
"release_versions": {
"2014.2-6.0": {
"VERSION": {
"build_id": "2014-12-26_14-25-46",
"ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
"build_number": "58",
"api": "1.0",
"nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
"production": "docker",
"fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8",
"astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
"feature_groups":
["mirantis"],
"release": "6.0",
"fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}},
"2014.2-6.1": {
"VERSION": {
"build_id": "2015-04-17_15-24-00",
"ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847",
"build_number": "321",
"api": "1.0",
"nailgun_sha": "939e5780cd0f7b1af3afd2926eda30f81bfc3e3f",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "0698062e9b044becf07bf9918fa16613aa3d93ad",
"astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51",
"feature_groups":
["mirantis"],
"release": "6.1",
"fuelmain_sha": "5981d230e9484c196022a027c5c1600e36b17a72",
"fuellib_sha": "65617981bef34ea96b85d2d389cc037c304516e5"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "939e5780cd0f7b1af3afd2926eda30f81bfc3e3f",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "0698062e9b044becf07bf9918fa16613aa3d93ad",
"astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51",
"feature_groups":
["mirantis"],
"release": "6.1",
"fuelmain_sha": "5981d230e9484c196022a027c5c1600e36b17a72",
"fuellib_sha": "65617981bef34ea96b85d2d389cc037c304516e5"}

Steps:
1. Create and deploy next 6.0-58 cluster - CentOS, Neutron Vlan, simple mode, ceph for volumes and images, 1 controller+ceph, 2 compute+ceph
2. Upgrade fuel to 6.1
3. Create new 6.1 cluster after upgrade - CentOS, HA, Neutron Vlan, 3 controller, 2 compute, 1 cinder
4. Start deployment

Actual result - deployment failed with
2015-04-20 03:39:01 ERR

 Execution of '/usr/bin/yum -d 0 -e 0 -y install htop' returned 1: Error: Cannot retrieve repository metadata (repomd.xml) for repository: Auxiliary. Please verify its path and try again

[Ihor Kalnytskyi (ikalnytskyi)]

Looks like the issue is that we apply version.yaml in Docker upgrader, but we need to apply it before Host upgrader (since puppet uses it for retrieving openstack_version).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175769

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/177188

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/177725

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176779
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d3d4e8bcdfc19858c1f5c627b2772251a4b03ea2
Submitter: Jenkins
Branch: master

commit d3d4e8bcdfc19858c1f5c627b2772251a4b03ea2
Author: Matthew Mosesohn <email address hidden>
Date: Thu Apr 23 17:22:09 2015 +0300

    Run docker containers with explicit version

    Supervisor should start containers based on the
    version of Fuel, not specific to dockerctl. This
    will alleviate some issues with container versioning
    during upgrade and rollback of Fuel Master.

    Change-Id: Ia839cc2aa4e584b7b01793a5738f6542b9a07c8e
    Related-Bug: #1446075
    Closes-Bug: #1447567

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177188
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=abfe335b24ea7d8ca9942b42f23bed7ac8086080
Submitter: Jenkins
Branch: master

commit abfe335b24ea7d8ca9942b42f23bed7ac8086080
Author: Igor Kalnitsky <email address hidden>
Date: Thu Apr 23 18:54:47 2015 +0300

    Set fixed version in containers' supervisor confs

    Currently, containers' supervisor configs don't have the Fuel version
    in the command line. It means that when supervisor tries to start a
    container its version retrieved on fly from the '/etc/fuel/version.yaml'.
    Since Fuel 6.1, the '/etc/fuel/version.yaml` has to be new, because
    otherwise the 'host-upgrade.pp' will use incorrect current version.
    Unfortunately, if '/etc/fuel/version.yaml' is new and the puppet
    upgrades Docker package, the Docker containers will be stopped and
    they won't up again because 'detecting container version on fly' will
    give us wrong result (e.g. 6.1 instead of 6.0). So, the only thing
    we can do is to set fixed container version in supervisor's configs,
    so it won't rely on current state of '/etc/fuel/version.yaml'.

    Required-For: Ic68f19c7f6bb68a04fb5cc457dae7f92c1faacf2
    Related-Bug: #1446075

    Change-Id: I1a32827952fcfbe705f37f46df3e80f6c4dc8bb3
    Signed-off-by: Igor Kalnitsky <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/177725
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=c28f3fb1f227d701e49fd9e571c867b590f5fb17
Submitter: Jenkins
Branch: master

commit c28f3fb1f227d701e49fd9e571c867b590f5fb17
Author: Igor Kalnitsky <email address hidden>
Date: Mon Apr 27 12:19:54 2015 +0300

    Install repos in HostSystem upgrader

    Currently, the OpenStack upgrader copies repos to '/var/www/nailgun',
    but before copying it checks whether they are installed or not and if
    they do - removes them in order to make clear installation.

    That cause issues when HostSystem upgrader creates 'Auxiliary' repos
    and then OpenStack upgrader removes them as a part of repos
    installation. In order to prevent that smeared logic about repos
    installation, let's move it all to HostSystem upgrader.

    Required-For: Ic68f19c7f6bb68a04fb5cc457dae7f92c1faacf2
    Related-Bug: #1446075

    Change-Id: I56e64e44786bcf08f83fc62ff59af747192e2470
    Signed-off-by: Igor Kalnitsky <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/175769
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b55a5e50da400c491621a2eb915d375e42ddc360
Submitter: Jenkins
Branch: master

commit b55a5e50da400c491621a2eb915d375e42ddc360
Author: Igor Kalnitsky <email address hidden>
Date: Mon Apr 20 20:04:16 2015 +0300

    Change version.yaml link in UpgradeManager

    We have to change version.yaml link before starting upgrade process,
    because it might be used by different upgraders. For instance, we used
    to manipulate this link in Docker Upgrader, but recently Host Upgrader
    has been started rely on the values of new version.yaml too.

    Closes-Bug: 1446075

    Change-Id: Ic68f19c7f6bb68a04fb5cc457dae7f92c1faacf2
    Signed-off-by: Igor Kalnitsky <email address hidden>

[Maksym Strukov (unbelll)]

Verified as fixed

{"build_id": "2015-05-12_17-03-01", "build_number": "408", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}, "2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-12_17-03-01", "build_number": "408", "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "042ed77ffdff22a5242ac7f8a6980836d0a5ef1a", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "d9d488c4c675e6dd33eb68b33a79abe591b4c26f", "fuel-ostf_sha": "21afa436e725be1debadf1c207018753b537c7b3", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "042ed77ffdff22a5242ac7f8a6980836d0a5ef1a", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "d9d488c4c675e6dd33eb68b33a79abe591b4c26f", "fuel-ostf_sha": "21afa436e725be1debadf1c207018753b537c7b3", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"}