upgrade: after rollback the system isn't in working state

[Ihor Kalnytskyi]

Some time ago, we have accept the patch into fuel_upgrade which moves repos installation logic from openstack upgrader into host-system upgrader. That cause non-obvious issue with rollback and here is it.

1. docker/openstack upgrader fails and rollback procedure is initiated
2. we're stopping new containters and run old ones
3. the new repo is still on host filesystem, therefore it will be mounted into old containers
4. puppets inside old containers tries to install latest available packages, and that breaks dependencies (for instance, nailgun may depends on urllib3==1.7 but urllib3=1.7,1 will be installed, so it will be impossible to run nailgun anymore).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/180988

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180988
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=6ad2da045fecad0d6a99f020a01f6d89f0e16532
Submitter: Jenkins
Branch: master

commit 6ad2da045fecad0d6a99f020a01f6d89f0e16532
Author: Igor Kalnitsky <email address hidden>
Date: Thu May 7 16:16:20 2015 +0300

    Run HostSystem's rollback before others

    We have to run HostSystem's rollback before others, because its result
    may affect other upgraders. For example, if we don't remove new CentOS
    repo from the master node the old containers may use it and puppet
    inside will install latest available packages. In some cases, that may
    lead to the fact when Fuel won't work after rollback at all because of
    broken strict dependencies.

    Closes-Bug: #1452378

    Change-Id: Ia17838b84909e5bd0edc7570ea4e0ebb21d94746
    Signed-off-by: Igor Kalnitsky <email address hidden>

[Maksym Strukov (unbelll)]

Verified as fixed in 6.1-432

Steps:
1. Install 6.0-58
2. Deploy cluster (centos, ha, 3 controllers, 2 compute, neutron gre)
3. Run upgrade with fake raise-error, wait until rollback completed
4. Ensure fuel not upgraded (still 6.0)
5. Add compute to this cluster
6. Deploy

Expected:
Compute can be successful added to 6.0-cluster after upgrade-rollback