[fuel-upgrade-system] HostSystemUpgrader is not idempotent

[Bulat Gaifullin]

steps to reproduce:
1. deploy the product of version 6.1
2. download 7.0 upgrade tarball
3. extract upgrade files from tarball
4. start upgrader. upgrader.sh
5. upgrader completed successfully.
6. start upgrader.sh again or manually start host-system upgrade.
  upgrade/.fuel-upgrade-venv/bin/fuel-upgrade host-system --src=upgradel

expected result:
  host system upgrader completed successfully.
actual result:
  host system upgrade failed with error

 File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 176, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 168, in run_upgrade
    upgrade_manager.run()
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 97, in upgrade
    self.install_packages()
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 143, in install_packages
    utils.exec_cmd('yum install -v -y {0}'.format(package))
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/root/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: yum install -v -y fuel-7.0.0

in case if start host-upgrader again it completes successfully.

the root cause in the following:
during upgrade the puppet installs the repo auxiliary /etc/yum.repos.d/7.0_auxiliary.repo (file:///var/www/nailgun/2015.1.0-7.0/centos/auxiliary/)
in case if starts host-upgrader again:
  it removes the repository folder, but keeps repository config file, that contain reference to non-existent repository.

[Bulat Gaifullin (bulat.gaifullin)]

I suggest to ensure that file /etc/yum.repos.d/<version>_auxiliary.rep does not exists before starts host-system upgrade.
or set option skip_if_unavailable=1 for repository.

[Sebastian Kalinowski (prmtl)]

I thought that it's imposible to run upgrade again after successful upgrade (we check version of Fuel).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/223511

[Artem Roma (aroma-x)]

JFYI, I couldn't start upgrade again after previous had successfully completed with error message 'WrongVersionError: Cannot upgrade to the same version of fuel 7.0 -> 7.0' , so @Sebastian got the point with his comment. I manually executed 'upgrade.sh' script if you are interested in performed actions.

[Bulat Gaifullin (bulat.gaifullin)]

the version of fuel checks only in DockerUpgrader.
also when you start upgrade.sh - yes, it checks and upgrade stops with message that already upgraded. but if you specify argument
--no-checker - upgrade will fail.
when start fuel-upgrade.py host-system --src=<path to upgrade files> upgrader does not check that already upgraded and fail during execute upgrade.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223511
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=6d0e40a208dc03b90790850715152260c80cec09
Submitter: Jenkins
Branch: master

commit 6d0e40a208dc03b90790850715152260c80cec09
Author: Bulat Gaifullin <email address hidden>
Date: Tue Sep 15 13:11:06 2015 +0300

    HostSystemUpgrader is not idempotent - fixed

    enable version checker for host-system
    remove config of auxiliary-repo before upgrade

    Closes-Bug: #1495481
    Change-Id: Ifcca5d436ffd87a2f268d86ad80fdb733170c23c

[Veronica Krayneva (vkrayneva)]

Checked on upgrade 7.0 (301 iso) to 8.0 (http://srv89-bud.infra.mirantis.net/fuelweb-iso/fuel-8.0-upgrade-84-2015-10-28_01-42-00.tar.lrz)
Steps:
1. first upgrade
2. Second upgrade (upgrade/.fuel-upgrade-venv/bin/fuel-upgrade host-system --src=upgrade --no-checker)
 Bug was fixed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/240213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240213
Committed: https://git.openstack.org/cgit/openstack/fuel-upgrade/commit/?id=9331d7d775a2272a64fc6f1dfa8c7f392abc9ced
Submitter: Jenkins
Branch: master

commit 9331d7d775a2272a64fc6f1dfa8c7f392abc9ced
Author: Bulat Gaifullin <email address hidden>
Date: Tue Sep 15 13:11:06 2015 +0300

    HostSystemUpgrader is not idempotent - fixed

    enable version checker for host-system
    remove config of auxiliary-repo before upgrade

    Closes-Bug: #1495481
    Change-Id: Ifcca5d436ffd87a2f268d86ad80fdb733170c23c