Sequential upgrade of Fuel Master node 5.1.1 -> 6.0 -> 6.1 failed

[Oleg S. Gelbukh]

Steps to reproduce:
1. Install fuel master 5.1.1
2. Upgrade master from 5.1.1 to 6.0 via tar ball
3. Upgrade master from 6.0 to 6.1 via tar ball

Actual result:
Upgrade from 6.0 to 6.1.1 fails with the following trace:

2015-08-09 13:07:05 DEBUG 4392 (utils) cp: target `/var/lib/cobbler/config/' is not a directory
2015-08-09 13:07:05 ERROR 4392 (upgrade) DockerUpgrader: failed to upgrade: "Shell command executed with "1" exit code: dockerctl shell b5c1deb6bbcd517d68ee5750d50f369a54b2515551081
a49dde7b210ec03d15e bash -c 'cp -rn /tmp/upgrade/cobbler_configs/config/* /var/lib/cobbler/config/' "
Traceback (most recent call last):
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
   upgrader.upgrade()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 79, in upgrade
   self.create_and_start_new_containers()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 273, in create_and_start_new_containers
   self.run_after_container_creation_command(container)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 288, in run_after_container_creation_command
   '', retries=30, interval=4)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 329, in exec_with_retries
   return func()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 284, in execute
   self.exec_cmd_in_container(container['container_name'], command)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 296, in exec_cmd_in_container
   utils.exec_cmd("dockerctl shell {0} {1}".format(db_container_id, cmd))
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
   _wait_and_check_exit_code(cmd, child)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
   'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: dockerctl shell b5c1deb6bbcd517d68ee5750d50f369a54b2515551081a49dde7b210ec03d15e bash -c 'cp -rn /tmp/upgrad
e/cobbler_configs/config/* /var/lib/cobbler/config/'

Version information

{"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}, "2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

[Oleg S. Gelbukh (gelbuhos)]

Please, see diagnostic snapshot by the following link: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-08-10_10-31-31.tgz

[Roman Prykhodchenko (romcheg)]

@Oleg, please put version json's to the bottom of the description.

[Ihor Kalnytskyi (ikalnytskyi)]

Move to sustaining team, since it's about previous release.

[Vitaly Sedelnik (vsedelnik)]

Please provide a fix in 7.0, then we could consider backporting it into previous release(s).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212029

[Roman Prykhodchenko (romcheg)]

>> 3. Upgrade master from 6.0 to 6.1.1 via tar ball

I suppose it should be 6.1

[Roman Prykhodchenko (romcheg)]

I tried to reproduce the bug by following the described steps. However, the master node was upgraded successfully. I will analyze the attached snapshot to find the reason.

Meanwhile I have the question: was that master node fresh and untouched, or perhaps some modifications had been made?

[Roman Prykhodchenko (romcheg)]

Definitely seems to be a Docker issue. Looks like the bug requires a live problematic environment. Please poke me or Igor as soon as you hit it one more time.

[Andrey Maximov (maximov)]

No updates since 8/14, we will have to move this bug to 8.0 because of HCF.

[Dmitry Pyzhov (dpyzhov)]

No response for a month. Marking the bug as Invalid. Please reopen it if you have live environment with the issue.