Sequential upgrade of Fuel Master node 5.1.1 -> 6.0 -> 6.1 failed

Bug #1483216 reported by Oleg S. Gelbukh on 2015-08-10
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Roman Prykhodchenko

Bug Description

Steps to reproduce:
1. Install fuel master 5.1.1
2. Upgrade master from 5.1.1 to 6.0 via tar ball
3. Upgrade master from 6.0 to 6.1 via tar ball

Actual result:
Upgrade from 6.0 to 6.1.1 fails with the following trace:

2015-08-09 13:07:05 DEBUG 4392 (utils) cp: target `/var/lib/cobbler/config/' is not a directory
2015-08-09 13:07:05 ERROR 4392 (upgrade) DockerUpgrader: failed to upgrade: "Shell command executed with "1" exit code: dockerctl shell b5c1deb6bbcd517d68ee5750d50f369a54b2515551081
a49dde7b210ec03d15e bash -c 'cp -rn /tmp/upgrade/cobbler_configs/config/* /var/lib/cobbler/config/' "
Traceback (most recent call last):
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
   upgrader.upgrade()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 79, in upgrade
   self.create_and_start_new_containers()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 273, in create_and_start_new_containers
   self.run_after_container_creation_command(container)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 288, in run_after_container_creation_command
   '', retries=30, interval=4)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 329, in exec_with_retries
   return func()
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 284, in execute
   self.exec_cmd_in_container(container['container_name'], command)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 296, in exec_cmd_in_container
   utils.exec_cmd("dockerctl shell {0} {1}".format(db_container_id, cmd))
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
   _wait_and_check_exit_code(cmd, child)
 File "/var/upgrade/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
   'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: dockerctl shell b5c1deb6bbcd517d68ee5750d50f369a54b2515551081a49dde7b210ec03d15e bash -c 'cp -rn /tmp/upgrad
e/cobbler_configs/config/* /var/lib/cobbler/config/'

Version information

{"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}, "2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

Oleg S. Gelbukh (gelbuhos) wrote :

Please, see diagnostic snapshot by the following link: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-08-10_10-31-31.tgz

Roman Prykhodchenko (romcheg) wrote :

@Oleg, please put version json's to the bottom of the description.

description: updated
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → High
milestone: none → 7.0
status: New → Confirmed
tags: added: feature-upgrade
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)
Ihor Kalnytskyi (ikalnytskyi) wrote :

Move to sustaining team, since it's about previous release.

Changed in fuel:
milestone: 7.0 → 6.1-updates
assignee: Fuel Python Team (fuel-python) → MOS Sustaining (mos-sustaining)
Vitaly Sedelnik (vsedelnik) wrote :

Please provide a fix in 7.0, then we could consider backporting it into previous release(s).

Changed in fuel:
assignee: MOS Sustaining (mos-sustaining) → Igor Kalnitsky (ikalnitsky)
milestone: 6.1-updates → 7.0

Fix proposed to branch: master
Review: https://review.openstack.org/212029

Changed in fuel:
status: Confirmed → In Progress
Changed in fuel:
status: In Progress → Confirmed
Roman Prykhodchenko (romcheg) wrote :

>> 3. Upgrade master from 6.0 to 6.1.1 via tar ball

I suppose it should be 6.1

Roman Prykhodchenko (romcheg) wrote :

I tried to reproduce the bug by following the described steps. However, the master node was upgraded successfully. I will analyze the attached snapshot to find the reason.

Meanwhile I have the question: was that master node fresh and untouched, or perhaps some modifications had been made?

Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Roman Prykhodchenko (romcheg)
Roman Prykhodchenko (romcheg) wrote :

Definitely seems to be a Docker issue. Looks like the bug requires a live problematic environment. Please poke me or Igor as soon as you hit it one more time.

Changed in fuel:
status: Confirmed → Incomplete
Andrew Maksimov (maximov) wrote :

No updates since 8/14, we will have to move this bug to 8.0 because of HCF.

Changed in fuel:
status: Incomplete → Won't Fix
description: updated
tags: added: customer-found
Dmitry Pyzhov (dpyzhov) wrote :

No response for a month. Marking the bug as Invalid. Please reopen it if you have live environment with the issue.

no longer affects: fuel/8.0.x
Changed in fuel:
status: Won't Fix → Invalid
milestone: 7.0 → 8.0
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-python
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers