[upgrade] Postgres and Cobbler containers remain non-stateless after upgrade

Bug #1469399 reported by Miroslav Anashkin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Kuklin
6.1.x
Fix Released
Critical
Fuel Library (Deprecated)
7.0.x
Fix Released
Critical
Vladimir Kuklin

Bug Description

upgrade.sh prints the following message at the end of successful master node upgrade:

_____________________________________________________________
Both updates and security repos were setuped. If you want to
retrieve latest updates make sure you have access to these repos
and run:

    # yum update
    # dockerctl destroy all
    # dockerctl start all

______________________________________________________________
This procedure completely REMOVES ALL already existing environments from Nailgun DB on master node.

Root cause: upgrade script does not create the directories, used as external mount points for Cobbler and Postgres. So, the contents of these directories remains inside the Docker file system.

description: updated
Changed in fuel:
assignee: nobody → Fuel for Openstack (fuel)
tags: removed: feature-upgrade
description: updated
Changed in fuel:
assignee: Fuel for Openstack (fuel) → Fuel Python Team (fuel-python)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Miroslav Anashkin (manashkin) wrote :

Found the root cause of incorrect 'dockerctl destroy all' results.
It appears, during upgrade puppet code does not creates end directories to store that data, which we moved outside postgres, cobbler and astute containers in order to make them stateless.
As a result we getting empty
/var/lib/fuel/container_data/${release}/
directory without ./cobbler, ./postgres and ./astute sub-directories.

During upgrade and after, Docker starts containers and complains nothing about the absent mount points. Instead, inside containers it thinks the appropriate directories are mount points, while actually these directories remain in the local internal container file system.

We need to fix puppet code to create the necessary directories (postgres, astute and cobbler) inside
/var/lib/fuel/container_data/${release}/

The message 'dockerctl destroy all' is actually correct for 6.1 and normally should not lead to data loss.

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Matthew Mosesohn (raytrac3r)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/196727

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Evgeniy L (rustyrobot)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/196748

Changed in fuel:
assignee: Evgeniy L (rustyrobot) → Vladimir Kuklin (vkuklin)
summary: - [upgrade] Remove "dockerctl destroy all" advice message from upgrade
- script
+ [upgrade] Postgres and Cobbler containers remain non-stateless after
+ upgrade
description: updated
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/6.1)

Reviewed: https://review.openstack.org/196748
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b5504919593f9075e230719b463eaf615c51d838
Submitter: Jenkins
Branch: stable/6.1

commit b5504919593f9075e230719b463eaf615c51d838
Author: Evgeniy L <email address hidden>
Date: Mon Jun 29 20:02:13 2015 +0300

    Correct postgres and cobbler bind mounts

    Change-Id: I4925bc16a1787e79cd1e21d819239babfbad516e
    Closes-bug: #1469399

Maksym Strukov (unbelll)
tags: added: on-verification
Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 7.0-26

{"build_id": "2015-07-06_18-08-24", "build_number": "26", "release_versions": {"2014.2.2-7.0": {"VERSION": {"build_id": "2015-07-06_18-08-24", "build_number": "26", "api": "1.0", "fuel-library_sha": "251c54e8de2f41aacd260751e7a891e9fbffc45d", "nailgun_sha": "d040c5cebc9cdd24ef20cb7ecf0a337039baddec", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "a752c857deafd2629baf646b1b3188f02ff38084", "release": "7.0", "fuelmain_sha": "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"}}, "2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "251c54e8de2f41aacd260751e7a891e9fbffc45d", "nailgun_sha": "d040c5cebc9cdd24ef20cb7ecf0a337039baddec", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "a752c857deafd2629baf646b1b3188f02ff38084", "release": "7.0", "fuelmain_sha": "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"}

tags: removed: on-verification
tags: added: to-rca
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/196727

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.