Upgrades, previous containers are not running and upgrade fails because cannot make db dump

Bug #1360161 reported by Evgeniy L
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
Critical
Fuel Python (Deprecated)

Bug Description

1. install released 5.0 iso
2. run upgrade to 5.0.1 it succeed
3. run upgrade to 5.1

Actual result:
Upgrade script cannot make database dump, because postgresql container was not running

Error: Cannot start container fuel-core-5.0.1-postgres: Error getting container d3c72b317a82de0935472255be5e9d917cfe84d5dc6dbb1f2e0b0b2391ceab71 from driver devicemapper: Error mounting '/dev/mapper/docker-253:2-651-d3c72b317a82de0935472255be5e9d917cfe84d5dc6dbb1f2e0b0b2391ceab71' on '/var/lib/docker/devicemapper/mnt/d3c72b317a82de0935472255be5e9d917cfe84d5dc6dbb1f2e0b0b2391ceab71': device or resource busy

The reason is during the docker upgrade we update docker and restart the service, docker stops all of the containers, but doesn't umount volumes.

Expected result:
Upgrade succeed

*5.0.1*
VERSION:
  mirantis: "yes"
  production: "docker"
  release: "5.0.1"
  api: "1.0"
  build_number: "170"
  build_id: "2014-08-14_19-52-36"
  astute_sha: "6db5f5031b74e67b92fcac1f7998eaa296d68025"
  fuellib_sha: "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"
  ostf_sha: "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f"
  nailgun_sha: "af3d1922bfc21345f81be3454115ab6139675c35"
  fuelmain_sha: "fd58828f404e4298ed338e8f44c6a326cebd31de"

*5.1*
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "465"
  build_id: "2014-08-21_10-53-07"
  astute_sha: "efe3cb3668b9079e68fb1534fd4649ac45a344e1"
  fuellib_sha: "52f3ebfa968f0338e0584edf47cff10911109de5"
  ostf_sha: "c6ecd0137b5d7c1576fa65baef0fc70f9a150daa"
  nailgun_sha: "25eba6fbb2047f26d9da4d27ffdb742c9c27832a"
  fuelmain_sha: "25a0c228d998707f90e90877559f17817a749d2f"

Tags: upgrade
Revision history for this message
Evgeniy L (rustyrobot) wrote :
Changed in fuel:
milestone: none → 5.1
Revision history for this message
Evgeniy L (rustyrobot) wrote :

There are several ways to fix it

1. patch supervisor configs and use dockerctl to run containers, when I tried it last time, we had timeout issues (need to retest it again)
2. stop supervisor and containers before we run docker upgrade
3. before puppet run do docker upgrade explicitly in upgrade script and handle all possible cases there

Personally I like the third solution.

Revision history for this message
Evgeniy L (rustyrobot) wrote :

Set won't fix because this problem was caused by docker updating, we won't update docker for this release

https://bugs.launchpad.net/fuel/+bug/1360274 - remove new docker from repo
https://bugs.launchpad.net/fuel/+bug/1357357 - reopened the issue and moved to 6.0

Changed in fuel:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.