[Upgrade] Upgrade failed with Cannot find running container with name "fuel-core-5.0-postgres"

[Andrey Sledzinskiy]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "445"
  build_id: "2014-08-16_11-15-55"
  astute_sha: "8e1db3926b2320b30b23d7a772122521b0d96166"
  fuellib_sha: "81741445d4ab2db198585149815bb80a238a1214"
  ostf_sha: "d2a894d228c1f3c22595a77f04b1e00d09d8e463"
  nailgun_sha: "bc9e377dbe010732bc2ba47161ed9d433998e07b"
  fuelmain_sha: "08f04775dcfadd8f5b438a31c63e81f29276b7d3"

Steps:
1. Install fuel 5.0-26
2. Create simple cluster with all other default values - 1 controller, 1 compute, 1 cinder
3. Deploy cluster
4. Upload 5.1 upgrade script to master and run upgrade

Expected - master was successfully upgraded to 5.1
Actual - upgrade failed with
2014-08-16 13:52:14 ERROR 20041 (cli) Cannot find running container with name "fuel-core-5.0-postgres"
Traceback (most recent call last):
  File "/var/upgrade/site-packages/fuel_upgrade/cli.py", line 142, in main
    run_upgrade(parse_args())
  File "/var/upgrade/site-packages/fuel_upgrade/cli.py", line 135, in run_upgrade
    upgrade_manager.run()
  File "/var/upgrade/site-packages/fuel_upgrade/upgrade.py", line 51, in run
    upgrader.upgrade()
  File "/var/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 69, in upgrade
    self.save_db()
  File "/var/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 168, in save_db
    pg_dump_tmp_path))
  File "/var/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 329, in exec_cmd_in_container
    db_container_id = self.container_docker_id(container_name)
  File "/var/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 717, in container_docker_id
    'Cannot find running container with name "{0}"'.format(name))
CannotFindContainerError: Cannot find running container with name "fuel-core-5.0-postgres"

Also after rollback all containers were down

Logs are attached

[Ihor Kalnytskyi (ikalnytskyi)]

After exectuing puppets on host system we have no running containers. They even failed to start:

    Error mounting '/dev/mapper/docker-253:2-651-23ec6c18a0fdebf6981c3d94b67d06d02bc27b2bd1b3021dbee04a37a0306588' on '/var/lib/docker/devicemapper/mnt/23ec6c18a0fdebf6981c3d94b67d06d02bc27b2bd1b3021dbee04a37a0306588': device or resource busy

So it's probable library side.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114921

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/115315

[Matthew Mosesohn (raytrac3r)]

Further writeup and discussion about this bug here:
https://etherpad.openstack.org/p/fuel-upgrade-docker-fix

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/115616

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/115636

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112077
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0f27198dc46084546c505f41a5382a37b1952917
Submitter: Jenkins
Branch: master

commit 0f27198dc46084546c505f41a5382a37b1952917
Author: Aleksandr Didenko <email address hidden>
Date: Tue Aug 5 19:40:50 2014 +0300

    Revamp supervisor docker, Create host upgrade w/o supervisor

    - Create new file host-upgrade.pp that should be used for Fuel
      upgrades instead of host-only.pp. Upgrade mode excludes
      supervisor from upgrades. Fuel upgrade utility will install
      new supervisor configuration independent of Puppet.
    - Refactor dockerctl attach function to provide possibility to
      correctly handle SIGINT sent from supervisord during stop
      process. In order to do so we fork "docker attach" and run loop
      in order to "trap" (catch) SIGINT/SIGTERM and shutdown container
      properly
    - Use SIGTERM instead of SIGINT for dhcrelay_monitor supervisor
      service, because we have "sleep 30" loop in the monitor script,
      so it will handle SIGINT only when sleep ends.
    - Break dependency of docker build on supervisor

    Change-Id: I100c7705eb5515158bd940519fc02323c0bbe983
    Related-bug: #1350764
    Partial-bug: #1358144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115315
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=809cd6c2e64cb0cb144472b5c8927c915d4eef05
Submitter: Jenkins
Branch: master

commit 809cd6c2e64cb0cb144472b5c8927c915d4eef05
Author: Matthew Mosesohn <email address hidden>
Date: Tue Aug 19 19:11:59 2014 +0400

    Use host-upgrade.pp and keep old fuel_version

    This enables docker package upgrades on host system,
    which requires dockerctl to be able to restart
    all containers. The switch to host-upgrade.pp
    is necessary to run puppet and not break
    supervisor config.

    Change-Id: I2c4fa2f030cda00b456499af5d5245162714ccaf
    Related-Bug: #1358144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115616
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=94c509e5cd36b541b4ffbac10fc6db30ebda083a
Submitter: Jenkins
Branch: master

commit 94c509e5cd36b541b4ffbac10fc6db30ebda083a
Author: Evgeniy L <email address hidden>
Date: Wed Aug 20 12:21:48 2014 +0000

    Revert "fuel_upgrade: add KillSupervisord hook"

    We don't need to kill supervisor, because
    during host-system upgrade we don't restart
    supervisor anymore
    https://review.openstack.org/#/c/115315/
    https://review.openstack.org/#/c/112077/

    As result when we kill container in pre-upgrade
    hook, nobody runs it and upgrade fails because
    we use supervisor during docker containers
    upgrade.

    Related-Bug: #1358144
    This reverts commit 71e55fe025a38e4c734a41e2e202462f0c29e7e0.

    Change-Id: I0329b5110d632ac675f7d2e12c64c9ca4ea446d3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115636
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3fa5de9eec7d0ff08e28a33e6acfa7a25a1fe0d1
Submitter: Jenkins
Branch: master

commit 3fa5de9eec7d0ff08e28a33e6acfa7a25a1fe0d1
Author: Evgeniy L <email address hidden>
Date: Tue Aug 19 20:42:39 2014 +0400

    Upgrades, add retries on first database dump

    Add retries for for db dump, because
    there could be cases when postgres
    container didn't have time to start
    and run postgres service.

    Related-Bug: #1358144
    Change-Id: Ib86b3b56a7a52616603c0d8143282276395cb749

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114921
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=aed47c868e46f05dedd66f992e19bfa7c9ff29b6
Submitter: Jenkins
Branch: master

commit aed47c868e46f05dedd66f992e19bfa7c9ff29b6
Author: Matthew Mosesohn <email address hidden>
Date: Mon Aug 18 13:37:58 2014 +0400

    Restart containers on docker service refresh

    Devicemapper bugs in docker 0.10 cause containers
    to incorrectly restart when docker service is
    restarted when Puppet upgrades the RPM. Added
    lazy unmounts to dockerctl and a refreshonly
    exec to start all containers when Docker service
    is restarted (because of package upgrade).

    Puppet Docker service resource tries to also
    start all Fuel containers. This will show a
    warning if storage containers are not yet built.

    Includes bugfixes for purging iptables rules and
    starting nailgun and ostf containers while nginx
    container is not yet started.

    Change-Id: I9c6d9b04bdaa5cf1fcc9752cc1e8f42cf693de1b
    Closes-Bug: #1358144

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-08-21_02-01-17",
    "ostf_sha": "c6ecd0137b5d7c1576fa65baef0fc70f9a150daa",
    "build_number": "464",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "25eba6fbb2047f26d9da4d27ffdb742c9c27832a",
    "production": "docker",
    "fuelmain_sha": "25a0c228d998707f90e90877559f17817a749d2f",
    "astute_sha": "efe3cb3668b9079e68fb1534fd4649ac45a344e1",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "52f3ebfa968f0338e0584edf47cff10911109de5"

}