Cannot find running container with name "fuel-core-8.0-postgres"

[Veronica Krayneva]

Steps to reproduce:
1. Deploy cluster on 7.0 (301 iso)
2. Download upgrade arch from here: HTTP_LINK=http://srv89-bud.infra.mirantis.net/fuelweb-iso/fuel-8.0-upgrade-33-2015-10-16_01-42-00.tar.lrz
3. Extract arch, execute ./upgrade.sh

Expected result:
Upgraded cluster to 8.0

Actual result:
2015-10-16 09:28:32 ERROR 13896 (upgrade) *** UPGRADE FAILED
2015-10-16 09:28:32 ERROR 13896 (cli) Cannot find running container with name "fuel-core-8.0-postgres"
Traceback (most recent call last):
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 182, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 175, in run_upgrade
    upgrade_manager.run()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 77, in upgrade
    self.create_and_start_new_containers()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 265, in create_and_start_new_containers
    self.run_after_container_creation_command(container)
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 279, in run_after_container_creation_command
    '', retries=30, interval=4)
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 312, in exec_with_retries
    return func()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 275, in execute
    self.exec_cmd_in_container(container['container_name'], command)
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 286, in exec_cmd_in_container
    db_container_id = self.container_docker_id(container_name)
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 559, in container_docker_id
    'Cannot find running container with name "{0}"'.format(name))
CannotFindContainerError: Cannot find running container with name "fuel-core-8.0-postgres"
==================================================
Upgrade failed

[Ihor Kalnytskyi (ikalnytskyi)]

Well, here's log of postgresql container (probable won't be found in diagnostic snapshot): http://xsnippet.org/361087/raw/

The main problem is here -

Notice: /Stage[main]/Main/Postgresql::Server::Db[keystone]/Postgresql::Server::Database[keystone]/Postgresql_psql[Check for existence of db 'keystone']/command: command changed 'notrun' to 'SELECT 1'
Info: /Stage[main]/Main/Postgresql::Server::Db[keystone]/Postgresql::Server::Database[keystone]/Postgresql_psql[Check for existence of db 'keystone']: Scheduling refresh of Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone']
Notice: /Stage[main]/Main/Postgresql::Server::Db[keystone]/Postgresql::Server::Database[keystone]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone']/returns: createdb: database creation failed: ERROR: database "keystone" already exists
Error: /Stage[main]/Main/Postgresql::Server::Db[keystone]/Postgresql::Server::Database[keystone]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone']: Failed to call refresh: /usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone' returned 1 instead of one of [0]
Error: /Stage[main]/Main/Postgresql::Server::Db[keystone]/Postgresql::Server::Database[keystone]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone']: /usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'keystone' returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Postgresql::Server::Db[ostf]/Postgresql::Server::Database[ostf]/Postgresql_psql[Check for existence of db 'ostf']/command: command changed 'notrun' to 'SELECT 1'
Info: /Stage[main]/Main/Postgresql::Server::Db[ostf]/Postgresql::Server::Database[ostf]/Postgresql_psql[Check for existence of db 'ostf']: Scheduling refresh of Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf']
Notice: /Stage[main]/Main/Postgresql::Server::Db[ostf]/Postgresql::Server::Database[ostf]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf']/returns: createdb: database creation failed: ERROR: database "ostf" already exists
Error: /Stage[main]/Main/Postgresql::Server::Db[ostf]/Postgresql::Server::Database[ostf]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf']: Failed to call refresh: /usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf' returned 1 instead of one of [0]
Error: /Stage[main]/Main/Postgresql::Server::Db[ostf]/Postgresql::Server::Database[ostf]/Exec[/usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf']: /usr/pgsql-9.3/bin/createdb --port='5432' --owner='postgres' --template=template0 'ostf' returned 1 instead of one of [0]

[Ihor Kalnytskyi (ikalnytskyi)]

Move to Fuel library

[Matthew Mosesohn (raytrac3r)]

This bug is caused by a race condition with fuel-upgrade and puppet.

As best as I can tell, the sequence of events is this:
fuel-upgrade starts postgres container
puppet apply starts
prefetch inside puppet thinks these databases don't exist
fuel-upgrade starts creating tables and databases, altering, inserting, etc
by the time puppet gets to trying to create databases, they now exist and the commands fail because the prefetch value doesn't match the current DB state because fuel-upgrade wrote to it

[Ihor Kalnytskyi (ikalnytskyi)]

Ok, so basically that means - it's a bad idea to use puppet manifests inside container since there's no way to control its flow :(

Obviously, we can't remove them now. The only easy workaround is to wait until container is up (puppet is finished) and only then run "after_creation_command". We can use dockerctl to block execution until container is up.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238961

[Ihor Kalnytskyi (ikalnytskyi)]

Should be fixed by https://review.openstack.org/#/c/239395/

[Ihor Kalnytskyi (ikalnytskyi)]

Move it to Fix Commited, since master node upgrade procedure start working on CI either.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Igor Kalnitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/238961
Reason: Looks like the origin problem was fixed by Ie97f3af3a658f1d8244068d425dc9a6319e75823

[Sergey Kraynev (skraynev)]

Checked. Fixed by https://review.openstack.org/#/c/239395/