Docker upgrade script crashes

[Sergii Golovatiuk]

Docker upgrade script (run_upgrade.sh) didn't perform upgrade and died with the following message. Moreover it left the node in non working condition rather than revert containers back to 5.0

2014-05-13 11:24:03 DEBUG 17839 (upgrade) Create container from image fuel/nailgun_5.1: {'command': 'manage.py syncdb', 'ports': None}
2014-05-13 11:24:03 DEBUG 17839 (upgrade) Start container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:25:12 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:26:17 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:27:23 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:28:28 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:29:34 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:30:39 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:31:45 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:32:50 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8": {'publish_all_ports': None, 'links': None, 'lxc_conf': None, 'binds': None, 'dns': None, 'volumes_from': None, 'port_bindings': None, 'privileged': None}
2014-05-13 11:32:52 ERROR 17839 (upgrade) Upgrade failed: Failed to execute migraion command "manage.py syncdb" exit code 1 container id 2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8
2014-05-13 11:32:52 ERROR 17839 (upgrade) Traceback (most recent call last):
  File "/web/fuel-web-4bp-fuel-upgarde/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/upgrade.py", line 656, in run
    self.upgrade()
  File "/web/fuel-web-4bp-fuel-upgarde/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/upgrade.py", line 671, in upgrade
    self.upgrade_engine.upgrade()
  File "/web/fuel-web-4bp-fuel-upgarde/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/upgrade.py", line 77, in upgrade
    self.run_post_build_actions()
  File "/web/fuel-web-4bp-fuel-upgarde/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/upgrade.py", line 388, in run_post_build_actions
    retries_count=8)
  File "/web/fuel-web-4bp-fuel-upgarde/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/upgrade.py", line 442, in run
    params.get('command'), exit_code, container['Id']))
DockerExecutedErrorNonZeroExitCode: Failed to execute migraion command "manage.py syncdb" exit code 1 container id 2b85f065420bf90ec9890777882ae206e75aa833a3e368957d6055b708080ed8

[root@nailgun ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b85f065420b fuel/nailgun_5.1:latest manage.py syncdb 15 minutes ago Exited (1) 5 minutes ago silly_poincare
2a60b4b141b9 fuel/postgres_5.1:latest /usr/local/bin/start 15 minutes ago Up 15 minutes 0.0.0.0:5432->5432/tcp fuel-core-5.1-postgresql
a6e59e48b321 busybox:latest true 15 minutes ago Exited (0) 15 minutes ago fuel-core-5.1-volume_db
d48d27ee8d83 fuel/mcollective_5.0:latest /usr/local/bin/start 42 minutes ago Exited (137) 23 minutes ago fuel-core-5.0-mcollective
1bd8ba40e1d9 fuel/cobbler_5.0:latest /bin/sh -c /usr/loca 42 minutes ago Exited (137) 23 minutes ago fuel-core-5.0-cobbler
5c1acf9e2cff fuel/nginx_5.0:latest /usr/local/bin/start 42 minutes ago Exited (137) 23 minutes ago fuel-core-5.0-nginx
1a11a276f85a fuel/ostf_5.0:latest /bin/sh -c /usr/loca 42 minutes ago Exited (137) 23 minutes ago fuel-core-5.0-ostf
22a804bd08b5 fuel/nailgun_5.0:latest /bin/sh -c /usr/loca 42 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-nailgun
0fb2e623cff0 fuel/rsyslog_5.0:latest /usr/local/bin/start 42 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-rsyslog
fa09e6be7b72 fuel/astute_5.0:latest /bin/sh -c /usr/loca 42 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-astute
4e0cdd283a29 fuel/rsync_5.0:latest /bin/sh -c /usr/loca 42 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-rsync
50bd9bf1c5bf fuel/rabbitmq_5.0:latest /usr/local/bin/start 43 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-rabbitmq
6307daabe3df fuel/postgres_5.0:latest /usr/local/bin/start 43 minutes ago Exited (137) 22 minutes ago fuel-core-5.0-postgres
13ed3378e8a1 storage/puppet:latest /bin/sh -c '/bin/ech 43 minutes ago Exited (0) 43 minutes ago fuel-storage-5.0-puppet
ab2a9212e8fe storage/repo:latest /bin/sh -c '/bin/ech 43 minutes ago Exited (0) 43 minutes ago fuel-storage-5.0-repo
773ea8a3e813 storage/dump:latest /bin/sh -c '/bin/ech 43 minutes ago Exited (0) 43 minutes ago fuel-storage-5.0-dump

[Evgeniy L (rustyrobot)]

Regarding to migration error, I build new containers, checked on 198 iso, it works fine now.

We don't run rollback automatically right now, because in this case it's harder to debug why upgrade failed, you can check that rollback works manually, here is the instruction

https://github.com/rustyrobot/fuel-web/blob/4bp/fuel-upgarde/fuel_upgrade_system/fuel_upgrade/README.md#how-to-run-rollback-manually

If you will see this error again, provide additional information

* make sure that fuel-core-5.1-postgresql container is up and running
* run "docker run -it fuel/nailgun_5.1 manage.py syncdb" to check that migration works

[Andrey Sledzinskiy (asledzinskiy)]

verified on fuel-5.1-upgrade-4-2014-09-12_00-01-11.tar.lrz