[upgrade] Upgrade failed with /tmp/upgrade/pg_dump_all.sql: No such file or directory when run after rollback

[Andrey Sledzinskiy]

Upgrade failed if run it right after rollback

Steps:
1. Create and deploy any 6.0-58 cluster
2. Upload upgrade 6.1 script and run it with fake rollback upgrader
UPGRADERS='host-system docker openstack raise-error' ./upgrade.sh --password *****
3. After successful rollback run upgrade again
./upgrade.sh --no-rollback --password *****

Expected - upgrade is successful
Actual - it failed with
2015-06-20 05:22:26 ERROR 21873 (cli) Shell command executed with "1" exit code: dockerctl shell c5ea9370733a23ad06e9449fbb6b33f3d8f0a49a041351e136d4b4ab041f0e1e su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"
Traceback (most recent call last):
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 176, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 168, in run_upgrade
    upgrade_manager.run()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 79, in upgrade
    self.create_and_start_new_containers()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 273, in create_and_start_new_containers
    self.run_after_container_creation_command(container)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 288, in run_after_container_creation_command
    '', retries=30, interval=4)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 329, in exec_with_retries
    return func()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 284, in execute
    self.exec_cmd_in_container(container['container_name'], command)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 296, in exec_cmd_in_container
    utils.exec_cmd("dockerctl shell {0} {1}".format(db_container_id, cmd))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: dockerctl shell c5ea9370733a23ad06e9449fbb6b33f3d8f0a49a041351e136d4b4ab041f0e1e su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"

Logs are attached

[Ihor Kalnytskyi (ikalnytskyi)]

Ok, looks like a bug in Docker itself. Here's what happened:

1. fuel_upgrade fails to import database dump due to "file not found"

2. we don't have this file inside container under /tmp/upgrade. moreover, we don't have any files inside this folder

3. /tmp/upgrade resource is mounted from host's /var/lib/fuel_upgrade/6.1/

4. /var/lib/fuel_upgrade/6.1/ contains all needed file

5. docker inspect shows, that this folder is indeed mounted resource

    "/tmp/upgrade": "/var/lib/docker/vfs/dir/df6b4cd6a4ecb5f1a244f7bf9dcd4de0c2c755a65d7da34324f574fad87ad564"

6. The virtual fs resource is empty "/var/lib/docker/vfs/dir/df6b4cd6a4ecb5f1a244f7bf9dcd4de0c2c755a65d7da34324f574fad87ad564".

[Ihor Kalnytskyi (ikalnytskyi)]

Andrii, how ofter this bug is reproduced? Perhaps we can set medium prio if it wasn't reproduced since then?

[Andrey Sledzinskiy (asledzinskiy)]

Unfortunately, I can't say about reproduces because during system tests refactoring this test disappeared from CI.
It had 100% reproduces and we can try to reproduce it again

[Andrey Sledzinskiy (asledzinskiy)]

Sorry, I found that test in SWARM
It is constantly reproducing

[Ihor Kalnytskyi (ikalnytskyi)]

Doesn't reproduced for 7.0 on CI. Move it to Invalid.

[Denis Meltsaykin (dmeltsaykin)]

Won't fix for 6.x series as they are unsupported and there was no progress on the issue.